Impact of memory on time to add Gurobi matrix variables
Answered
Hi there,
I am using .addMvar() from Python API to add several variable vectors, each containing about 150,000 variables, for my project. I do a lot of data processing and storing, which incurs a Python activity exceeding 10 GB according to Activity Monitor (I have 16GB physical memory in my laptop so all activities combined can easily exceed the total), prior to the initialization of Gurobi model, hence before the step of adding these variable vectors. What I experience is the time of .addMvar() is quite long and inconsistent. The first vector takes around 20 seconds to build, the second takes 5 seconds, the third may take 7 seconds.
As a test, I create another script and just implement .addMvar() with the same shape, without any activities that cause high memory usage. .addMvar() in this case is fast and stable, each at around 0.8 seconds.
My question is, can we say in general that a high memory usage might cause the slowness of Gurobi performance, not on the solve part, but on the adding-variable part? Sorry I cannot provide any re-producible code so it's work-related, but I think the test I described is fairly straightforward.
Thanks,
Yuzhou
-
Yes, high memory usage can indeed impact the performance of Gurobi, not just during the solve phase but also when adding variables to the model. When the physical memory (RAM) is heavily utilized or nearly full, operations that require additional memory allocation, such as adding variable vectors via `.addMVar()`, can become slower. This is because the system may need to use swap space on the disk, which is significantly slower than RAM, to manage memory demands. The inconsistency in the time it takes to add variable vectors could be attributed to varying levels of memory availability at the time of each operation, influenced by what other processes are running and how much memory they are consuming. Reducing the memory usage of other processes or optimizing the memory usage of your script before initializing the Gurobi model could help in achieving more consistent performance.
0 -
Also, what does your typical addMVar look like in the script with "modeling activities"? This might have an impact as well.
0 -
Hi Gwyneth,
Can you clarify what you mean by "modeling activities"? I have a procedure which starts with querying data and storing data into an object, as its attributes, and then is followed by a function that takes this object as an input and constructs a Gurobi model. The function .addMVar is literally the first thing I do after the Gurobi model is created. As I mentioned, I observed that, after this procedure starts, there is a Python 3.11 process in the Activity Monitor whose memory usage is over 10G. My understanding is that this Python process takes too much memory which results in the slowness of the subsequent Gurobi process.
Thanks,
Yuzhou
0 -
I was referring to your statement "without any activities that cause high memory usage". I am curious as to what is happening in your addMVar... what arguments are you passing it? How do these differ from your test "without any activities".
You state: "Sorry I cannot provide any re-producible code so it's work-related". If you are a commercial customer, you can open up a Support Request through the Gurobi Help Center for confidential sharing of files. Also please note that Academic Licenses can only be used for academic purposes like coursework, teaching, and academic, published research.
0
Please sign in to leave a comment.
Comments
4 comments