Lower memory usage for MILP problem
Answered
Dear Gurobi Support Team,
I am using Gurobi to solve a mixed integer linear programming problem with ~10^7 continuous variables and ~10^6 binary variables. I am trying to figure out the best parameters for the solver, since I have tried the cluster setting of 128 cores (on one single node) with each core having 4 G of RAM while still getting an out-of-memory error on the cluster. That setting is approaching the maximum amount of memory that I can access.
What I have now tried is setting Methods = 2 and Cuts = 2, while Threads = 128, but I am not entirely sure if I should set Threads as many as my core number. From what I understand from Threads, the number of Threads seems to characterize the number of parallel computation, such as the different branches of branch-and-bound algorithm. Therefore, the more threads I set, the more parallel branches could be computed at once, and likely the sooner the optimal objective would be found. However, the more threads, the more requirement of memory, according to the documentation. How should I actually set the Threads, and if you could be so kind as to help me, how are the threads related to the cores and their RAM space on the cluster?
Additional information: I am using python and pyomo (python-based optimization language) to build the model, if that helps.
Below are the log file that I get during a model run:
And afterwards the job was killed on the cluster due to the out-of-memory problem.
Thanks a lot in advance!
Best regards,
Meijun
-
Hi Meijun,
Seeing the log, there are multiple things that you should consider.
- Update to latest Gurobi version 10
- Your model is very large and Gurobi is able to prove a MIPGap of <1% quite quickly. You might want to consider setting the MIPGap parameter to something bigger than 0.01% unless your application definitely requires such tight mip gap. Please note that given the size of your model, the "time to solution improvement" ratio may not be beneficial. You can see that you get a solution with a 0.98% gap after 10k seconds, but waiting to reach 0.1% takes ~27k seconds. You need to think about whether the wait is worth it in the end.
- You are setting Method=2 which will use the Barrier algorithm. I am quite sure that this is the best setting for this large model and the Barrier algorithm will definitely benefit from the large number of threads you are using. However, the memory consumption increases by a lot because each thread has to hold its own data structures. If memory is an issue I would try reducing the number of threads gradually to 64 and then to 32. Another reason for reducing the number of used threads is also that in the B&B the nodes are processed very slowly and all the threads cannot be utilized properly. The B&B algorithm benefits best from many threads if there are many B&B nodes present which is not the case here. See also Does using more threads make Gurobi faster? and How do I avoid an out-of-memory condition?
- You might want to try to experiment with the NoRelHeurTime which runs the no relaxation heuristic. This heuristic will try to find feasible points even before solving the root relaxation. Given the size of your model I would give the heuristic at least 600s when experimenting with it.
- You are using the BarHomogeneous algorithm which is numerically more stable but can be significantly slower than the traditional Barrier algorithm. You are also setting the ScaleFlag and Cuts parameters. Did you try to run your model without setting those parameters?
Best regards,
Jaromił0 -
Dear Jaromił,
Thank you very much for your help! I have tried several combinations, and have also tried splitting the problem into two parts to solve. However, there is something that still does not work out, and I am a bit confused how I could actually "add up memory" to help the optimization finish. Since you have adviced reducing thread number, I reduce the value of Threads parameter to 64 and also only assign 64 cores with 8 G each, which is the maximum I could get. Under this condition, the optimization cannot finish the branch-and-bound process, as shown below.
I see that the nodes are deeper, does that mean more memory is needed on each thread? Thank you very much in advance!
Best regards,
Meijun
0 -
Hi Meijun,
However, there is something that still does not work out, and I am a bit confused how I could actually "add up memory" to help the optimization finish
By adding up memory, we mean to buy physical hardware to increase the size of your RAM. This would mean to buy additional memory chips and put them into your machine to increase from 8GB to, e.g., 16GB per core.
I see that the nodes are deeper, does that mean more memory is needed on each thread? Thank you very much in advance!
Yes, the more nodes there are in the B&B the more memory is required. I see that you set the NodefileStart parameter which would write nodes to a file if 0.5GB of memory is surpassed. Note that this parameter writes a file to disk, thus you need write-permissions on the machine you are working on. The error message in your screenshot is cut-off, so I cannot really tell why the process has been terminated.
You could try to run this particular model without the NodefileStart parameter and reduce the number of threads to 32 or even 16.
Best regards,
Jaromił0 -
Hi Jaromił,
Thank you very much for your quick answer! Does a core number of 64 help when the number of threads is 32? Or does the core number have to be the same as thread number? Thanks again!
Best,
Meijun
0 -
Hi Meijun,
The number of threads for a process is independent of the number of available processors or physical cores because it is the task of the operating system scheduler to assign threads to processors to execute the instructions of the thread. Threads can be thought of as the maximal number of parallel processes permitted, wherein the allocation of threads to cores is handled by the OS.
Thus, it may make good sense to use "only" 32 Threads even with 64 physical cores if this leads to lower memory consumption or helps avoid other limitations.
Best regards,
Jaromił0 -
Dear Jaromił,
Thank you very much for the explanation! Now it works very well without any memory problems.
Best,
Meijun
0
Please sign in to leave a comment.
Comments
6 comments