Out of memory error while running a large MIP problem
Ongoing
Hi,
I am trying to solve a MIP model in Python using the Gurobi package, but I am getting an Out of Memory error.
I have already set the parameters bellow:
Set parameter TimeLimit to value 3600
Set parameter NodefileStart to value 1
Set parameter PreSparsify to value 2
Set parameter Presolve to value 0
Set parameter MIPFocus to value 3
Set parameter Threads to value 15
Following is the log information:
Gurobi Optimizer version 11.0.1 build v11.0.1rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz, instruction set [SSE2]
Thread count: 20 physical cores, 20 logical processors, using up to 15 threads
Optimize a model with 765630 rows, 702268 columns and 2347653656 nonzeros
Model fingerprint: 0xda7fb1ae
Variable types: 3261 continuous, 699007 integer (690581 binary)
Coefficient statistics:
Matrix range [1e+00, 3e+05]
Objective range [1e+00, 1e+00]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 5e+02]
Presolve removed 0 rows and 0 columns (presolve time = 253s) ...
Presolve removed 0 rows and 0 columns (presolve time = 289s) ...
Presolve removed 0 rows and 0 columns (presolve time = 589s) ...
Variable types: 2461 continuous, 699807 integer (691381 binary)
Explored 0 nodes (0 simplex iterations) in 1812.22 seconds (368.36 work units)
Thread count was 1 (of 20 available processors)
Solution count 0
Solve interrupted (error code 10001)
Best objective -, best bound -, gap -
Are there any tips or suggestions on how to deal with the Out of Memory error?
Thank you for your help!
0
-
Hi Rainne,
It seems that your model is very dense, there are more than 2.3 billions non-zeroes in the matrix.
Your machine needs a substantial amount of memory to handle this.Here are a few comments on the solver parameters you set:
- NodeFileStart=1: This would only help when the solver reaches the branch-and-bound phase and only if many thousands of open branch-and-bound nodes can be written to disk while waiting to be handled.
- Presolve=0: For large models like this, I strongly recommend keeping presolving enabled to reduce the model. You could even consider more aggressive presolving with Presolve=2.
- Threads=15: The model needs to be copied for each thread, so you might first try with Threads=1 and see whether you can save memory with this. Then, increase to 2 or 4 to see if it still fits into memory.
Best regards,
Mario0 -
Hi Mario,
Thank you for your tips!
When I keep presolve enabled, it cannot find a feasible solution considering the time limit setted.
Do you have any other suggestions that I can try to apply to my model?
Set parameter TimeLimit to value 3600
Set parameter NodefileStart to value 0.5
Set parameter PreSparsify to value 2
Set parameter Presolve to value 2
Set parameter MIPFocus to value 3
Set parameter Threads to value 1
CPU model: Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz, instruction set [SSE2]
Thread count: 20 physical cores, 20 logical processors, using up to 1 threads
Optimize a model with 760140 rows, 696802 columns and 2308170820 nonzeros
Model fingerprint: 0xc006377b
Variable types: 3261 continuous, 693541 integer (685181 binary)
Coefficient statistics:
Matrix range [1e+00, 3e+05]
Objective range [1e+00, 1e+00]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 5e+02]
Presolve removed 0 rows and 0 columns (presolve time = 248s) ...
Presolve removed 0 rows and 0 columns (presolve time = 282s) ...
Presolve removed 0 rows and 0 columns (presolve time = 312s) ...
Presolve removed 0 rows and 1 columns (presolve time = 394s) ...
Presolve removed 0 rows and 1 columns (presolve time = 424s) ...
Presolve removed 0 rows and 1 columns (presolve time = 467s) ...
Presolve removed 0 rows and 1 columns (presolve time = 497s) ...
Presolve removed 3540 rows and 1771 columns (presolve time = 743s) ...
Presolve removed 3540 rows and 1771 columns (presolve time = 827s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 857s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 899s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 929s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1061s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1084s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1114s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1156s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1186s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1241s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1323s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1724s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1883s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 1966s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2033s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2035s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2040s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2045s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2050s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2055s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2060s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2065s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2070s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2075s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2080s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2085s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2090s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2095s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2100s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2105s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2110s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2115s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2120s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2125s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2130s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2135s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2140s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2145s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2150s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2155s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2160s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2165s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2170s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2175s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2180s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2185s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2190s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2195s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2200s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2205s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2210s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2215s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2220s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2225s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2230s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2235s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2240s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2245s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2250s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2255s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2260s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2265s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2270s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2275s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2280s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2285s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2290s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2295s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2300s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2305s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2310s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2315s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2320s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2325s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2658s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2789s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 2902s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3015s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3124s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3125s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3130s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3135s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3140s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3145s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3150s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3155s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3160s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3165s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3170s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3175s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3180s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3185s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3190s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3195s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3200s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3205s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3210s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3215s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3220s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3225s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3230s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3235s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3240s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3245s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3250s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3255s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3260s) ...
Presolve removed 108522 rows and 1771 columns (presolve time = 3560s) ...
Presolve removed 108522 rows and 1771 columns
Presolve time: 3559.70s
Explored 0 nodes (0 simplex iterations) in 3892.70 seconds (2601.27 work units)
Thread count was 1 (of 20 available processors)
Solution count 00 -
Hi Rainne,
With parameters
Set parameter PreSparsify to value 2 Set parameter Presolve to value 2
you enabled aggressive presolving, which takes too much time for your model.
What happens if you keep presolving at the default, i.e., remove those two settings?Best regards,
Mario0 -
Hi Mario,
It still cannot find a feasible solution considering the time limit setted.
Set parameter TimeLimit to value 3600
Set parameter NodefileStart to value 0.5
Set parameter MIPFocus to value 3
Set parameter Threads to value 1
CPU model: Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz, instruction set [SSE2]
Thread count: 20 physical cores, 20 logical processors, using up to 1 threads
Optimize a model with 765630 rows, 702268 columns and 2347653656 nonzeros
Model fingerprint: 0xda7fb1ae
Variable types: 3261 continuous, 699007 integer (690581 binary)
Coefficient statistics:
Matrix range [1e+00, 3e+05]
Objective range [1e+00, 1e+00]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 5e+02]
Presolve removed 0 rows and 0 columns (presolve time = 251s) ...
Presolve removed 0 rows and 0 columns (presolve time = 286s) ...
Presolve removed 0 rows and 0 columns (presolve time = 317s) ...
Presolve removed 0 rows and 1 columns (presolve time = 401s) ...
Presolve removed 0 rows and 1 columns (presolve time = 431s) ...
Presolve removed 0 rows and 1 columns (presolve time = 474s) ...
Presolve removed 0 rows and 1 columns (presolve time = 505s) ...
Presolve removed 3540 rows and 1771 columns (presolve time = 757s) ...
Presolve removed 3540 rows and 1771 columns (presolve time = 844s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 874s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 917s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 947s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1084s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1107s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1137s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1179s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1210s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1263s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1346s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1750s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1913s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 1997s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2066s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2070s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2075s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2080s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2085s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2090s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2095s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2100s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2105s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2110s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2115s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2120s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2125s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2130s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2135s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2140s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2145s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2150s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2155s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2160s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2165s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2170s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2175s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2180s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2185s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2190s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2195s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2200s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2205s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2210s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2215s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2220s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2225s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2230s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2235s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2240s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2245s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2250s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2255s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2260s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2265s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2270s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2275s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2280s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2285s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2290s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2295s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2300s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2305s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2310s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2315s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2320s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2325s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2330s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2335s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2340s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2345s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2350s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2355s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2360s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2365s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2370s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2703s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2837s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 2952s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3066s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3177s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3180s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3185s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3190s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3195s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3200s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3205s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3210s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3215s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3220s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3225s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3230s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3235s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3240s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3245s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3250s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3255s) ...
Presolve removed 103302 rows and 1771 columns (presolve time = 3260s) ...
Presolve removed 103302 rows and 1771 columns
Presolve time: 3260.30s
Explored 0 nodes (0 simplex iterations) in 3600.71 seconds (2248.80 work units)
Thread count was 1 (of 20 available processors)
Solution count 0
Time limit reached
Best objective -, best bound -, gap -0 -
Hi Rainne,
This model seems to be tough.
However, I also oversaw an important point: MIPFocus=3 is a meta-parameter and will automatically set Presolve=2. If you want to keep MIPFocus=3 (you should also try 1 and 2), then you could overrule it with Presolve=1 or PrePasses=1.0
Please sign in to leave a comment.
Comments
5 comments