Unexpected gurobi returned code (137) error: out-of-memory. How to debug?
AnsweredI am running a flavor of the direct current optimal-power-flow (DC-OPF) problem with an integer variable with some budget k. (e.g. sum z_i <= k)
For IEEE 300-bus k = 8 I can successfully run:
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 202.11150 0 98 -0.00000 202.11150 - - 0s
...
37783114 12797 cutoff 51 41.88360 41.88969 0.01% 95.8 58056s
377D86465 9851 cutoff 68 41.88360 41.88841 0.01% 95.8 58062s
...
Optimal solution found (tolerance 1.00e-04)
But for IEEE 300-bus k = 9 I get the OOM error:
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 202.11150 0 92 -0.00000 202.11150 - - 0s
...
21983730 15628169 52.06837 50 97 46.69360 52.80233 13.1% 176 48491s
22012025 15634457 49.63377 50 120 46.69360 52.79746 13.1% 175 48513s
...
ERROR: Solver (gurobi) returned non-zero return code (137)
ERROR: See the solver log above for diagnostic information.
As you can see, the number of explored nodes and total run time is significantly shorter for k=9 when the test case fails. Do you have any suggestions on how to debug this problem?
Thanks in advance!
I will also include a longer version of the logs below. (Could not post full logs due to char limit.)
For k = 8:
Number of buses = 300
Number of lines = 411
Number of generators = 69
...
Set parameter Presolve to value 2
Set parameter MIPFocus to value 3
Set parameter Heuristics to value 0.5
Set parameter Cuts to value 3
Set parameter Threads to value 36
Set parameter TimeLimit to value 86400
Gurobi Optimizer version 12.0.0 build v12.0.0rc1 (linux64 - "Red Hat Enterprise Linux 8.10 (Ootpa)")
CPU model: Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz, instruction set [SSE2|AVX|AVX2|AVX512]
Thread count: 56 physical cores, 112 logical processors, using up to 36 threads
Non-default parameters:
TimeLimit 86400
Heuristics 0.5
MIPFocus 3
Cuts 3
TSPort 28522
Presolve 2
Threads 36
Optimize a model with 10009 rows, 5802 columns and 20958 nonzeros
Model fingerprint: 0x3be4daab
Variable types: 3762 continuous, 2040 integer (2040 binary)
Coefficient statistics:
Matrix range [5e-04, 1e+02]
Objective range [5e-03, 1e+02]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+01]
User MIP start did not produce a new incumbent solution
User MIP start violates constraint c_e_x2717_ by 1.000000000
Found heuristic solution: objective -0.0000000
Presolve removed 6556 rows and 2948 columns
Presolve time: 0.03s
Presolved: 3453 rows, 2854 columns, 9565 nonzeros
Variable types: 2443 continuous, 411 integer (411 binary)
Root relaxation presolve removed 1 rows and 0 columns
Root relaxation presolved: 3452 rows, 2854 columns, 9561 nonzeros
Root relaxation: objective 2.021115e+02, 3136 iterations, 0.14 seconds (0.17 work units)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 202.11150 0 98 -0.00000 202.11150 - - 0s
H 0 0 7.5700000 167.99426 2119% - 0s
0 0 167.99426 0 98 7.57000 167.99426 2119% - 0s
0 0 161.89046 0 104 7.57000 161.89046 2039% - 1s
0 0 161.89046 0 107 7.57000 161.89046 2039% - 1s
0 0 147.30367 0 97 7.57000 147.30367 1846% - 1s
0 0 142.77477 0 100 7.57000 142.77477 1786% - 1s
0 0 142.62629 0 101 7.57000 142.62629 1784% - 1s
0 0 142.61093 0 104 7.57000 142.61093 1784% - 1s
0 0 132.62387 0 103 7.57000 132.62387 1652% - 2s
0 0 132.54095 0 103 7.57000 132.54095 1651% - 2s
0 0 132.53275 0 108 7.57000 132.53275 1651% - 2s
0 0 125.21216 0 111 7.57000 125.21216 1554% - 2s
0 0 124.85872 0 110 7.57000 124.85872 1549% - 2s
0 0 124.84809 0 113 7.57000 124.84809 1549% - 2s
H 0 0 10.1054765 124.84809 1135% - 2s
H 0 0 11.1152735 124.84809 1023% - 2s
H 0 0 11.2115255 124.84809 1014% - 2s
H 0 0 11.2485765 122.69635 991% - 3s
H 0 0 11.2620765 122.69635 989% - 3s
H 0 0 11.2788765 122.69635 988% - 3s
H 0 0 11.3555765 122.69635 980% - 3s
0 0 122.69635 0 122 11.35558 122.69635 980% - 3s
H 0 0 11.4585255 122.27674 967% - 3s
H 0 0 11.4666765 122.27674 966% - 3s
H 0 0 11.4939765 122.27674 964% - 3s
0 0 122.27674 0 121 11.49398 122.27674 964% - 3s
0 0 121.76437 0 120 11.49398 121.76437 959% - 3s
0 0 121.75492 0 125 11.49398 121.75492 959% - 3s
0 0 117.76630 0 116 11.49398 117.76630 925% - 4s
0 0 117.36323 0 119 11.49398 117.36323 921% - 4s
0 0 117.33796 0 120 11.49398 117.33796 921% - 4s
0 0 116.01543 0 112 11.49398 116.01543 909% - 5s
0 0 115.87674 0 122 11.49398 115.87674 908% - 5s
0 0 115.86690 0 123 11.49398 115.86690 908% - 5s
0 0 110.88186 0 118 11.49398 110.88186 865% - 5s
0 0 110.56694 0 120 11.49398 110.56694 862% - 6s
0 0 110.55304 0 123 11.49398 110.55304 862% - 6s
H 0 0 12.1181223 109.19005 801% - 6s
0 0 109.19005 0 127 12.11812 109.19005 801% - 7s
0 0 109.18694 0 127 12.11812 109.18694 801% - 7s
H 0 0 12.1181223 109.17651 801% - 8s
0 2 109.17651 0 127 12.11812 109.17651 801% - 9s
H 31 64 12.1181224 106.18957 776% 141 10s
...
377D86465 9851 cutoff 68 41.88360 41.88841 0.01% 95.8 58062s
Cutting planes:
Gomory: 123
Lift-and-project: 171
MIR: 128
Flow cover: 589
Relax-and-lift: 3
Explored 37788115 nodes (3621402867 simplex iterations) in 58062.13 seconds (120939.00 work units)
Thread count was 36 (of 112 available processors)
Solution count 10: 41.8836 41.8836 41.8831 ... 20.3433
Optimal solution found (tolerance 1.00e-04)
Best objective 4.188359153250e+01, best bound 4.188772613464e+01, gap 0.0099%
For k is 9:
Number of buses = 300
Number of lines = 411
Number of generators = 69
...
Optimize a model with 10009 rows, 5802 columns and 20958 nonzeros
Model fingerprint: 0xdf84d490
Variable types: 3762 continuous, 2040 integer (2040 binary)
Coefficient statistics:
Matrix range [5e-04, 1e+02]
Objective range [5e-03, 1e+02]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+01]
User MIP start did not produce a new incumbent solution
User MIP start violates constraint c_e_x2717_ by 1.000000000
Found heuristic solution: objective -0.0000000
Presolve removed 6556 rows and 2948 columns
Presolve time: 0.14s
Presolved: 3453 rows, 2854 columns, 9565 nonzeros
Variable types: 2443 continuous, 411 integer (411 binary)
Root relaxation presolve removed 1 rows and 0 columns
Root relaxation presolved: 3452 rows, 2854 columns, 9561 nonzeros
Root relaxation: objective 2.021115e+02, 3238 iterations, 0.15 seconds (0.18 work units)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 202.11150 0 92 -0.00000 202.11150 - - 0s
H 0 0 0.2800000 202.11150 - - 0s
0 0 176.27361 0 98 0.28000 176.27361 - - 1s
H 0 0 2.7600000 175.86192 6272% - 1s
0 0 168.36168 0 99 2.76000 168.36168 6000% - 1s
0 0 151.85582 0 98 2.76000 151.85582 5402% - 1s
0 0 150.93659 0 97 2.76000 150.93659 5369% - 1s
0 0 150.92366 0 97 2.76000 150.92366 5368% - 1s
0 0 136.61508 0 105 2.76000 136.61508 4850% - 2s
0 0 136.09149 0 111 2.76000 136.09149 4831% - 2s
0 0 136.07380 0 110 2.76000 136.07380 4830% - 2s
0 0 128.04129 0 111 2.76000 128.04129 4539% - 2s
0 0 127.39266 0 117 2.76000 127.39266 4516% - 3s
0 0 127.37871 0 119 2.76000 127.37871 4515% - 3s
H 0 0 13.1922661 123.09985 833% - 3s
0 0 123.09985 0 122 13.19227 123.09985 833% - 3s
0 0 122.37708 0 129 13.19227 122.37708 828% - 3s
0 0 122.37708 0 126 13.19227 122.37708 828% - 3s
0 0 118.03739 0 127 13.19227 118.03739 795% - 4s
0 0 117.49392 0 133 13.19227 117.49392 791% - 4s
0 0 117.46058 0 135 13.19227 117.46058 790% - 4s
0 0 115.13619 0 129 13.19227 115.13619 773% - 5s
0 0 114.71228 0 136 13.19227 114.71228 770% - 5s
0 0 114.71228 0 137 13.19227 114.71228 770% - 5s
0 0 113.81503 0 130 13.19227 113.81503 763% - 6s
0 0 113.56221 0 130 13.19227 113.56221 761% - 6s
H 0 0 13.2955952 113.55292 754% - 7s
H 0 0 13.7044525 113.55292 729% - 7s
0 2 113.55292 0 130 13.70445 113.55292 729% - 7s
63 100 109.95901 6 133 13.70445 110.43561 706% 192 11s
...
H21864999 15576146 46.6935949 52.82114 13.1% 176 48305s
21865009 15588940 50.92726 48 121 46.69359 52.82111 13.1% 176 48342s
21901045 15598412 47.17932 47 82 46.69359 52.81569 13.1% 176 48376s
21926980 15605346 52.04246 40 135 46.69359 52.81116 13.1% 176 48398s
21946241 15615114 cutoff 56 46.69359 52.80839 13.1% 176 48431s
H21961084 15615109 46.6935962 52.80817 13.1% 176 48435s
21974213 15618324 52.24869 27 86 46.69360 52.80367 13.1% 176 48458s
21983730 15628169 52.06837 50 97 46.69360 52.80233 13.1% 176 48491s
22012025 15634457 49.63377 50 120 46.69360 52.79746 13.1% 175 48513s
/usr/.../gurobi.sh: line 16: 223725 Killed $PYTHONHOME/bin/python3.11 "$@"
ERROR: Solver (gurobi) returned non-zero return code (137)
ERROR: See the solver log above for diagnostic information.
Traceback (most recent call last):
...
628, in solve
raise ApplicationError("Solver (%s) did not exit normally" % self.name)
pyomo.common.errors.ApplicationError: Solver (gurobi) did not exit normally
-
Hi Joshua,
You might want to have a look at the Knowledge Base article How do I avoid an out-of-memory condition?
In general, given the complexity of optimal power flow problems, you might want to consider increasing the requested MIPGap value. The default value is 0.01% which is very tight for problems of high complexity. When setting the MIPGap parameter value, you should think about how much of possible improvement could you allow for your model. Moreover, you should check whether for your test set it is more often the dual bound which takes a long time to converge or whether it is a good primal feasible solution that takes a long time to be found. My guess would be that MIPGap of ~1% should be good enough for a DCOPF model with integers.
Best regards,
Jaromił0
Please sign in to leave a comment.
Comments
1 comment