Same model, same parameters, but different solutions
Answered
Hi all,
I am running a MIP model on a cluster but getting inconsistencies across identical runs of the same instance. I suspect multi-threading or issues with the hardware on the cluster are affecting the solution paths.
The first inconsistency is that all runs except one ran out of memory. I suspect the concurrent LP optimization at the root node caused a memory spike, as the logs show slight differences in how the relaxation was handled right before crashing.
The second inconsistency is that one specific run found a better solution for another instance. I think the other runs suffered from hardware delays or cluster load, as the "Node 0" log for the best run includes several lines completely missing from the slower ones. Since that successful run was executed on a different day, it probably got lucky and landed on a more free or cooler CPU core.
Am I right in my assumptions? Or could it be another problem?
Following some of the experiments logs:
======================================================================
Log for the first inconsistency (out of memory except for one run):
---------------------------------------------
Run 1 (only one that got a solution)
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [4e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 10s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 20s)...
Presolve removed 4649948 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 80s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 85s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 105s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 130s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 133s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 135s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 135.48s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 110s
Elapsed ordering time = 119s
Elapsed ordering time = 121s
Elapsed ordering time = 125s
Elapsed ordering time = 130s
Elapsed ordering time = 135s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 160s
Elapsed ordering time = 165s
Elapsed ordering time = 171s
Elapsed ordering time = 175s
Elapsed ordering time = 181s
Elapsed ordering time = 185s
Ordering time: 185.14s
Barrier performed 0 iterations in 399.22 seconds (330.51 work units)
Barrier solve interrupted - model solved by another algorithm
Concurrent spin time: 18.80s
Solved with dual simplex
Root relaxation: objective 4.712755e+04, 29613 iterations, 217.62 seconds (120.46 work units)
Total elapsed time = 412.90s (DegenMoves)
Total elapsed time = 426.53s (DegenMoves)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 47127.5467 0 386 - 47127.5467 - - 454s
H 0 0 63959.200000 47127.5467 26.3% - 930s
0 0 47188.4279 0 324 63959.2000 47188.4279 26.2% - 1063s
H 0 0 54301.200000 47189.5079 13.1% - 1488s
H 0 0 54274.680000 47189.5079 13.1% - 1489s
H 0 0 54079.840000 47189.5079 12.7% - 1560s
Cutting planes:
Learned: 1
Implied bound: 26
Clique: 26
MIR: 43
Zero half: 7
RLT: 1
BQP: 20
Explored 1 nodes (80157 simplex iterations) in 1800.61 seconds (1137.77 work units)
Thread count was 4 (of 40 available processors)
Solution count 4: 54079.8 54274.7 54301.2 63959.2
Time limit reached
Best objective 5.407984000000e+04, best bound 4.718950793651e+04, gap 12.7410%---------------------------------------------
Run 2 (no solution)
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [4e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 10s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 20s)...
Presolve removed 4649948 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 80s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 86s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 106s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 120s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 125s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 142s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 147s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 149.63s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 116s
Elapsed ordering time = 120s
Elapsed ordering time = 125s
Elapsed ordering time = 130s
Elapsed ordering time = 135s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 163s
Elapsed ordering time = 168s
Elapsed ordering time = 174s
Elapsed ordering time = 179s
Ordering time: 179.23s
Barrier performed 0 iterations in 414.37 seconds (325.25 work units)
Barrier solve interrupted - model solved by another algorithm
Warning: Possible non-determinism after error
Explored 0 nodes (0 simplex iterations) in 429.44 seconds (246.05 work units)
Thread count was 1 (of 40 available processors)
Solution count 0
Solve interrupted (error code 10001)
Best objective -, best bound -, gap -
---------------------------------------------
Run 3 (no solution)
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [4e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 11s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 21s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 81s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 86s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 105s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 120s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 136s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 143s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 147s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 146.62s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 110s
Elapsed ordering time = 115s
Elapsed ordering time = 120s
Elapsed ordering time = 131s
Elapsed ordering time = 136s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 160s
Elapsed ordering time = 165s
Elapsed ordering time = 170s
Elapsed ordering time = 178s
Elapsed ordering time = 183s
Elapsed ordering time = 189s
Elapsed ordering time = 194s
Ordering time: 193.95s
Barrier performed 0 iterations in 424.97 seconds (322.17 work units)
Barrier solve interrupted - model solved by another algorithm
Warning: Possible non-determinism after error
Explored 0 nodes (0 simplex iterations) in 438.07 seconds (246.05 work units)
Thread count was 1 (of 40 available processors)
Solution count 0
Solve interrupted (error code 10001)
Best objective -, best bound -, gap -======================================================================
Log for the second inconsistency (different values for objective functions)
---------------------------------------------
Run 1 (the only one with the best solution)
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [3e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 1153190 rows and 0 columns (presolve time = 5s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 10s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 15s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 20s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 25s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 31s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 36s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 37.53s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 31s
Elapsed ordering time = 32s
Elapsed ordering time = 32s
Elapsed ordering time = 32s
Ordering time: 32.01s
Barrier performed 0 iterations in 89.62 seconds (51.74 work units)
Barrier solve interrupted - model solved by another algorithm
Concurrent spin time: 1.49s
Solved with dual simplex
Root relaxation: objective 4.646917e+04, 12133 iterations, 40.78 seconds (19.57 work units)
Total elapsed time = 94.28s (DegenMoves)
Total elapsed time = 95.77s (DegenMoves)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 46469.1671 0 282 - 46469.1671 - - 108s
H 0 0 64865.400000 46469.1671 28.4% - 244s
0 0 46479.6442 0 297 64865.4000 46479.6442 28.3% - 284s
H 0 0 64865.000000 46480.5575 28.3% - 318s
H 0 0 59256.000000 46480.5575 21.6% - 318s
H 0 0 54476.120000 46480.5575 14.7% - 321s
H 0 0 54426.160000 46480.5575 14.6% - 321s
H 0 0 54244.640000 46480.5575 14.3% - 328s
H 0 0 54240.560000 46480.5575 14.3% - 329s
H 0 0 53989.200000 46480.5575 13.9% - 368s
H 0 0 53986.720000 46480.5575 13.9% - 368s
H 0 0 53841.240000 46480.5575 13.7% - 386s
H 0 0 53779.000000 46480.5575 13.6% - 405s
0 0 46504.1087 0 181 53779.0000 46504.1087 13.5% - 467s
0 0 46715.5960 0 214 53779.0000 46715.5960 13.1% - 485s
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 501s
0 0 53166.4853 0 215 53779.0000 53166.4853 1.14% - 533s
0 0 53166.4895 0 302 53779.0000 53166.4895 1.14% - 608s
H 0 0 53670.600000 53166.4931 0.94% - 761s
H 0 0 53606.360000 53166.4931 0.82% - 777s
H 0 0 53600.080000 53166.4931 0.81% - 777s
H 0 0 53547.280000 53166.4931 0.71% - 785s
H 0 0 53502.360000 53166.4931 0.63% - 886s
H 0 0 53492.600000 53166.4931 0.61% - 919s
H 0 0 53474.040000 53166.4931 0.58% - 953s
0 0 53166.4931 0 209 53474.0400 53166.4931 0.58% - 956s
0 0 53169.2724 0 202 53474.0400 53169.2724 0.57% - 980s
0 0 53170.0184 0 381 53474.0400 53170.0184 0.57% - 1035s
0 0 53170.0184 0 275 53474.0400 53170.0184 0.57% - 1044s
0 0 53170.7047 0 364 53474.0400 53170.7047 0.57% - 1072s
0 0 53170.7052 0 364 53474.0400 53170.7052 0.57% - 1100s
0 0 53171.4432 0 275 53474.0400 53171.4432 0.57% - 1115s
0 0 53171.4508 0 180 53474.0400 53171.4508 0.57% - 1124s
0 0 53171.8592 0 395 53474.0400 53171.8592 0.57% - 1157s
0 0 53172.4451 0 218 53474.0400 53172.4451 0.56% - 1170s
0 0 53172.4479 0 348 53474.0400 53172.4479 0.56% - 1203s
0 0 53172.5700 0 302 53474.0400 53172.5700 0.56% - 1210s
0 0 53172.9594 0 350 53474.0400 53172.9594 0.56% - 1229s
0 0 53172.9594 0 244 53474.0400 53172.9594 0.56% - 1243s
0 2 53174.5984 0 233 53474.0400 53174.5984 0.56% - 1358s
1 4 53174.5984 1 165 53474.0400 53174.5984 0.56% 3744 1372s
3 8 53174.5984 2 227 53474.0400 53174.5984 0.56% 2202 1395s
7 12 53174.5984 3 223 53474.0400 53174.5984 0.56% 1277 1406s
11 26 53174.5984 4 227 53474.0400 53174.5984 0.56% 1056 1453s
25 52 53183.5964 7 186 53474.0400 53174.5984 0.56% 678 1550s
51 56 53177.4736 11 181 53474.0400 53174.5984 0.56% 520 1606s
H 52 56 53456.080000 53174.5984 0.53% 510 1606s
55 94 53179.6048 12 133 53456.0800 53174.5984 0.53% 519 1748s
93 117 53183.8846 19 126 53456.0800 53174.5984 0.53% 446 1801s
Cutting planes:
Learned: 9
Gomory: 1
Lift-and-project: 2
Cover: 2
Implied bound: 148
Clique: 29
MIR: 24
GUB cover: 1
Zero half: 128
RLT: 52
Relax-and-lift: 2
BQP: 39
Explored 116 nodes (95241 simplex iterations) in 1801.26 seconds (1212.68 work units)
Thread count was 4 (of 40 available processors)
Solution count 10: 53456.1 53474 53492.6 ... 53841.2
Time limit reached
Best objective 5.345608000000e+04, best bound 5.317459839416e+04, gap 0.5266%
---------------------------------------------
Run 2
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [3e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 1152420 rows and 0 columns (presolve time = 6s)...
Presolve removed 1153190 rows and 0 columns (presolve time = 10s)...
Presolve removed 1153950 rows and 760 columns (presolve time = 15s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 20s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 25s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 30s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 35s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 40s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 45s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 51s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 55s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 60s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 65s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 70s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 75s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 80s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 85s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 90s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 112s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 116s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 115.94s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 64s
Elapsed ordering time = 65s
Elapsed ordering time = 65s
Elapsed ordering time = 65s
Ordering time: 64.90s
Barrier performed 0 iterations in 249.21 seconds (51.90 work units)
Barrier solve interrupted - model solved by another algorithm
Concurrent spin time: 3.41s
Solved with dual simplex
Root relaxation: objective 4.646917e+04, 12133 iterations, 94.39 seconds (19.57 work units)
Total elapsed time = 259.76s (DegenMoves)
Total elapsed time = 262.87s (DegenMoves)
Total elapsed time = 265.36s (DegenMoves)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 46469.1671 0 282 - 46469.1671 - - 283s
H 0 0 64865.400000 46469.1671 28.4% - 764s
0 0 46479.6442 0 297 64865.4000 46479.6442 28.3% - 932s
H 0 0 64865.000000 46480.5575 28.3% - 1086s
H 0 0 59256.000000 46480.5575 21.6% - 1086s
H 0 0 54476.120000 46480.5575 14.7% - 1099s
H 0 0 54426.160000 46480.5575 14.6% - 1100s
H 0 0 54244.640000 46480.5575 14.3% - 1124s
H 0 0 54240.560000 46480.5575 14.3% - 1125s
H 0 0 53989.200000 46480.5575 13.9% - 1285s
H 0 0 53986.720000 46480.5575 13.9% - 1285s
H 0 0 53841.240000 46480.5575 13.7% - 1344s
H 0 0 53779.000000 46480.5575 13.6% - 1396s
0 0 46504.1087 0 181 53779.0000 46504.1087 13.5% - 1598s
0 0 46715.5960 0 214 53779.0000 46715.5960 13.1% - 1664s
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 1718s
0 0 53166.4853 0 215 53779.0000 53166.4853 1.14% - 1800s
Cutting planes:
Learned: 9
Gomory: 1
Lift-and-project: 1
Implied bound: 49
Clique: 17
MIR: 5
Zero half: 39
BQP: 18
Explored 1 nodes (28161 simplex iterations) in 1800.79 seconds (334.82 work units)
Thread count was 4 (of 40 available processors)
Solution count 10: 53779 53841.2 53986.7 ... 64865
Time limit reached
Best objective 5.377900000000e+04, best bound 5.316648533333e+04, gap 1.1389%---------------------------------------------
Run 3
---------------------------------------------
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")
CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads
Non-default parameters:
TimeLimit 1800
MemLimit 30
SoftMemLimit 64
MIPGap 1e-09
Threads 4
Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
Matrix range [1e+00, 2e+02]
Objective range [3e-01, 1e+04]
Bounds range [1e+00, 2e+01]
RHS range [1e+00, 6e+01]
Presolve removed 1152420 rows and 0 columns (presolve time = 5s)...
Presolve removed 1153190 rows and 0 columns (presolve time = 11s)...
Presolve removed 1153950 rows and 760 columns (presolve time = 15s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 20s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 25s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 30s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 35s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 40s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 45s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 50s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 55s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 60s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 65s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 70s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 75s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 80s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 85s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 90s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 95s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 103s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 120s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 126s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 126.09s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 46s
Elapsed ordering time = 46s
Elapsed ordering time = 47s
Elapsed ordering time = 47s
Ordering time: 46.73s
Barrier performed 0 iterations in 232.84 seconds (51.80 work units)
Barrier solve interrupted - model solved by another algorithm
Concurrent spin time: 0.94s
Solved with dual simplex
Root relaxation: objective 4.646917e+04, 12133 iterations, 69.66 seconds (19.57 work units)
Total elapsed time = 239.27s (DegenMoves)
Total elapsed time = 241.26s (DegenMoves)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 46469.1671 0 282 - 46469.1671 - - 259s
H 0 0 64865.400000 46469.1671 28.4% - 795s
0 0 46479.6442 0 297 64865.4000 46479.6442 28.3% - 968s
H 0 0 64865.000000 46480.5575 28.3% - 1094s
H 0 0 59256.000000 46480.5575 21.6% - 1094s
H 0 0 54476.120000 46480.5575 14.7% - 1105s
H 0 0 54426.160000 46480.5575 14.6% - 1105s
H 0 0 54244.640000 46480.5575 14.3% - 1125s
H 0 0 54240.560000 46480.5575 14.3% - 1125s
H 0 0 53989.200000 46480.5575 13.9% - 1273s
H 0 0 53986.720000 46480.5575 13.9% - 1273s
H 0 0 53841.240000 46480.5575 13.7% - 1327s
H 0 0 53779.000000 46480.5575 13.6% - 1386s
0 0 46504.1087 0 181 53779.0000 46504.1087 13.5% - 1597s
0 0 46715.5960 0 214 53779.0000 46715.5960 13.1% - 1672s
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 1741s
0 0 53166.4853 0 215 53779.0000 53166.4853 1.14% - 1800s
Cutting planes:
Learned: 9
Gomory: 1
Lift-and-project: 1
Implied bound: 49
Clique: 17
MIR: 5
Zero half: 39
BQP: 18
Explored 1 nodes (28161 simplex iterations) in 1800.84 seconds (333.86 work units)
Thread count was 4 (of 40 available processors)
Solution count 10: 53779 53841.2 53986.7 ... 64865
Time limit reached
Best objective 5.377900000000e+04, best bound 5.316648533333e+04, gap 1.1389%
0
-
-
Gurobi Staff
Hi Arthur,
The first inconsistency is that all runs except one ran out of memory. I suspect the concurrent LP optimization at the root node caused a memory spike, as the logs show slight differences in how the relaxation was handled right before crashing.
Since all three runs show the same fingerprint for the model, we expect that running the model on the same machine with the same parameters leads to the same solution path. You have set both the MemLimit and SoftMemLimit parameters, with the latter set to a higher value. Setting the MemLimit=30 limits the total amount of memory available to Gurobi to 30 GB. This indicates that if more memory is required, Gurobi will fail with an out-of-memory error. The setting SoftMemLimit=64 is irrelevant here. Why did you set both MemLimit and SoftMemLimit parameters?
The ordering in the barrier algorithm appears to be memory-intensive, and it appears that the dual simplex managed to solve the root relaxation to optimality in one of the runs slightly before hitting the memory limit (note that the concurrent method for solving the LP relaxation is non-deterministic). To avoid the issue for this model, you need to either increase the amount of memory available to Gurobi or set the Method parameter to 1, forcing the Gurobi Optimizer to solve the root LP relaxation with the dual simplex.
The second inconsistency is that one specific run found a better solution for another instance. I think the other runs suffered from hardware delays or cluster load, as the "Node 0" log for the best run includes several lines completely missing from the slower ones. Since that successful run was executed on a different day, it probably got lucky and landed on a more free or cooler CPU core.
The solution paths in all three runs are the same. The difference stems from the amount of time that it takes to explore the tree. For example, the gap 1.4% (see below) was reached at 501 seconds in the first run, but it took 1718 and 1741 seconds to reach the same point in the second and the third runs, respectively. Your conjecture is likely correct. For the second and third runs, the machine appears to be oversubscribed. Do you know how many Gurobi jobs were running concurrently on the same machine with the second and the third run?
- First run
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 501s- Second run
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 1718s- Third run
0 0 53166.4853 0 150 53779.0000 53166.4853 1.14% - 1741sBest regards,
Maliheh
0 -
Please sign in to leave a comment.
Comments
1 comment