Skip to main content

Same model, same parameters, but different solutions

Answered
Arthur Cruz
Edited
  • First Comment
  • First Question

Hi all,

I am running a MIP model on a cluster but getting inconsistencies across identical runs of the same instance. I suspect multi-threading or issues with the hardware on the cluster are affecting the solution paths.

The first inconsistency is that all runs except one ran out of memory. I suspect the concurrent LP optimization at the root node caused a memory spike, as the logs show slight differences in how the relaxation was handled right before crashing.

The second inconsistency is that one specific run found a better solution for another instance. I think the other runs suffered from hardware delays or cluster load, as the "Node 0" log for the best run includes several lines completely missing from the slower ones. Since that successful run was executed on a different day, it probably got lucky and landed on a more free or cooler CPU core.

Am I right in my assumptions? Or could it be another problem?

Following some of the experiments logs:
======================================================================

Log for the first inconsistency (out of memory except for one run):

---------------------------------------------
Run 1 (only one that got a solution)
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [4e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 10s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 20s)...
Presolve removed 4649948 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 80s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 85s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 105s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 130s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 133s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 135s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 135.48s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 110s
Elapsed ordering time = 119s
Elapsed ordering time = 121s
Elapsed ordering time = 125s
Elapsed ordering time = 130s
Elapsed ordering time = 135s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 160s
Elapsed ordering time = 165s
Elapsed ordering time = 171s
Elapsed ordering time = 175s
Elapsed ordering time = 181s
Elapsed ordering time = 185s
Ordering time: 185.14s

Barrier performed 0 iterations in 399.22 seconds (330.51 work units)
Barrier solve interrupted - model solved by another algorithm

Concurrent spin time: 18.80s

Solved with dual simplex

Root relaxation: objective 4.712755e+04, 29613 iterations, 217.62 seconds (120.46 work units)
Total elapsed time = 412.90s (DegenMoves)
Total elapsed time = 426.53s (DegenMoves)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0 47127.5467    0  386          - 47127.5467      -     -  454s
H    0     0                    63959.200000 47127.5467  26.3%     -  930s
     0     0 47188.4279    0  324 63959.2000 47188.4279  26.2%     - 1063s
H    0     0                    54301.200000 47189.5079  13.1%     - 1488s
H    0     0                    54274.680000 47189.5079  13.1%     - 1489s
H    0     0                    54079.840000 47189.5079  12.7%     - 1560s

Cutting planes:
  Learned: 1
  Implied bound: 26
  Clique: 26
  MIR: 43
  Zero half: 7
  RLT: 1
  BQP: 20

Explored 1 nodes (80157 simplex iterations) in 1800.61 seconds (1137.77 work units)
Thread count was 4 (of 40 available processors)

Solution count 4: 54079.8 54274.7 54301.2 63959.2 

Time limit reached
Best objective 5.407984000000e+04, best bound 4.718950793651e+04, gap 12.7410%

---------------------------------------------
Run 2 (no solution)
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [4e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 10s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 20s)...
Presolve removed 4649948 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 80s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 86s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 106s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 120s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 125s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 142s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 147s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 149.63s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 116s
Elapsed ordering time = 120s
Elapsed ordering time = 125s
Elapsed ordering time = 130s
Elapsed ordering time = 135s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 163s
Elapsed ordering time = 168s
Elapsed ordering time = 174s
Elapsed ordering time = 179s
Ordering time: 179.23s

Barrier performed 0 iterations in 414.37 seconds (325.25 work units)
Barrier solve interrupted - model solved by another algorithm

Warning: Possible non-determinism after error


Explored 0 nodes (0 simplex iterations) in 429.44 seconds (246.05 work units)
Thread count was 1 (of 40 available processors)

Solution count 0

Solve interrupted (error code 10001)
Best objective -, best bound -, gap -

 

---------------------------------------------
Run 3 (no solution)
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 7858184 rows, 1612900 columns and 23727890 nonzeros
Model fingerprint: 0x1c2b8421
Variable types: 5600 continuous, 1607300 integer (1607300 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [4e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 4628760 rows and 0 columns (presolve time = 5s)...
Presolve removed 4628760 rows and 0 columns (presolve time = 11s)...
Presolve removed 4629490 rows and 0 columns (presolve time = 15s)...
Presolve removed 4630210 rows and 720 columns (presolve time = 21s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 25s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 30s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 35s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 40s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 45s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 50s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 55s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 60s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 65s)...
Presolve removed 4652526 rows and 720 columns (presolve time = 70s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 75s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 81s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 86s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 90s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 95s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 100s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 105s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 110s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 115s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 120s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 136s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 143s)...
Presolve removed 4658126 rows and 720 columns (presolve time = 147s)...
Presolve removed 4658106 rows and 700 columns
Presolve time: 146.62s
Presolved: 3200078 rows, 1612200 columns, 15913512 nonzeros
Variable types: 5600 continuous, 1606600 integer (1606600 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 65s
Elapsed ordering time = 70s
Elapsed ordering time = 75s
Elapsed ordering time = 80s
Elapsed ordering time = 85s
Elapsed ordering time = 90s
Elapsed ordering time = 95s
Elapsed ordering time = 100s
Elapsed ordering time = 105s
Elapsed ordering time = 110s
Elapsed ordering time = 115s
Elapsed ordering time = 120s
Elapsed ordering time = 131s
Elapsed ordering time = 136s
Elapsed ordering time = 140s
Elapsed ordering time = 145s
Elapsed ordering time = 150s
Elapsed ordering time = 155s
Elapsed ordering time = 160s
Elapsed ordering time = 165s
Elapsed ordering time = 170s
Elapsed ordering time = 178s
Elapsed ordering time = 183s
Elapsed ordering time = 189s
Elapsed ordering time = 194s
Ordering time: 193.95s

Barrier performed 0 iterations in 424.97 seconds (322.17 work units)
Barrier solve interrupted - model solved by another algorithm

Warning: Possible non-determinism after error


Explored 0 nodes (0 simplex iterations) in 438.07 seconds (246.05 work units)
Thread count was 1 (of 40 available processors)

Solution count 0

Solve interrupted (error code 10001)
Best objective -, best bound -, gap -

======================================================================

Log for the second inconsistency (different values for objective functions)

---------------------------------------------
Run 1 (the only one with the best solution)
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [3e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 1153190 rows and 0 columns (presolve time = 5s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 10s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 15s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 20s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 25s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 31s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 36s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 37.53s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 31s
Elapsed ordering time = 32s
Elapsed ordering time = 32s
Elapsed ordering time = 32s
Ordering time: 32.01s

Barrier performed 0 iterations in 89.62 seconds (51.74 work units)
Barrier solve interrupted - model solved by another algorithm

Concurrent spin time: 1.49s

Solved with dual simplex

Root relaxation: objective 4.646917e+04, 12133 iterations, 40.78 seconds (19.57 work units)
Total elapsed time = 94.28s (DegenMoves)
Total elapsed time = 95.77s (DegenMoves)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0 46469.1671    0  282          - 46469.1671      -     -  108s
H    0     0                    64865.400000 46469.1671  28.4%     -  244s
     0     0 46479.6442    0  297 64865.4000 46479.6442  28.3%     -  284s
H    0     0                    64865.000000 46480.5575  28.3%     -  318s
H    0     0                    59256.000000 46480.5575  21.6%     -  318s
H    0     0                    54476.120000 46480.5575  14.7%     -  321s
H    0     0                    54426.160000 46480.5575  14.6%     -  321s
H    0     0                    54244.640000 46480.5575  14.3%     -  328s
H    0     0                    54240.560000 46480.5575  14.3%     -  329s
H    0     0                    53989.200000 46480.5575  13.9%     -  368s
H    0     0                    53986.720000 46480.5575  13.9%     -  368s
H    0     0                    53841.240000 46480.5575  13.7%     -  386s
H    0     0                    53779.000000 46480.5575  13.6%     -  405s
     0     0 46504.1087    0  181 53779.0000 46504.1087  13.5%     -  467s
     0     0 46715.5960    0  214 53779.0000 46715.5960  13.1%     -  485s
     0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     -  501s
     0     0 53166.4853    0  215 53779.0000 53166.4853  1.14%     -  533s
     0     0 53166.4895    0  302 53779.0000 53166.4895  1.14%     -  608s
H    0     0                    53670.600000 53166.4931  0.94%     -  761s
H    0     0                    53606.360000 53166.4931  0.82%     -  777s
H    0     0                    53600.080000 53166.4931  0.81%     -  777s
H    0     0                    53547.280000 53166.4931  0.71%     -  785s
H    0     0                    53502.360000 53166.4931  0.63%     -  886s
H    0     0                    53492.600000 53166.4931  0.61%     -  919s
H    0     0                    53474.040000 53166.4931  0.58%     -  953s
     0     0 53166.4931    0  209 53474.0400 53166.4931  0.58%     -  956s
     0     0 53169.2724    0  202 53474.0400 53169.2724  0.57%     -  980s
     0     0 53170.0184    0  381 53474.0400 53170.0184  0.57%     - 1035s
     0     0 53170.0184    0  275 53474.0400 53170.0184  0.57%     - 1044s
     0     0 53170.7047    0  364 53474.0400 53170.7047  0.57%     - 1072s
     0     0 53170.7052    0  364 53474.0400 53170.7052  0.57%     - 1100s
     0     0 53171.4432    0  275 53474.0400 53171.4432  0.57%     - 1115s
     0     0 53171.4508    0  180 53474.0400 53171.4508  0.57%     - 1124s
     0     0 53171.8592    0  395 53474.0400 53171.8592  0.57%     - 1157s
     0     0 53172.4451    0  218 53474.0400 53172.4451  0.56%     - 1170s
     0     0 53172.4479    0  348 53474.0400 53172.4479  0.56%     - 1203s
     0     0 53172.5700    0  302 53474.0400 53172.5700  0.56%     - 1210s
     0     0 53172.9594    0  350 53474.0400 53172.9594  0.56%     - 1229s
     0     0 53172.9594    0  244 53474.0400 53172.9594  0.56%     - 1243s
     0     2 53174.5984    0  233 53474.0400 53174.5984  0.56%     - 1358s
     1     4 53174.5984    1  165 53474.0400 53174.5984  0.56%  3744 1372s
     3     8 53174.5984    2  227 53474.0400 53174.5984  0.56%  2202 1395s
     7    12 53174.5984    3  223 53474.0400 53174.5984  0.56%  1277 1406s
    11    26 53174.5984    4  227 53474.0400 53174.5984  0.56%  1056 1453s
    25    52 53183.5964    7  186 53474.0400 53174.5984  0.56%   678 1550s
    51    56 53177.4736   11  181 53474.0400 53174.5984  0.56%   520 1606s
H   52    56                    53456.080000 53174.5984  0.53%   510 1606s
    55    94 53179.6048   12  133 53456.0800 53174.5984  0.53%   519 1748s
    93   117 53183.8846   19  126 53456.0800 53174.5984  0.53%   446 1801s

Cutting planes:
  Learned: 9
  Gomory: 1
  Lift-and-project: 2
  Cover: 2
  Implied bound: 148
  Clique: 29
  MIR: 24
  GUB cover: 1
  Zero half: 128
  RLT: 52
  Relax-and-lift: 2
  BQP: 39

Explored 116 nodes (95241 simplex iterations) in 1801.26 seconds (1212.68 work units)
Thread count was 4 (of 40 available processors)

Solution count 10: 53456.1 53474 53492.6 ... 53841.2

Time limit reached
Best objective 5.345608000000e+04, best bound 5.317459839416e+04, gap 0.5266%

 

---------------------------------------------
Run 2
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [3e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 1152420 rows and 0 columns (presolve time = 6s)...
Presolve removed 1153190 rows and 0 columns (presolve time = 10s)...
Presolve removed 1153950 rows and 760 columns (presolve time = 15s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 20s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 25s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 30s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 35s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 40s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 45s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 51s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 55s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 60s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 65s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 70s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 75s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 80s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 85s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 90s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 112s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 116s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 115.94s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 50s
Elapsed ordering time = 55s
Elapsed ordering time = 60s
Elapsed ordering time = 64s
Elapsed ordering time = 65s
Elapsed ordering time = 65s
Elapsed ordering time = 65s
Ordering time: 64.90s

Barrier performed 0 iterations in 249.21 seconds (51.90 work units)
Barrier solve interrupted - model solved by another algorithm

Concurrent spin time: 3.41s

Solved with dual simplex

Root relaxation: objective 4.646917e+04, 12133 iterations, 94.39 seconds (19.57 work units)
Total elapsed time = 259.76s (DegenMoves)
Total elapsed time = 262.87s (DegenMoves)
Total elapsed time = 265.36s (DegenMoves)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0 46469.1671    0  282          - 46469.1671      -     -  283s
H    0     0                    64865.400000 46469.1671  28.4%     -  764s
     0     0 46479.6442    0  297 64865.4000 46479.6442  28.3%     -  932s
H    0     0                    64865.000000 46480.5575  28.3%     - 1086s
H    0     0                    59256.000000 46480.5575  21.6%     - 1086s
H    0     0                    54476.120000 46480.5575  14.7%     - 1099s
H    0     0                    54426.160000 46480.5575  14.6%     - 1100s
H    0     0                    54244.640000 46480.5575  14.3%     - 1124s
H    0     0                    54240.560000 46480.5575  14.3%     - 1125s
H    0     0                    53989.200000 46480.5575  13.9%     - 1285s
H    0     0                    53986.720000 46480.5575  13.9%     - 1285s
H    0     0                    53841.240000 46480.5575  13.7%     - 1344s
H    0     0                    53779.000000 46480.5575  13.6%     - 1396s
     0     0 46504.1087    0  181 53779.0000 46504.1087  13.5%     - 1598s
     0     0 46715.5960    0  214 53779.0000 46715.5960  13.1%     - 1664s
     0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     - 1718s
     0     0 53166.4853    0  215 53779.0000 53166.4853  1.14%     - 1800s

Cutting planes:
  Learned: 9
  Gomory: 1
  Lift-and-project: 1
  Implied bound: 49
  Clique: 17
  MIR: 5
  Zero half: 39
  BQP: 18

Explored 1 nodes (28161 simplex iterations) in 1800.79 seconds (334.82 work units)
Thread count was 4 (of 40 available processors)

Solution count 10: 53779 53841.2 53986.7 ... 64865

Time limit reached
Best objective 5.377900000000e+04, best bound 5.316648533333e+04, gap 1.1389%

---------------------------------------------
Run 3
---------------------------------------------

Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "CentOS Linux 7 (Core)")

CPU model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, instruction set [SSE2|AVX]
Thread count: 20 physical cores, 40 logical processors, using up to 4 threads

Non-default parameters:
TimeLimit  1800
MemLimit  30
SoftMemLimit  64
MIPGap  1e-09
Threads  4

Optimize a model with 1972304 rows, 411280 columns and 5961750 nonzeros
Model fingerprint: 0xaa0683fd
Variable types: 2800 continuous, 408480 integer (408480 binary)
Coefficient statistics:
  Matrix range     [1e+00, 2e+02]
  Objective range  [3e-01, 1e+04]
  Bounds range     [1e+00, 2e+01]
  RHS range        [1e+00, 6e+01]
Presolve removed 1152420 rows and 0 columns (presolve time = 5s)...
Presolve removed 1153190 rows and 0 columns (presolve time = 11s)...
Presolve removed 1153950 rows and 760 columns (presolve time = 15s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 20s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 25s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 30s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 35s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 40s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 45s)...
Presolve removed 1160448 rows and 760 columns (presolve time = 50s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 55s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 60s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 65s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 70s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 75s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 80s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 85s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 90s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 95s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 103s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 120s)...
Presolve removed 1163248 rows and 760 columns (presolve time = 126s)...
Presolve removed 1163228 rows and 740 columns
Presolve time: 126.09s
Presolved: 809076 rows, 410540 columns, 4002546 nonzeros
Variable types: 2800 continuous, 407740 integer (407740 binary)

Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
Elapsed ordering time = 25s
Elapsed ordering time = 30s
Elapsed ordering time = 35s
Elapsed ordering time = 40s
Elapsed ordering time = 45s
Elapsed ordering time = 46s
Elapsed ordering time = 46s
Elapsed ordering time = 47s
Elapsed ordering time = 47s
Ordering time: 46.73s

Barrier performed 0 iterations in 232.84 seconds (51.80 work units)
Barrier solve interrupted - model solved by another algorithm

Concurrent spin time: 0.94s

Solved with dual simplex

Root relaxation: objective 4.646917e+04, 12133 iterations, 69.66 seconds (19.57 work units)
Total elapsed time = 239.27s (DegenMoves)
Total elapsed time = 241.26s (DegenMoves)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0 46469.1671    0  282          - 46469.1671      -     -  259s
H    0     0                    64865.400000 46469.1671  28.4%     -  795s
     0     0 46479.6442    0  297 64865.4000 46479.6442  28.3%     -  968s
H    0     0                    64865.000000 46480.5575  28.3%     - 1094s
H    0     0                    59256.000000 46480.5575  21.6%     - 1094s
H    0     0                    54476.120000 46480.5575  14.7%     - 1105s
H    0     0                    54426.160000 46480.5575  14.6%     - 1105s
H    0     0                    54244.640000 46480.5575  14.3%     - 1125s
H    0     0                    54240.560000 46480.5575  14.3%     - 1125s
H    0     0                    53989.200000 46480.5575  13.9%     - 1273s
H    0     0                    53986.720000 46480.5575  13.9%     - 1273s
H    0     0                    53841.240000 46480.5575  13.7%     - 1327s
H    0     0                    53779.000000 46480.5575  13.6%     - 1386s
     0     0 46504.1087    0  181 53779.0000 46504.1087  13.5%     - 1597s
     0     0 46715.5960    0  214 53779.0000 46715.5960  13.1%     - 1672s
     0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     - 1741s
     0     0 53166.4853    0  215 53779.0000 53166.4853  1.14%     - 1800s

Cutting planes:
  Learned: 9
  Gomory: 1
  Lift-and-project: 1
  Implied bound: 49
  Clique: 17
  MIR: 5
  Zero half: 39
  BQP: 18

Explored 1 nodes (28161 simplex iterations) in 1800.84 seconds (333.86 work units)
Thread count was 4 (of 40 available processors)

Solution count 10: 53779 53841.2 53986.7 ... 64865

Time limit reached
Best objective 5.377900000000e+04, best bound 5.316648533333e+04, gap 1.1389%

Related to

0

Comments

4 comments

Date Votes
  • Maliheh Aramon
    • Gurobi Staff

    Hi Arthur, 

    The first inconsistency is that all runs except one ran out of memory. I suspect the concurrent LP optimization at the root node caused a memory spike, as the logs show slight differences in how the relaxation was handled right before crashing.

    Since all three runs show the same fingerprint for the model, we expect that running the model on the same machine with the same parameters leads to the same solution path. You have set both the MemLimit and SoftMemLimit parameters, with the latter set to a higher value. Setting the MemLimit=30 limits the total amount of memory available to Gurobi to 30 GB. This indicates that if more memory is required, Gurobi will fail with an out-of-memory error. The setting SoftMemLimit=64 is irrelevant here. Why did you set both MemLimit and SoftMemLimit parameters? 

    The ordering in the barrier algorithm appears to be memory-intensive, and it appears that the dual simplex managed to solve the root relaxation to optimality in one of the runs slightly before hitting the memory limit (note that the concurrent method for solving the LP relaxation is non-deterministic). To avoid the issue for this model, you need to either increase the amount of memory available to Gurobi or set the Method parameter to 1, forcing the Gurobi Optimizer to solve the root LP relaxation with the dual simplex. 

    The second inconsistency is that one specific run found a better solution for another instance. I think the other runs suffered from hardware delays or cluster load, as the "Node 0" log for the best run includes several lines completely missing from the slower ones. Since that successful run was executed on a different day, it probably got lucky and landed on a more free or cooler CPU core.

    The solution paths in all three runs are the same. The difference stems from the amount of time that it takes to explore the tree. For example, the gap 1.4% (see below) was reached at 501 seconds in the first run, but it took 1718 and 1741 seconds to reach the same point in the second and the third runs, respectively. Your conjecture is likely correct. For the second and third runs, the machine appears to be oversubscribed. Do you know how many Gurobi jobs were running concurrently on the same machine with the second and the third run?

    - First run

        0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     -  501s

    - Second run

         0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     - 1718s

    - Third run

         0     0 53166.4853    0  150 53779.0000 53166.4853  1.14%     - 1741s

     

    Best regards,

    Maliheh

    0
  • Arthur Cruz
    • First Comment
    • First Question

    Hello Maliheh,

    thank you for the answer.

    Why did you set both MemLimit and SoftMemLimit parameters? 

    MemLimit and SoftMemLimit are both input parameters in my program. For these experiments, I only needed MemLimit. However, rather than modifying the code to remove SoftMemLimit, I simply set its value higher than MemLimit to ensure it would not be triggered.

    To avoid the issue for this model, you need to either increase the amount of memory available to Gurobi or set the Method parameter to 1, forcing the Gurobi Optimizer to solve the root LP relaxation with the dual simplex. 

    I expected some instances to get out-of-memory errors, as previous experiments indicated that this could occur with larger datasets. I consider these to be valid results, as my goal is to map the model's limitations. The main problem was the inconsistency between runs.

    I looked up the Method parameter. Would setting it to 4 (deterministic concurrent) ensure or improve the chances that all runs get the same result (all of them solving the LP with simplex or all of them running out of memory)?

    Do you know how many Gurobi jobs were running concurrently on the same machine with the second and the third run?

    I do not know the number of jobs, but knowing that my assumption is probably correct will help me in my future experiments. Thank you!

    Best regards,

    Arthur.

    0
  • Maliheh Aramon
    • Gurobi Staff

    I looked up the Method parameter. Would setting it to 4 (deterministic concurrent) ensure or improve the chances that all runs get the same result (all of them solving the LP with simplex or all of them running out of memory)?

    Yes, the deterministic concurrent method should ensure the exact same results.

    I do not know the number of jobs, but knowing that my assumption is probably correct will help me in my future experiments.

    Yes, our best explanation for your results is that the machine was oversubscribed (you can also compare the work units value across all three runs). By oversubscription, I mean that the total number of threads over all running Gurobi processes was likely larger than the number of logical processors on the machine, where the operating system scheduler needed to periodically pause threads to give all threads their share of processor time, causing some or all of Gurobi's jobs to slow down. 

    Best regards,

    Maliheh

    0
  • Arthur Cruz
    • First Comment
    • First Question

    I see.

    I do not have any more questions about this issue.

    Thank you, Maliheh.

    Best regards,

    Arthur.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post