gurobipy.GurobiError: Job was not found

Dear Gurobi Community,

I am currently running a numerical study in which I send several MIP-Models to a Gurobi Server in sequence. After every few solved models however the optimization process stops with the error code gurobipy.GurobiError: Job was not found. The job in question also does not seem to be listed the Gurobi Cluster Manager.

Could you tell me what I have to fix in order to prevent this from happening?

Attached you will find the log and stacktrace:

Gurobi 9.1.2 (linux64) logging started Wed Jan 5 16:37:14 2022

Changed value of parameter LogFile to ./logs/2022_01_05/16_37_14.log
Prev: Default: 
Changed value of parameter MIPGap to 0.01
Prev: 0.0001 Min: 0.0 Max: inf Default: 0.0001
Changed value of parameter MIPFocus to 2
Prev: 0 Min: 0 Max: 3 Default: 0
Changed value of parameter StartNodeLimit to -2
Prev: -1 Min: -3 Max: 2000000000 Default: -1
Changed value of parameter Method to 2
Prev: -1 Min: -1 Max: 5 Default: -1
Changed value of parameter Cuts to 3
Prev: -1 Min: -1 Max: 3 Default: -1
Changed value of parameter Presolve to 2
Prev: -1 Min: -1 Max: 2 Default: -1
Changed value of parameter Heuristics to 0.0
Prev: 0.05 Min: 0.0 Max: 1.0 Default: 0.05
Changed value of parameter MIPGap to 0.0
Prev: 0.01 Min: 0.0 Max: inf Default: 0.0001
Gurobi Optimizer version 9.1.2 build v9.1.2rc0 (linux64)
Gurobi Compute Server Worker version 9.1.2 build v9.1.2rc0 (linux64)
Thread count: 16 physical cores, 16 logical processors, using up to 16 threads
Optimize a model with 18231 rows, 39061 columns and 118872 nonzeros
Model fingerprint: 0x5db81c3c
Coefficient statistics:
Matrix range [1e+00, 1e+04]
Objective range [1e+00, 2e+05]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+04]
Presolve removed 2103 rows and 1615 columns
Presolve time: 0.10s
Presolved: 16128 rows, 37446 columns, 114727 nonzeros

Ordering time: 0.15s

Barrier statistics:
AA' NZ : 7.513e+04
Factor NZ : 1.181e+06 (roughly 30 MBytes of memory)
Factor Ops : 2.921e+08 (less than 1 second per iteration)
Threads : 16

Objective Residual
Iter Primal Dual Primal Dual Compl Time
0 3.20481117e+11 -4.39448028e+07 2.55e+05 2.00e+02 5.43e+07 0s
1 2.15093156e+11 -1.03161471e+10 1.66e+05 1.44e+03 3.22e+07 0s
2 6.37926496e+10 -1.67539841e+10 3.94e+04 4.57e+02 8.12e+06 0s
3 3.16068197e+10 -1.13718288e+10 1.44e+04 4.99e+01 2.49e+06 0s
4 4.30114708e+09 -6.40718840e+09 9.63e+02 2.19e+00 3.11e+05 0s
5 2.08195590e+09 -3.26594598e+09 4.61e+02 1.13e+00 1.41e+05 0s
6 1.38971131e+09 -1.87545741e+09 3.10e+02 5.36e-01 8.34e+04 0s
7 9.16064210e+08 -1.05166366e+09 2.08e+02 4.48e-02 5.10e+04 0s
8 6.47930624e+08 -5.99525165e+08 1.43e+02 2.92e-10 3.41e+04 0s
9 4.12782336e+08 -4.06624602e+08 7.76e+01 1.93e-10 2.11e+04 0s
10 2.30901274e+08 -1.67186497e+08 3.17e+01 1.90e-10 9.40e+03 1s
11 1.94727941e+08 -5.18285588e+07 2.38e+01 1.49e-10 5.63e+03 1s
12 1.65527934e+08 -1.43211754e+07 1.74e+01 1.42e-03 3.98e+03 1s
13 1.51278740e+08 2.48610515e+07 1.42e+01 4.37e-03 2.69e+03 1s
14 1.30126207e+08 4.62952668e+07 9.54e+00 7.28e-11 1.73e+03 1s
15 1.18856428e+08 6.16589470e+07 7.17e+00 4.27e-11 1.14e+03 1s
16 1.07202224e+08 6.63225643e+07 4.93e+00 3.82e-11 8.14e+02 1s
17 9.40401119e+07 7.37042871e+07 2.37e+00 2.00e-11 4.01e+02 1s
18 8.74731614e+07 7.71185723e+07 1.17e+00 1.30e-11 2.03e+02 1s
19 8.47795931e+07 7.87015489e+07 6.74e-01 9.55e-12 1.19e+02 1s
20 8.32672726e+07 7.93443144e+07 4.03e-01 1.00e-11 7.68e+01 1s
21 8.26424611e+07 7.98310986e+07 2.91e-01 7.28e-12 5.48e+01 1s
22 8.19963083e+07 8.01899286e+07 1.73e-01 8.19e-12 3.53e+01 1s
23 8.13996230e+07 8.04551319e+07 7.44e-02 7.73e-12 1.88e+01 1s
24 8.12105612e+07 8.06770803e+07 4.06e-02 7.28e-12 1.06e+01 1s
25 8.10596281e+07 8.08277192e+07 1.49e-02 1.64e-11 4.67e+00 1s
26 8.10315458e+07 8.08713976e+07 1.01e-02 7.28e-12 3.23e+00 1s
27 8.10180683e+07 8.09141638e+07 7.82e-03 5.80e-12 2.07e+00 1s
28 8.09854584e+07 8.09673764e+07 2.12e-03 1.18e-11 3.42e-01 1s
29 8.09718390e+07 8.09709457e+07 8.58e-05 7.28e-12 1.73e-02 1s
30 8.09711391e+07 8.09711331e+07 1.36e-06 7.28e-12 1.20e-04 1s
31 8.09711353e+07 8.09711353e+07 2.44e-07 1.09e-11 9.92e-09 1s

Barrier solved model in 31 iterations and 0.84 seconds
Optimal objective 8.09711353e+07

Crossover log...

1803 DPushes remaining with DInf 0.0000000e+00 1s
0 DPushes remaining with DInf 0.0000000e+00 1s

6803 PPushes remaining with PInf 0.0000000e+00 1s
0 PPushes remaining with PInf 0.0000000e+00 1s

Push phase complete: Pinf 0.0000000e+00, Dinf 3.1709746e-11 1s

Iteration Objective Primal Inf. Dual Inf. Time
8602 8.0971135e+07 0.000000e+00 0.000000e+00 1s

Solved in 8602 iterations and 1.41 seconds
Optimal objective 8.097113528e+07
Parameter Heuristics unchanged
Value: 0.0 Min: 0.0 Max: 1.0 Default: 0.05
Changed value of parameter MIPGap to 0.005
Prev: 0.0 Min: 0.0 Max: inf Default: 0.0001
Changed value of parameter TimeLimit to 3600.0
Prev: inf Min: 0.0 Max: inf Default: inf
Gurobi Optimizer version 9.1.2 build v9.1.2rc0 (linux64)
Gurobi Compute Server Worker version 9.1.2 build v9.1.2rc0 (linux64)
Thread count: 16 physical cores, 16 logical processors, using up to 16 threads
Optimize a model with 18231 rows, 39061 columns and 118872 nonzeros
Model fingerprint: 0xbec77494
Variable types: 33337 continuous, 5724 integer (5724 binary)
Coefficient statistics:
Matrix range [1e+00, 1e+04]
Objective range [1e+00, 2e+05]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+04]
Presolve removed 2107 rows and 1640 columns
Presolve time: 0.55s
Presolved: 16124 rows, 37421 columns, 114648 nonzeros
Variable types: 31145 continuous, 6276 integer (5652 binary)
Presolved: 16124 rows, 37421 columns, 114648 nonzeros

Root barrier log...

Ordering time: 0.14s

Barrier statistics:
AA' NZ : 7.505e+04
Factor NZ : 1.194e+06 (roughly 30 MBytes of memory)
Factor Ops : 3.011e+08 (less than 1 second per iteration)
Threads : 16

Objective Residual
Iter Primal Dual Primal Dual Compl Time
0 1.00874120e+14 -2.52725129e+13 1.48e+07 0.00e+00 1.13e+10 1s
1 4.28866113e+13 -1.10623750e+13 5.82e+06 2.14e+03 4.13e+09 1s
2 6.83309853e+12 -4.18026873e+12 4.43e+05 8.00e-11 3.94e+08 1s
3 9.40137579e+11 -1.28941273e+12 5.37e+04 1.02e-10 5.78e+07 1s
4 2.74524611e+11 -3.92317880e+11 1.44e+04 2.62e-10 1.55e+07 1s
5 1.03993464e+11 -2.13145546e+11 5.00e+03 2.91e-10 6.47e+06 1s
6 5.67495703e+10 -1.02423313e+11 2.62e+03 2.04e-10 3.22e+06 1s
7 1.61012168e+10 -4.71753897e+10 7.02e+02 3.20e-10 1.14e+06 1s
8 8.03516487e+09 -1.57367029e+10 3.40e+02 3.20e-10 4.26e+05 1s
9 5.58146620e+09 -8.56928777e+09 2.33e+02 4.66e-10 2.49e+05 1s
10 2.74821715e+09 -4.36308693e+09 1.10e+02 1.72e-09 1.21e+05 1s
11 1.58251162e+09 -1.96746040e+09 6.17e+01 1.02e-09 5.94e+04 1s
12 4.25360472e+08 -7.89404638e+08 1.29e+01 1.55e-09 1.90e+04 1s
13 1.80438393e+08 -2.60789635e+08 3.53e+00 5.02e-10 6.67e+03 1s
14 1.40577300e+08 -1.01775820e+08 1.93e+00 2.62e-10 3.60e+03 1s
15 1.21118391e+08 -2.30340265e+07 1.19e+00 1.49e-10 2.12e+03 1s
16 1.05949399e+08 1.07678484e+07 6.53e-01 1.13e-10 1.39e+03 1s
17 9.57833319e+07 4.31238094e+07 3.33e-01 6.00e-11 7.67e+02 1s
18 8.92705887e+07 6.39259166e+07 1.67e-01 2.96e-11 3.67e+02 1s
19 8.63048672e+07 7.00406365e+07 1.02e-01 2.23e-11 2.35e+02 1s
20 8.34963912e+07 7.45164009e+07 4.56e-02 2.18e-11 1.30e+02 1s
21 8.22420435e+07 7.72442776e+07 2.27e-02 2.18e-11 7.26e+01 1s
22 8.16884989e+07 7.86631616e+07 1.25e-02 1.46e-11 4.39e+01 1s
23 8.12974681e+07 7.97993979e+07 5.41e-03 2.91e-11 2.18e+01 1s
24 8.12227134e+07 8.03709793e+07 4.10e-03 2.91e-11 1.23e+01 1s
25 8.11214161e+07 8.06704456e+07 2.38e-03 2.91e-11 6.49e+00 1s
26 8.10583636e+07 8.07676570e+07 1.33e-03 2.91e-11 4.20e+00 1s
27 8.10413586e+07 8.08149820e+07 1.06e-03 2.18e-11 3.27e+00 1s
28 8.10180917e+07 8.08948769e+07 6.76e-04 2.91e-11 1.77e+00 1s
29 8.09851279e+07 8.09703544e+07 1.27e-04 2.91e-11 2.08e-01 1s
30 8.09762045e+07 8.09757192e+07 2.30e-06 4.37e-11 7.00e-03 1s
31 8.09759319e+07 8.09759311e+07 8.03e-07 2.91e-11 1.26e-05 1s
32 8.09759316e+07 8.09759316e+07 9.95e-10 4.37e-11 2.73e-10 1s

Barrier solved model in 32 iterations and 1.41 seconds
Optimal objective 8.09759316e+07


Root relaxation: objective 8.097593e+07, 8166 iterations, 1.50 seconds
Total elapsed time = 15.13s

Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time

0 0 8.0976e+07 0 76 - 8.0976e+07 - - 17s
0 0 8.1002e+07 0 276 - 8.1002e+07 - - 50s
0 0 8.1002e+07 0 498 - 8.1002e+07 - - 50s
0 0 8.1002e+07 0 150 - 8.1002e+07 - - 73s
0 0 8.1002e+07 0 595 - 8.1002e+07 - - 75s
0 0 8.1002e+07 0 643 - 8.1002e+07 - - 77s
0 0 8.1002e+07 0 670 - 8.1002e+07 - - 77s
0 0 8.1005e+07 0 256 - 8.1005e+07 - - 101s
0 0 8.1006e+07 0 477 - 8.1006e+07 - - 102s
0 0 8.1006e+07 0 588 - 8.1006e+07 - - 103s
0 0 8.1007e+07 0 173 - 8.1007e+07 - - 128s
0 0 8.1007e+07 0 504 - 8.1007e+07 - - 129s
0 0 8.1007e+07 0 495 - 8.1007e+07 - - 130s
0 0 8.1008e+07 0 572 - 8.1008e+07 - - 154s
0 0 8.1008e+07 0 607 - 8.1008e+07 - - 156s
0 0 8.1008e+07 0 672 - 8.1008e+07 - - 157s
0 0 8.1011e+07 0 309 - 8.1011e+07 - - 185s
0 0 8.1011e+07 0 501 - 8.1011e+07 - - 187s
0 0 8.1011e+07 0 508 - 8.1011e+07 - - 187s
0 0 8.1012e+07 0 270 - 8.1012e+07 - - 210s
0 0 8.1012e+07 0 641 - 8.1012e+07 - - 214s
0 0 8.1013e+07 0 234 - 8.1013e+07 - - 236s
0 0 8.1013e+07 0 627 - 8.1013e+07 - - 238s
0 0 8.1013e+07 0 578 - 8.1013e+07 - - 239s
0 0 8.1014e+07 0 281 - 8.1014e+07 - - 267s
0 0 8.1014e+07 0 493 - 8.1014e+07 - - 267s
H 0 0 8.219130e+07 8.1014e+07 1.43% - 295s
0 0 8.1014e+07 0 253 8.2191e+07 8.1014e+07 1.43% - 306s
0 0 8.1014e+07 0 559 8.2191e+07 8.1014e+07 1.43% - 306s
0 0 8.1014e+07 0 288 8.2191e+07 8.1014e+07 1.43% - 340s
0 0 8.1015e+07 0 548 8.2191e+07 8.1015e+07 1.43% - 342s
0 0 8.1015e+07 0 621 8.2191e+07 8.1015e+07 1.43% - 343s
0 0 8.1016e+07 0 339 8.2191e+07 8.1016e+07 1.43% - 383s
0 0 8.1016e+07 0 567 8.2191e+07 8.1016e+07 1.43% - 385s
0 0 8.1016e+07 0 631 8.2191e+07 8.1016e+07 1.43% - 386s
0 0 8.1017e+07 0 289 8.2191e+07 8.1017e+07 1.43% - 423s
0 0 8.1017e+07 0 584 8.2191e+07 8.1017e+07 1.43% - 424s
0 0 8.1017e+07 0 330 8.2191e+07 8.1017e+07 1.43% - 469s
0 0 8.1017e+07 0 615 8.2191e+07 8.1017e+07 1.43% - 470s
0 0 8.1017e+07 0 346 8.2191e+07 8.1017e+07 1.43% - 510s
0 0 8.1017e+07 0 342 8.2191e+07 8.1017e+07 1.43% - 525s
0 2 8.1017e+07 0 338 8.2191e+07 8.1017e+07 1.43% - 534s
1 4 8.1017e+07 1 592 8.2191e+07 8.1017e+07 1.43% 7872 559s
3 8 8.1018e+07 2 593 8.2191e+07 8.1017e+07 1.43% 4699 577s

User-callback calls 560, time in user-callback 0.02 sec

Traceback (most recent call last):
File "gurobi_runtime_testing.py", line 31, in <module>
model, decision_variables = solve_model(config, input_data, db_connection)
File "/home/venv/lib/python3.8/site-packages/optimizer/controller.py", line 63, in solve_model
mip_model, objective_value = gurobi_solve_mip(config, mip_model, decision_variables,
File "/home/venv/lib/python3.8/site-packages/optimizer/gurobi_solve_mip.py", line 25, in gurobi_solve_mip
model.model.optimize(first_solution_callback)
File "src/gurobipy/model.pxi", line 847, in gurobipy.Model.optimize
gurobipy.GurobiError: Job was not found

Thank you for your help.

Best,

Markus