Delayed Result Printing in Gurobi Optimization (Python)
Answered
Dear Support Team,
I hope this message finds you well. I am writing to seek assistance with an issue I've encountered while running an optimization problem using Gurobi for Python.
The optimization process itself runs efficiently and completes in a reasonable time frame, taking approximately 32800 seconds (equivalent to about 9 hours), which is expected given the complexity of the problem.
However, I've noticed a significant delay in printing the results after the optimization process is completed. It takes an additional 8 hours to print the results following the completion message as shown below.
Here is an example of the output lines:
387 6.40622057e+11 8.81885950e+11 1.31e-03 1.54e-08 6.79e-15 32695s
388 6.40622115e+11 8.81885950e+11 1.32e-03 1.49e-08 6.97e-15 32809s
Barrier solved model in 388 iterations and 32809.52 seconds (9049.73 work units)
Optimal objective 6.40622115e+11
I would greatly appreciate your assistance in resolving this issue. It would be ideal if the results could be printed more promptly after the optimization process is completed, as this extended delay significantly impacts my workflow.
Thank you for your attention to this matter, and I look forward to your guidance on resolving this printing delay.
Best regards,
Amir
-
Hi Amir,
Do you get additional output after the log snippet you showed in your message?
It seems you are solving a continuous problem, probably a linear problem. When the barrier algorithm finishes, it may have found a (near-)optimal objective value, but the corresponding solution may have some violations left and even may be slightly sub-optimal. Additionally, the solution is usually not a basic solution.
Therefore, the so-called crossover phase follows the barrier to clean up the solution and moves it to a corner of the polytope, i.e., transforms it to a basic solution. This can take a considerable amount of time in some cases.
If you are already satisfied with the solution you get immediately after barrier, then you can switch off the crossover phase with parameter Crossover=0.
Best regards,
Mario0 -
Thanks for your response. My model is quadratic, continuous, and convex. I use the barrier method to solve the optimization problem.
I set the parameters below for solving the optimization problem:model.Params.Crossover = 0
model.Params.Method = 2
model.Params.BarHomogeneous = 0
model.Params.BarConvTol = 1e-6
model.optimize()As you see, I set the crossover parameter at 0. Although I got the message that says "the barrier method solved model" and also the optimal objective is shown after 9 hours, it takes an additional 8 hours to print the optimal values of the variables I need.
Here is the output I got:
Set parameter TokenServer to value "lic.hpc.unimelb.edu.au"
Set parameter Crossover to value 0
Set parameter Method to value 2
Set parameter BarHomogeneous to value 0
Set parameter BarConvTol to value 1e-06
Gurobi Optimizer version 10.0.1 build v10.0.1rc0 (linux64)
CPU model: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz, instruction set [SSE2|AVX|AVX2|AVX512]
Thread count: 72 physical cores, 72 logical processors, using up to 32 threads
Optimize a model with 66129016 rows, 41131802 columns and 186920724 nonzeros
Model fingerprint: 0xb4489054
Model has 12902080 quadratic objective terms
Coefficient statistics:
Matrix range [7e-07, 1e+03]
Objective range [6e-02, 1e+12]
QObjective range [1e-05, 3e+01]
Bounds range [0e+00, 0e+00]
RHS range [1e-01, 9e+11]
Warning: Model contains large objective coefficients
Warning: Model contains large rhs
Consider reformulating model or setting NumericFocus parameter
to avoid numerical issues.
Presolve removed 0 rows and 0 columns (presolve time = 8s) ...
Presolve removed 9056872 rows and 4702406 columns (presolve time = 10s) ...
Presolve removed 22813842 rows and 7872700 columns (presolve time = 22s) ...
Presolve removed 22813842 rows and 7872700 columns (presolve time = 25s) ...
Presolve removed 22813842 rows and 7872705 columns (presolve time = 31s) ...
Presolve removed 22813842 rows and 7872705 columns (presolve time = 40s) ...
Presolve removed 22821137 rows and 7880000 columns (presolve time = 43s) ...
Presolve removed 30838479 rows and 15897342 columns (presolve time = 47s) ...
Presolve removed 30883153 rows and 15942016 columns (presolve time = 51s) ...
Presolve removed 30883153 rows and 15942016 columns (presolve time = 58s) ...
Presolve removed 30931549 rows and 15942016 columns (presolve time = 64s) ...
Presolve removed 30931549 rows and 15942016 columns (presolve time = 66s) ...
Presolve removed 30931549 rows and 15942016 columns (presolve time = 71s) ...
Presolve removed 30979933 rows and 15942016 columns (presolve time = 80s) ...
Presolve removed 30979933 rows and 15942016 columns (presolve time = 81s) ...
Presolve removed 30979933 rows and 15942016 columns (presolve time = 88s) ...
Presolve removed 30979933 rows and 15942016 columns (presolve time = 90s) ...
Presolve removed 31463856 rows and 15942016 columns (presolve time = 98s) ...
Presolve removed 31463856 rows and 15942016 columns
Presolve time: 100.82s
Presolved: 34665160 rows, 25189786 columns, 113041249 nonzeros
Presolved model has 9232849 quadratic objective terms
Elapsed ordering time = 5s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Ordering time: 19.79s
Barrier statistics:
Dense cols : 1065
AA' NZ : 1.945e+08
Factor NZ : 1.067e+09 (roughly 30.0 GB of memory)
Factor Ops : 1.158e+11 (roughly 3 seconds per iteration)
Threads : 32
Objective Residual
Iter Primal Dual Primal Dual Compl Time
0 -1.62067448e+22 1.62063773e+22 1.43e+12 8.06e+07 1.73e+14 488s
1 -1.29996564e+22 1.29993172e+22 1.28e+12 7.19e+07 1.54e+14 539s
2 -1.20395165e+22 1.20391872e+22 1.23e+12 6.92e+07 1.48e+14 594s
3 -1.05710291e+22 1.05707162e+22 1.15e+12 6.47e+07 1.39e+14 669s
4 -9.66351561e+21 9.66321353e+21 1.10e+12 6.17e+07 1.33e+14 726s
...
...
...
387 6.40622057e+11 8.81885950e+11 1.31e-03 1.54e-08 6.79e-15 32695s
388 6.40622115e+11 8.81885950e+11 1.32e-03 1.49e-08 6.97e-15 32809s
Barrier solved model in 388 iterations and 32809.52 seconds (9049.73 work units)
Optimal objective 6.40622115e+11
/data/gpfs/projects/punim1865/Code/Output_Market_Cournot_planning_Energy_inertia_with_BESS_2.py:2628: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.subplots(figsize =(3.5 , 3))Also, I attempted to retrieve the variable values without generating their corresponding figures. Nevertheless, this process incurred some time delay.
0 -
Mario, could you please address my problem?
Thanks in advance
0 -
Please do not create support tickets and further threads in the community forum if we do not answer immediately to your forum questions. This just leads to more work on our side. We give the license for free to academics, but individual support is not included. As you can see, we still answer most of the questions in the community forum, but probably with a delay.
To your case:
- Your model has 41M variables, so retrieving the values of those variables might take some time. Of course, 8 hours is far too long.
- How do you retrieve the solution values? Could you post the corresponding code snippets? Maybe there is a more efficient way of doing it.
0 -
M=6
R=5
Y=20
Variable_1 = model.addVars(M , R , Y , vtype=GRB.CONTINUOUS)
Output_1 = np.zeros((M,R,Y))
for m in range(M):
for r in range(R):
for y in range(Y):
Output_1[m,r,y] = Variable_1[m,r,y].x
0 -
Instead of retrieving the value for each single variable in for loops, you could try the following:
`Output_1 = model.getAttr('X', Variable_1)`
This should be more efficient.
Are you retrieving any further results than the variable values?
If the delay of 8h happens after optimize() finishes, you can analyze where exactly the time is spent in your code by printing timestamps between the lines etc.0
Please sign in to leave a comment.
Comments
6 comments