Out of memory though memory is not exceeded
Awaiting user inputI am trying to solve a MILP using Bender's decomposition. Since there are many binary decision variables in the master, I need a lot of iterations. In the first iterations everything works just fine. However when I get to iteration 254 for some reason I get an out of memory error even though the model does not even take 1% of the available RAM. I try to build a new environment and model and solve it again with fewer threads but then I receive a warning " Unable to write to file 'master_model_it_254.lp' " with the corresponding error code. Afterwards for some reason I also cannot solve any more iterations because gurobi cannot find the gurobi license anymore (which is still there). This problem only occurs in the master model. The submodels work just fine. For each model I always create a new environment and model instance.
I also worte the .lp file of the model where the error occured and when I solve the model via the command line it works just fine. I struggle to find any idea of why this is happening. Here are the error messages when it hits the iteration 254 (and later):
// first try
Warning: Failed to open log file 'test.log'
ERROR: Out of memory
// second try and next iterations I receive errors like this
ERROR: No Gurobi license found (user XXXX, host XXXX, hostid XXXX, cores 128)
Here the .log file when it fails due to memory error:
Gurobi 11.0.1 (win64) logging started Thu Jul 25 20:49:58 2024
Set parameter LogToConsole to value 0
Set parameter Username
Set parameter ScaleFlag to value 0
Set parameter BarHomogeneous to value 1
Set parameter AggFill to value 1000
Set parameter IISMethod to value 1
Set parameter LogFile to value "test.log"
Set parameter Threads to value 32
Academic license - for non-commercial use only - expires 2025-02-01
Warning for adding constraints: zero or small (< 1e-13) coefficients, ignored
Gurobi Optimizer version 11.0.1 build v11.0.1rc0 (win64 - Windows Server 2022.0 (20348.2))
CPU model: AMD EPYC 7763 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 32 threads
Optimize a model with 1713 rows, 885 columns and 159602 nonzeros
Model fingerprint: 0x604635c0
Variable types: 3 continuous, 882 integer (882 binary)
Coefficient statistics:
Matrix range [1e-04, 4e+07]
Objective range [1e+00, 4e+03]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+07]
Found heuristic solution: objective 2169.3698223
Presolve removed 847 rows and 637 columns
Presolve time: 0.40s
Presolved: 866 rows, 248 columns, 96574 nonzeros
Found heuristic solution: objective 235.6792793
Variable types: 3 continuous, 245 integer (220 binary)
Found heuristic solution: objective 232.0928374
Root relaxation: objective 8.039470e+01, 104 iterations, 0.01 seconds (0.01 work units)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 80.39470 0 11 232.09284 80.39470 65.4% - 0s
H 0 0 218.9332928 80.39470 63.3% - 0s
H 0 0 206.4067925 80.39470 61.1% - 0s
Explored 0 nodes (0 simplex iterations) in 0.73 seconds (0.39 work units)
Thread count was 1 (of 256 available processors)
Solution count 0
Solve interrupted (error code 10001)
Best objective -, best bound -, gap -
And here if I write the .lp file and solve it from the command line:
Warning: Failed to open log file 'gurobi.log'
Set parameter Username
Set parameter LogFile to value "gurobi.log"
Using license file C:\Users\ws0734\gurobi.lic
Academic license - for non-commercial use only - expires 2025-02-01
Gurobi Optimizer version 11.0.2 build v11.0.2rc0 (win64 - Windows Server 2022.0 (20348.2))
Copyright (c) 2024, Gurobi Optimization, LLC
Read LP format model from file C:\Users\ws0734\source\repos\synerginet\Cases\PaperEJOR\2024-07-24_14-27-50\temp\master_model_it_254.lp
Reading time = 0.27 seconds
: 1713 rows, 885 columns, 159602 nonzeros
CPU model: AMD EPYC 7763 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 32 threads
Optimize a model with 1713 rows, 885 columns and 159602 nonzeros
Model fingerprint: 0xc952a530
Variable types: 3 continuous, 882 integer (882 binary)
Coefficient statistics:
Matrix range [1e-04, 4e+07]
Objective range [1e+00, 4e+03]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+07]
Found heuristic solution: objective 2169.3698223
Presolve removed 847 rows and 637 columns
Presolve time: 0.63s
Presolved: 866 rows, 248 columns, 96574 nonzeros
Found heuristic solution: objective 235.6792793
Variable types: 3 continuous, 245 integer (220 binary)
Found heuristic solution: objective 232.0928374
Root relaxation: objective 7.602992e+01, 58 iterations, 0.02 seconds (0.01 work units)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 76.02992 0 10 232.09284 76.02992 67.2% - 0s
H 0 0 209.9214359 76.02992 63.8% - 1s
0 0 76.45649 0 10 209.92144 76.45649 63.6% - 1s
0 0 76.45649 0 10 209.92144 76.45649 63.6% - 2s
0 0 117.10096 0 10 209.92144 117.10096 44.2% - 3s
H 0 0 203.6509749 133.36870 34.5% - 4s
0 0 133.39897 0 11 203.65097 133.39897 34.5% - 4s
0 0 136.71679 0 11 203.65097 136.71679 32.9% - 5s
0 0 136.80813 0 11 203.65097 136.80813 32.8% - 5s
0 0 139.52401 0 11 203.65097 139.52401 31.5% - 5s
0 0 141.47893 0 11 203.65097 141.47893 30.5% - 5s
0 2 141.47893 0 11 203.65097 141.47893 30.5% - 5s
H 593 578 203.3389077 141.47893 30.4% 7.2 6s
* 1198 855 21 198.1270524 141.47893 28.6% 6.1 6s
H 1473 941 197.2598516 141.47893 28.3% 6.0 6s
H 2675 1409 196.3485070 141.47893 27.9% 5.8 9s
2949 1562 145.96176 11 12 196.34851 141.47893 27.9% 5.8 10s
H 4033 1847 195.5813959 146.23100 25.2% 5.8 10s
6723 2442 193.67201 17 10 195.58140 160.60485 17.9% 5.7 16s
9830 2911 160.60485 26 13 195.58140 160.60485 17.9% 5.9 22s
*12846 2505 35 193.6791761 160.60485 17.1% 5.8 23s
H13899 2421 193.6447043 165.46980 14.5% 5.8 23s
*20998 2263 33 193.4132957 176.04023 8.98% 5.7 24s
21594 2105 186.07903 32 7 193.41330 176.56562 8.71% 5.7 25s
Cutting planes:
Cover: 1
MIR: 1
RLT: 1
Explored 26986 nodes (155261 simplex iterations) in 26.39 seconds (10.92 work units)
Thread count was 32 (of 256 available processors)
Solution count 10: 193.413 193.645 193.679 ... 209.921
Optimal solution found (tolerance 1.00e-04)
Best objective 1.934132956517e+02, best bound 1.934132956517e+02, gap 0.0000%
Any help is much appreciated! I also don´t know why it says gurobi 11.0.1 in the first logs since that version is not installed on my machine. I´m using the C-API.
-
Hi Christian,
Are you using GRBfreemodel and GRBfreeenv to free up the memory of models once you're done with them? If not, this would explain the out of memory error if the models are not large.
I also don´t know why it says gurobi 11.0.1 in the first logs since that version is not installed on my machine
This logfile was produced with Gurobi 11.0.1 - no doubt about it. Are you connecting to a cluster manager? This would suggest so: `Set parameter Username`. I think this could explain a few things you observe, arising from machine attached to the cluster with different versions of Gurobi installed, and machines attached to the cluster that are not licensed (or at least Gurobi cannot find the license on them).
- Riley
0 -
Hi Riley,
yes I am using first GRBfreemodel and right after GRBfreeenv to free the memory. But I also monitored it in during runtime and the total memory usage of the machine was at its peak less than 10%.
No, I am running the code on a local machine. And I checked again and there is no gurobi1101 version installed (there might have been in the past but I deinstalled it and upgraded to 1102). So I guess I´ll try to uninstall all gurobi versions an reinstall the newest 11 version and see if that helps. I´ll share my findings afterwards.
Best regards,
Christian
0 -
Hi Christian,
Are you setting the Username parameter in the code? If not then I'd open your license file in a text editor for clues.
- Riley
0 -
Hi Riley,
I don´t set the Username in the code. However, the license file seems ok and it does work the first 253 iterations.
I removed all gurobi versions from my machine and installed 11.0.3 again. I added following lines to my code:
int major, minor, technical;
GRBversion(&major, &minor, &technical);
printf("Gurobi library version %d.%d.%d\n", major, minor, technical);
fflush(NULL);Which outputs
Gurobi library version 11.0.1
So I am really confused why it does not get the right version. Maybe I did something wrong with my setup? I am working in Visual Studio and made the following changes in my solution properties to get gurobi going:
- Under C/C++ -> General -> Additional Include Directories I added "C:\gurobi1103\win64\include"
- Under Linker -> General -> Additional Library Directories I added "C:\gurobi1103\win64\lib"
- Under Linker -> Input - > Additional Dependencies I added "gurobi110.lib"
Is there anything you can think of that I forgot or did wrong? In the command console it works correctly:
C:\>gurobi_cl --version
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (win64 - Windows Server 2022.0 (20348.2))
Copyright (c) 2024, Gurobi Optimization, LLC0 -
There's a 11.0.1 library sneaking in somewhere. Can you check if your setup is consistent with the following article: How do I configure a new Gurobi C++ project with Microsoft Visual Studio?
I'd also look at the values of your environment variables and see if you can spot any gurobi1101 directories in their values.
0 -
After a while I found the source. I call the C-code from within python (via ctypes dll). In the corresponding anaconda environment I had gurobipy 11.0.1 installed. After installing gurobipy 11.0.3 the output was correct. I still don´t get why this solve the issue with the other versions though. Maybe gurobipy sets some global variable that is called with in the C-function (my only guess).
However, the original Problem regarding the Out of memory and subsequent error in the writing the log files remains. The output is:
// solve master model in iteration 254
Out of memory
// free env and model
// build environment and model again and try to solve
Unable to write to file 'master_model_it_254.lp'
// solve sub model (withouth any problem)
// next iteration, solve master model 255
Unable to write to file 'master_model_it_255.lp'
// and so onIf I look in the .log files (they are created) number 254 says
Gurobi 11.0.3 (win64) logging started Fri Jul 26 19:27:54 2024
Set parameter LogToConsole to value 0
Set parameter Username
Set parameter ScaleFlag to value 0
Set parameter BarHomogeneous to value 1
Set parameter AggFill to value 1000
Set parameter IISMethod to value 1
Set parameter LogFile to value "solver_output_master_model_it_254.log"
Set parameter Threads to value 32
Academic license - for non-commercial use only - expires 2025-02-01
Warning for adding constraints: zero or small (< 1e-13) coefficients, ignored
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (win64 - Windows Server 2022.0 (20348.2))
CPU model: AMD EPYC 7763 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 32 threads
Optimize a model with 1713 rows, 885 columns and 159602 nonzeros
Model fingerprint: 0x604635c0
Variable types: 3 continuous, 882 integer (882 binary)
Coefficient statistics:
Matrix range [1e-04, 4e+07]
Objective range [1e+00, 4e+03]
Bounds range [1e+00, 1e+00]
RHS range [1e+00, 4e+07]
Found heuristic solution: objective 2169.3698223
Presolve removed 847 rows and 637 columns
Presolve time: 0.41s
Presolved: 866 rows, 248 columns, 96574 nonzeros
Found heuristic solution: objective 235.6792793
Variable types: 3 continuous, 245 integer (220 binary)
Found heuristic solution: objective 232.0928374
Root relaxation: objective 8.039470e+01, 104 iterations, 0.01 seconds (0.01 work units)
Nodes | Current Node | Objective Bounds | Work
Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time
0 0 80.39470 0 11 232.09284 80.39470 65.4% - 0s
H 0 0 218.9332928 80.39470 63.3% - 0s
H 0 0 206.4067925 80.39470 61.1% - 0s
Explored 0 nodes (0 simplex iterations) in 0.74 seconds (0.39 work units)
Thread count was 1 (of 256 available processors)
Solution count 0
Solve interrupted (error code 10001)
Best objective -, best bound -, gap -
Gurobi 11.0.3 (win64) logging started Fri Jul 26 19:29:28 2024
Set parameter LogToConsole to value 0
Set parameter Username
Set parameter ScaleFlag to value 0
Set parameter BarHomogeneous to value 1
Set parameter AggFill to value 1000
Set parameter IISMethod to value 1
Set parameter LogFile to value "solver_output_master_model_it_254.log"
Set parameter Threads to value 28
Academic license - for non-commercial use only - expires 2025-02-01
Warning for adding constraints: zero or small (< 1e-13) coefficients, ignoredand 255 says
Gurobi 11.0.3 (win64) logging started Fri Jul 26 19:32:46 2024
Set parameter LogToConsole to value 0
Set parameter Username
Set parameter ScaleFlag to value 0
Set parameter BarHomogeneous to value 1
Set parameter AggFill to value 1000
Set parameter IISMethod to value 1
Set parameter LogFile to value "solver_output_master_model_it_255.log"
Set parameter Threads to value 32
Academic license - for non-commercial use only - expires 2025-02-01
Warning for adding constraints: zero or small (< 1e-13) coefficients, ignoredSo it does create the .log files and write something to it. But somehow crashes afterwards.
Here the code:
GRBenv* env = NULL;
GRBmodel* model = NULL;
int error = 0;
int status = 0;
int i;
// Create empty environment
error = GRBemptyenv(&env);
if (error) goto QUIT;
if (log_file != NULL)
{
error = GRBsetstrparam(env, "LogFile", log_file);
if (error) goto QUIT;
}
// Set parameters
for (i = 0; i < n_int_params; i++)
{
error = GRBsetintparam(env, int_param_names[i], int_param_values[i]);
if (error) goto QUIT;
}
for (i = 0; i < n_double_params; i++)
{
error = GRBsetdblparam(env, double_param_names[i], double_param_values[i]);
if (error) goto QUIT;
}
for (i = 0; i < n_str_params; i++)
{
error = GRBsetstrparam(env, str_param_names[i], str_param_values[i]);
if (error) goto QUIT;
}
// Start environment
error = GRBstartenv(env);
if (error) goto QUIT;
// create model and solve it...
if (error)
{
printf("ERROR: Failed to build and solve the model.\n");
fflush(NULL);
printf("ERROR: %s\n", GRBgeterrormsg(env));
fflush(NULL);
status = error;
}
// Free data, model and environment
GRBfreemodel(model);
GRBfreeenv(env);
return status;Do you have any idea why this could happen?
0 -
Hi Christian,
I'll turn this into a ticket through our Help Center, it'll be easier for me to keep tabs on it.
You'll receive an email to this effect shortly.
- Riley
0
Please sign in to leave a comment.
Comments
7 comments