How to save and load a Gurobi model and solution without rerunning the model after loading
Answered
I am addressing this post to you directly, since you recently answered a very similar post here, which unfortunately did not help me ...
I need to save and load a model and its solution, so that I can restore the state of a tracking-software that I am working on. The software uses a Gurobi model tracking, where binary decision variables determine whether an assignment between objects is part of the final tracking solution.
The assignment class has the following method to determine if an assignment object is part of the solution (here grbVarOfAssignment is private field that has different reference for each assignment object):
public boolean assignmentIsSelected() {
/* here is some ILP validation code that I omit */
GRBVar grbVarOfAssignment = getGRBVar();
long binary_selection_variable_value_rounded = Math.round(grbVarOfAssignment.get(GRB.DoubleAttr.X));
return (binary_selection_variable_value_rounded == 1);
}
This works as expected when called directly after optimization. However I am not able to get this to work, when loading the model+solution that I saved to disk previously. In that case the call to grbVarOfAssignment.get(GRB.DoubleAttr.X) fails with this exception:
gurobi.GRBException: Error at GRBVar.get
Here is how I am saving the model+solution:
model.write("/path/to/ilpModel.mps");
model.write("/path/to/ilpModel.sol");
And this is how I load them:
GRBEnv env = new GRBEnv("/path/to/log_file.log");
GRBModel grbModel = new GRBModel(env,"/path/to/ilpModel.mps");
model.read("/path/to/ilpModel.sol");
I wrote a small test-program, which throws the same exception, when calling var.get(GRB.DoubleAttr.X):
Path pathToTempDir = Files.createTempDirectory("java-");
String modelFilePath = new File("").getAbsolutePath() + "/src/test/resources/gurobi_api_test_data/test_data_1/";
GRBEnv envLoaded = new GRBEnv(pathToTempDir.toAbsolutePath() + "/ilp_loaded.log");
GRBModel modelLoaded = new GRBModel(envLoaded, modelFilePath + "/ilpModel.mps");
modelLoaded.update(); /* inspired by https://support.gurobi.com/hc/en-us/community/posts/4415360665873/comments/4415418190609 */
modelLoaded.read(modelFilePath + "/ilpModel.sol");
modelLoaded.update();
int optimstatus = modelLoaded.get(GRB.IntAttr.Status); /* see https://www.gurobi.com/documentation/9.1/refman/status.html#attr:Status AND https://www.gurobi.com/documentation/9.1/refman/optimization_status_codes.html#sec:StatusCodes */
GRBVar[] vars = modelLoaded.getVars();
for (int ind = 0; ind < vars.length; ind++) {
GRBVar var = vars[ind];
System.out.println("VarName: " + var.get(GRB.StringAttr.VarName)); /* this works */
System.out.println("GRB.DoubleAttr.X: " + var.get(GRB.DoubleAttr.X)); /* this throws the excption: gurobi.GRBException: Error at GRBVar.get */
}
Notably the calls to var.get(GRB.StringAttr.VarName) works.
Some additional comments:
* I noticed that the status of the model after loading is optimstatus equals 1, which means "Model is loaded, but no solution information is available." (as defined here). Perhaps that has to do with it.
* I found this post, which seems is very close to what I want to do. It led me to try the calls to model.update(), but that does not help.
* I also tried saving/loading to *.lp file, but that did not help. I have to say that I am unclear about the interplay between the model-files and the solution file. The documentation did not help much unfortunately.
I would be very grateful for any suggestions on how to solve this issue.
Best regards and thank you,
Michael Mell
-
Hi Michael,
There may be a pretty simple answer to your question: it seems you are missing the modelLoaded.optimize() call. Even when "loading a solution" the variables will not have a solution assignment before the optimization - they will merely be copied into the solution (candidate) pool. This is also true for loading the optimal solution - it will only be checked as a proper solution when calling optimize(). The solver should then return very quickly and all the X attributes of your variables can be queried.
You may want to check out our Diet.java example that applies some model modifications and then calls optimize before querying the new solution values.
I hope that helps!
Cheers,
Matthias0 -
Hi Matthias,
thank you for the quick reply.
So I understand correctly, that the call to model.optimize() is unavoidable even when loading the solution from disk?
But then how can I make sure that the model state/solution is still identical to the loaded model state after that optimization?
I ask, because I want to load the previous model state so that my tracking program is able to reproduce exactly the same track as before.
I need this for reproducibility and also to output new/updated types of analysis that I develop after the original trackings were done (to not go too much into detail: the tracking often requires curation (done through imposing ILP constraints); I want to avoid to have to re-curate everything, so that we quickly re-analyze previous data).Also, this post from Runqiu Hu seems to imply that you can call model.update() instead of model.optimize() in order to access the X-variable that you loaded from the *.sol file (from his previous posts it seems that he tries to do the same thing as me). Did I misunderstand or he actually trying to achieve something different?
Best and thank you for your support,
Michael0 -
Hello Michael,
I received an email notification and saw you mentioned my post for a similar question before.
Actually, after my attempt later, I found that even only reading the result value from a .sol file, you have to
- create variables by calling model.addVars
- read the solution by calling model.read(<the sol file>)
- optimizing the model by calling model.optimize()
Although model.optimize() is still called here, it takes nearly no time to execute. And the values can be retrieved just with those methods like model.getVars().
Sorry for the misunderstanding in the previous post. The method here is currently being used by me and it works in my experiment. Hope that helps.
Best,
Runqiu0 -
Hi Michael,
Yes, you need to call optimize before you can access the X attribute of your variables. I don't understand what you mean by model state and what this tracking program is supposed to be doing.
A model in Gurobi refers to a set of (binary, integer, or continuous) variables that occur with certain coefficients in the constraints and the objective. A solution is merely an assignment of these variables to some values such that the constraints and the variable bounds are satisfied and the objective is hopefully good if not optimal.
There is not very much you can do about the internal state of the solver. After the optimization is complete, there are quite a few data structures that hold for example the branch-and-bound tree and the different solutions found along the way. When reading in a new solution, Gurobi will try to warm-start as best as possible from this internal state to save time during the optimization.
Concerning the other post: No, you cannot access the X attribute after calling the update() method. Gurobi will not even check the solution for feasibility - this only happens when calling optimize().
EDIT: Thank you for the clarification, Runqui! I only saw your reply after sending my own.
Cheers,
Matthias0 -
Dear Matthias, dear Runqiu,
thank you for your replies. They have helped clarify things for me.
@Matthias Miltenberger: I have two more questions and then hopefully I should be happy :) ...
First let me lead by hopefully clarifying/summarizing the issue a bit:
* In my previous post, by "model state" I meant the result of the model after optimization, which is the "state" (or rather result) of the Gurobi model that I want to restore when loading it from disk later on.
* Regarding my application and so that you have better understanding of the issue I am facing:
My application uses Gurobi as backend to track biological cells in images. It uses image-processing to locate cells in the image and then uses binary selection variables in Gurobi to represent assignments between these cells (i.e. using instances of GRBVar), which are optimized under constraints and coefficients that define the physical constraints (e.g. cells can only divide; cells can only move so far between frames; etc.). After optimization, I retrieve the X value from the binary GRBVar instances to determine which assignments are part of the tracking solution.
Now, implementation-wise the application is tightly coupled to the Gurobi model and if the Gurobi model is not correctly initialized/optimized (and hence the GRBVar do not have X-values) it will break. Therefore I need the model to be fully initialized (and hence optimized) to re-export my previous results that I load from disk (which is my goal here). And hence when I load a previous tracking-result to reproduce, I need to make sure that the model is initialized/reoptimized to the exact same state/result as before (since I can only access the X-values of the GRBVar after calling model.optimize()).Sorry for the lengthy lead ...
So with the above in mind, I have two more questions:
1. Which files and file-types should I save/load to/from to most accurately restore the previous optimization result? Since I need reoptimize the loaded Gurobi model before I can use it in the backend of my application, I want to make sure that I restore as much information about the model/previous optimization so that I accurately restore the previous result ...
Right now I am saving/loading to/from a pair of *.mps and *.sol files. But there are a number of other file-types mentioned in the documentation here (among them a JSON solution-file, which from the description here seems to store more information then the *.sol file (though I am unclear, if that can be used for loading or just export); I also saw there is parameter file-type *.prm, which I am unclear about what it is for)...
So which ones do I ideally pick/combine? Note that space is not an issue because my models are small.2. I have a question regarding this statement:
> When reading in a new solution, Gurobi will try to warm-start as best as possible from this internal state to save time during the optimization.First let me summarize my current understanding:
1. The *.mps file holds the model "definition" with all its variables, constraints, coefficients and the objective.
2. The *.sol file holds the resulting variable values from the optimization (i.e. the result).
3. When I load those files and call model.optimize() Gurobi will do a warm-start to quickly arrive at the optimal solution. In principle this solution should be the same as the one that was loaded from the *.sol file, if that one was already optimal.So is there any way to make sure that the result of the model after calling model.optimize() is the same as the one that was loaded from disk? I guess, I could parse the loaded *.sol file and compare the X values in it to the ones obtained from the loaded model after optimization. But is there a way to do something like this using only Gurobi?
Thank you again for your great support and best,
Michael0 -
Hi Michael,
There is no other way of exporting and saving the "state" of the solver than by creating a SOL file. You cannot store the branch-and-bound tree or the presolving transformations. Every time you load the MPS file and the SOL file you will re-create everything from scratch and only add the solution into Gurobi's solution pool that is then checked for feasibility. But to prove that this solution is optimal, you might need to perform a new branch-and-bound optimization - except that it is kickstarted using the provided incumbent.
Any other file type like JSON or PRM has a different use case. The JSON file is used to save some more information that is also reported in the log file and the PRM file is used to save and load parameters to control the behavior of the solver.
So, I suppose you are already the best you can with your approach. The one thing you should watch out for is to keep the Gurobi Model object alive in between optimizations - this will actually preserve the internal state and you are likely to get better performance than by reading in the MPS file every time.
I hope that helps.
Cheers,
Matthias0 -
Hi Matthias,
that clears my last doubts then. Thank you!
It is too bad (and a bit surprising) that you cannot store the full optimizer state (as a type of snapshot of system memory) and load it from disk. It would have been what I need for my use-case. But perhaps that would use too much data for large problems.
Anyway ... at least now, I understand this limitation.
Thanks again and best,
Michael
0 -
Hello everyone, thank you for this thread!
I have a little thing to add, I was doing a similar thing as Michael, but my model takes time at the optimize call, even after reading the solution.
The way to prevent the computation is to set TimeLimit to 1 second (0 sometimes was not enough)
model.params.TimeLimit = 1
model.optimize()The variables are accessible after that.
0 -
Matthias Miltenberger I have a question regarding your comment:
"The one thing you should watch out for is to keep the Gurobi Model object alive in between optimizations - this will actually preserve the internal state and you are likely to get better performance than by reading in the MPS file every time."
How would I go about keeping the Gurobi Model object 'alive' in between optimizations? In my use case I would want to be able to come back to the same model after days or weeks and still be able to use the API's that are available on that 'solved model'.
It's also important to note that I would like to avoid running .optimize() again.
Thanks!
Barrett
0 -
Hi Barrett,
That's simply not possible. You need to keep the model and solving data in memory (that's what I meant be 'alive') to be able to do a restart from that point. The only option to not start from scratch is to use the previous solution as start solution for the new run.
Cheers,
Matthias0
Please sign in to leave a comment.
Comments
10 comments