Gurobi token server - release tokens
Answered
Hi,
I am using the third-party modeling framework, Pyomo, as well as MPI (mpi4py) to run several instances of Gurobi in parallel.
I am following the advice given here, but disposeDefaultEnv prints the message,
'Freeing default Gurobi environment,' not, 'Freed default Gurobi environment.'
Thanks,
Jake
-
The Gurobi 9.5 documentation for disposeDefaultEnv() is outdated. The correct message is what you see - \(\texttt{Freeing default Gurobi environment}\). This will be corrected in the documentation for Gurobi 10.0.
(edited for correctness:) If you dispose of all models using the default environment and call disposeDefaultEnv(), you can assume the default environment was freed.
0 -
Hi Eli,
Returning to the article mentioned above, there is another discrepancy.
I don't need the lines,
del m
opt._solver_model.dispose()I, instead, only delete the objective; this is sufficient to free the default environment.
Further, after I dispose of the environment, I reuse components of the model.
Are these actions consistent with the current (but perhaps yet to be released) documentation?
Thanks,
Jake
0 -
Can you clarify what you mean by "delete the objective"? The Model objects associated with a Gurobi environment should be disposed prior to disposing the environment. You can read more about Gurobi environments in the Environments section of the documentation.
0 -
Hello Eli,
By deleting the objective, I mean, for a model m, with the objective set as,
import pyomo.environ as pyo
m.obj = pyo.Objective(rule=obj_expression, sense=pyo.minimize)After setting the instance and solving, I then,
m.del_component(m.obj)
I hope that clarifies.
Additionally, I get the same results whether I dispose of the environment or not (it is just quite a bit slower since I hit the token limit).
Thanks,
Jake
0 -
That helps, thanks. Referring to the \(\texttt{Freeing default Gurobi environment}\) message, I previously stated "If you see this message, you can assume the default environment was freed." This is not entirely correct - you need to first make sure all of the models are disposed, or the environment will not actually be freed. I edited that statement so as to not confuse anyone else who stumbles across this thread.
This is what is happening in your case; because the model is never disposed, the default environment is not free. As a result, no tokens are ever released back to the token server. Clearing out the objective function does not affect the Gurobi environment.
To dispose of the default environment and release the token back to the token server, follow the steps in the article:
opt = SolverFactory("gurobi_persistent")
# build and solve the model
opt._solver_model.dispose()
import gurobipy as gp
gp.disposeDefaultEnv()0 -
Hi Eli,
Thank you.
It would be desirable if the message reflected this to eliminate ambiguity.
To clarify, I also need the line:
del m
after I solve the model m, right?
Thanks,
Jake
0 -
You don't need to delete the Pyomo model object, as it is independent of Gurobi's model and environment objects. It should suffice to call Model.dispose() on the Gurobi Model object that Pyomo creates, which is stored in the \(\texttt{_solver_model}\) attribute of the solver object (e.g., \(\texttt{opt._solver_model}\)).
0 -
Hello Eli,
I am a bit unsure of the scope of Model.dispose().
Say after creating a model, I create an instance of GurobiPersistent via:
import pyomo.environ as pyo
opt = pyo.SolverFactory('gurobi_persistent')Say I then solve:
opt.set_instance(model)
opt.solve()Finally, suppose I then release the token back to the token server:
opt._solver_model.dispose()
gp.disposeDefaultEnv()In my next solve, do I need to recreate an instance of GurobiPersistent (pyo.SolverFactory('gurobi_persistent')) or just tell the solver about the new model (opt.set_instance(model))? Further, if I set a parameter, such as Method, do I need to reset this?
Thanks,
Jake
0 -
You can just call \(\texttt{opt.set_instance(model)}\) to set up the next solve. With this approach, any Gurobi parameters you set on Pyomo's \(\texttt{GurobiPersistent}\) object (e.g., \(\texttt{opt.options['Method'] = 2}\)) should be applied to the next solve.
0 -
Hi Eli,
Do you know if Pyomo creates a Gurobi environment at step a) or b)?
opt.set_instance(model) # a)
opt.solve() # b)Thanks,
Jake
0 -
The Gurobi environment is created in step a).
0 -
How, then, could thread safety ever be a concern (i.e., doesn't this guarantee one model per environment)?
0 -
You create a separate environment for each model in each parallel process, so the environments are only ever accessed by a single thread. That will work fine. Given that Pyomo doesn't give you direct access to the Gurobi environments, I think it would be difficult in this context to misuse a Gurobi environment so it is accessed by multiple threads.
Explicitly disposing of the model and environment has nothing to do with thread safety, but rather with ensuring the token from the token server is released.
Thread safety is more of a concern if you use Gurobi's native Python API (gurobipy). It can be useful from a licensing perspective to re-use one Gurobi environment for sequential solves. You can re-use Gurobi environments like this in a multiprocess program as long as each process uses its own distinct environment. Occasionally, someone will try to create one Gurobi environment and use it for every model in every process - this will likely cause issues, because Gurobi environments are not thread-safe.
0 -
Hello Eli,
You create a separate environment for each model in each parallel process, so the environments are only ever accessed by a single thread.
But you can solve a model with multiple threads.
I am a bit confused about how these are compatible.
Thanks,
Jake
0 -
Each model is tied to a specific environment. However, environments and models are two very different data structures:
- The environment defines the configuration and start/end of a Gurobi session. It is not safe to interact with this data structure from multiple user threads.
- The model is Gurobi's representation of the problem you are trying to solve. Like you noted, Gurobi may use multiple threads to solve the model. Gurobi controls when these threads are launched, what they do, and how they communicate. Gurobi does not access the environment data structure from multiple threads.
0 -
Hello Eli,
- The environment defines the configuration and start/end of a Gurobi session. It is not safe to interact with this data structure from multiple user threads.
What exactly do you mean by interact?
Can each process that creates an environment have multiple threads?
Thank you kindly,
Jake
0 -
What exactly do you mean by interact?
Directly or indirectly accessing or manipulating the environment data structure (in Python, the Env object).
Can each process that creates an environment have multiple threads?
Yes, but we should differentiate between solver threads and user threads.
- Solver threads: These are threads launched and managed by Gurobi to solve the model. Gurobi typically launches multiple threads to solve a model. These threads do not access the environment data structure in an unsafe way.
- User threads: Threads started by the user (e.g., using Python's \(\texttt{threading}\) module). You must not access the same Gurobi environment from multiple user threads.
- User processes: Processes started by the user (by \(\texttt{mpi4py}\) in this case) which do independent work and communicate their results back via message passing.
In your case: your program launches multiple user processes. Within each process, there is only one Gurobi environment, and only one user thread interacts with that environment. Gurobi may still launch multiple solver threads to solve each model in parallel. You will not encounter any thread safety issues with this scheme.
0 -
Hi Eli,
Within each process, there is only one Gurobi environment, and only one user thread interacts with that environment.
How do you know that only one user thread interacts with that environment?
Thanks,
Jake
0 -
By default, Python executes the Python code within a process using a single thread (the Gurobi shared library launches additional solver threads during solve time). In order for multiple user threads to interact with an environment within a process, you would have to explicitly launch threads (e.g., with the \(\texttt{threading}\) module) within that process and direct them to interact with the environment.
0 -
Hello Eli,
There are a few details that I left out of my scheme.
I use MPI for parallelization across multiple nodes. Then, I use the Python library Joblib for parallelization across multiple cores on the same node with the "loky" backend (process-based parallelism). There are two parameters worth mentioning, n_jobs, which controls the maximum number of concurrently running Python worker processes, and inner_max_num_threads, which controls the number of threads each Python worker process is allowed to use.
Additionally, within each process, I call:
opt.set_instance(model)
multiple times, creating multiple environments for each model (I do explicitly dispose of the model and environment before creating a new one). This happens within the Embarrassingly Parallel (Joblib) for loop.
I never work directly with the Env object, so perhaps my concern is not justified.
Also, I am running on a Moab cluster with Torque as the scheduler.
Thanks,
Jake
0 -
Since your code uses process-based parallelism (not thread-based parallelism) and you do not directly interact with the Gurobi environment, you shouldn't run into any issues related to thread safety.
0 -
Okay. Thank you, Eli!
0
Please sign in to leave a comment.
Comments
22 comments