Python's multiprocessing package can be used to implement process-based parallelism.
Pool example
import multiprocessing as mp
import gurobipy as gp
def solve_model(input_data):
with gp.Env() as env, gp.Model(env=env) as model:
# define model
model.optimize()
# retrieve data from model
if __name__ == '__main__':
with mp.Pool() as pool:
pool.map(solve_model, [input_data1, input_data2, input_data3])
Note: Thread-based parallelism, such as by using the threading module, is not possible because the gurobipy module is not thread-safe.
Environments
Each process should create its own environment when using multiprocessing.
It is important to properly dispose of the models and close the environments. Starting with Gurobi 9, the following pattern automatically discards the model and environment upon leaving the with-block:
with gp.Env() as env, gp.Model(env=env) as model:
# remaining model code
For Gurobi 8 and earlier, use:
env = gp.Env()
model = gp.Model(env=env)
# remaining model code
del model
del env
Issues on macOS 10.13 and later
The multiprocessing package supports different methods for starting the subprocesses. Until Python 3.7, the default method on macOS was forking. However, on macOS 10.13 and later there are some issues: Fork without exec may crash the subprocess; see this bug report. In this case, "spawn" should be used instead. Either change the setting globally:
mp.set_start_method("spawn")
or change it only for one Pool:
with mp.get_context("spawn").Pool() as pool:
# ...
Starting with Python 3.8, spawning is already the default on macOS.
Further information
- Why your multiprocessing Pool is stuck (it’s full of sharks!) from PythonSpeed
Comments
1 comment
good job, extremely helpful.
NOTE: changing the method from "fork" to "spawn" has major implications on your code: all globals won't be copied to your subprocess! you have to tunnel them thru the "input_data" inputs to your solve_model function.
Please sign in to leave a comment.