How do I use multiprocessing in Python with Gurobi?

Python's multiprocessing package can be used to implement process-based parallelism.

Pool example

import multiprocessing as mp
import gurobipy as gp

def solve_model(input_data):
    with gp.Env() as env, gp.Model(env=env) as model:
        # define model
        model.optimize()
        # retrieve data from model

if __name__ == '__main__':
    with mp.Pool() as pool:
        pool.map(solve_model, [input_data1, input_data2, input_data3])

Note on thread-based parallelism: Gurobi's environment creation and deletion are not thread-safe. In a multithreaded application, create and delete environments in the main thread. Pass each additional thread an initialized environment for its exclusive use.

Environments

Each process should create its own environment when using multiprocessing.

It is important to properly dispose of the models and close the environments. Starting with Gurobi 9, the following pattern automatically discards the model and environment upon leaving the with-block:

with gp.Env() as env, gp.Model(env=env) as model:
    # remaining model code

For Gurobi 8 and earlier, use:

env = gp.Env()
model = gp.Model(env=env)

# remaining model code

del model
del env

Issues on macOS 10.13 and later

The multiprocessing package supports different methods for starting the subprocesses. Until Python 3.7, the default method on macOS was forking. However, on macOS 10.13 and later there are some issues: Fork without exec may crash the subprocess; see this bug report. In this case, "spawn" should be used instead. Either change the setting globally:

mp.set_start_method("spawn")

or change it only for one Pool:

with mp.get_context("spawn").Pool() as pool:
    # ...

Starting with Python 3.8, spawning is already the default on macOS.

Further information

• Why your multiprocessing Pool is stuck (it’s full of sharks!) from PythonSpeed