Use multiprocessing in Python with Gurobi

Answered

Jing Li

July 05, 2022 07:37

• 
• 

Hello，

Now I am trying to write a parallelizable program using Gurobi (Python 3.6 + Gurobi 9.03), but I am encountering some problems.

As suggested by Gurobi, I should have written it in the following form (1).

 

import multiprocessing as mp

import gurobipy as gp

def solve_model(input_data):

    with gp.Env() as env, gp.Model(env=env) as model:

        # define model

        model.optimize()

        # retrieve data from model

 

if __name__ == '__main__':

    with mp.Pool() as pool:

        pool.map(solve_model, [input_data1, input_data2, input_data3])

 

However, for each input_data I need to call the function solve_model(input_data) several times, during which the constraints and variables of the model remain unchanged, but the objective function will change （like the subproblems of the L-shaped method）. Thus, I do not want to define the model in the solve_model(input_data) function. Instead, define the model before that, so that I don't need to define the model repeatedly every time I call the function, which also facilitates the hot start of the model (since the feasible domain doesn't change). Thus, I write the form (2).

 

with gp.Env() as env, gp.Model(env=env) as model: 

       # define model

 

def solve_model(input_data):      

       model.optimize() 

       # retrieve data from model

 

if __name__ == '__main__': 

       with mp.Pool() as pool: 

       pool.map(solve_model, [input_data1, input_data2, input_data3]

 

However, I found that form (2) is much less efficient than form (1), and the difference in computational time is very large. Form (2) is heavily blocked.

Why does this happen? and how can I solve this problem?

Thanks a lot!

Related to

• Parallelism

0

• Official comment

  

  Simranjit Kaur

  • Gurobi Staff

  November 16, 2025 23:33

  This post is more than three years old. Some information may not be up to date. For current information, please check the Gurobi Documentation or Knowledge Base. If you need more help, please create a new post in the community forum, or try Gurobot, our chatbot interface offering instant, expert-level support.
• 

  Jaromił Najman

  • Gurobi Staff

  August 09, 2022 16:15 Edited

  It looks like in your (2) form, 1 environment is used for every model solution. Since environments are not thread safe, this leads to blocking and possible errors during the optimization process.

  Thus, I do not want to define the model in the solve_model(input_data) function. Instead, define the model before that, so that I don't need to define the model repeatedly every time I call the function, which also facilitates the hot start of the model (since the feasible domain doesn't change)

  All Model objects have to be connected to a given environment. Thus, it is currently not possible to skip (or avoid) the "model building" part in your solve_model function.

  Best regards, 
  Jaromił

  0
• 

  Florian Götz

  March 11, 2023 15:08

  • 
  • 
  • 

  Is this information still up to date?

  Best regards

  Thus, it is currently not possible to skip (or avoid) the "model building" part in your solve_model function.

  0
• 

  Jaromił Najman

  • Gurobi Staff

  March 13, 2023 08:39

  Is this information still up to date?

  Yes, this information is currently still correct.

  0

Post is closed for comments.