Use multiprocessing in Python with Gurobi

Answered

Jing Li

• July 05, 2022 07:37

Hello，

Now I am trying to write a parallelizable program using Gurobi (Python 3.6 + Gurobi 9.03), but I am encountering some problems.

As suggested by Gurobi, I should have written it in the following form (1).

 

import multiprocessing as mp

import gurobipy as gp

def solve_model(input_data):

    with gp.Env() as env, gp.Model(env=env) as model:

        # define model

        model.optimize()

        # retrieve data from model

 

if __name__ == '__main__':

    with mp.Pool() as pool:

        pool.map(solve_model, [input_data1, input_data2, input_data3])

 

However, for each input_data I need to call the function solve_model(input_data) several times, during which the constraints and variables of the model remain unchanged, but the objective function will change （like the subproblems of the L-shaped method）. Thus, I do not want to define the model in the solve_model(input_data) function. Instead, define the model before that, so that I don't need to define the model repeatedly every time I call the function, which also facilitates the hot start of the model (since the feasible domain doesn't change). Thus, I write the form (2).

 

with gp.Env() as env, gp.Model(env=env) as model: 

       # define model

 

def solve_model(input_data):      

       model.optimize() 

       # retrieve data from model

 

if __name__ == '__main__': 

       with mp.Pool() as pool: 

       pool.map(solve_model, [input_data1, input_data2, input_data3]

 

However, I found that form (2) is much less efficient than form (1), and the difference in computational time is very large. Form (2) is heavily blocked.

Why does this happen? and how can I solve this problem?

Thanks a lot!

Related to

• Parallelism

0

• 

  Jaromił Najman

  Gurobi Staff

  • August 09, 2022 16:15
  • Edited

  It looks like in your (2) form, 1 environment is used for every model solution. Since environments are not thread safe, this leads to blocking and possible errors during the optimization process.

  Thus, I do not want to define the model in the solve_model(input_data) function. Instead, define the model before that, so that I don't need to define the model repeatedly every time I call the function, which also facilitates the hot start of the model (since the feasible domain doesn't change)

  All Model objects have to be connected to a given environment. Thus, it is currently not possible to skip (or avoid) the "model building" part in your solve_model function.

  Best regards, 
  Jaromił

  0
• 

  Florian Götz

  • March 11, 2023 15:08

  

  

  

  Is this information still up to date?

  Best regards

  Thus, it is currently not possible to skip (or avoid) the "model building" part in your solve_model function.

  0
• 

  Jaromił Najman

  Gurobi Staff

  • March 13, 2023 08:39

  Is this information still up to date?

  Yes, this information is currently still correct.

  0

Please sign in to leave a comment.