multiprocessing gurobi have errors using Process but no erros using Pool

Answered

zhen chen

February 25, 2025 19:20 Edited

• 
• 

I  have a linear programming problem that requiring changeing the right hand side of a constraint every time I solved the model. So I want to see if I can do it by parallel computing.

From several related posts in the community, I get to know that the gurobi environment is not thread safe and each subprocess should create its own environment.

However, my following codes run smoothly using Pool() but have errors when using Process().

Can someone help check the reason?

 

import multiprocessing
import numpy as np
from gurobipy import Model, GRB, Env
import time


# Create a new model
env = Env(params={"OutputFlag": 0})
m = Model(env=env)
m.params.OutputFlag = 0

# Create variables
x = m.addVar(
    vtype=GRB.BINARY, name="x"
)  # default bounds for continuous type is [0, infinite]
y = m.addVar(vtype=GRB.BINARY, name="y")
z = m.addVar(vtype=GRB.BINARY, name="z")

# Set objective
m.setObjective(x + y + 2 * z, GRB.MAXIMIZE)

# Add constraint: x + 2 y + 3 z <= 4
m.addConstr(x + 2 * y + 3 * z <= 4, "c0")

# Add constraint: x + y >= 1
m.addConstr(x + y >= 1, "c1")


# noinspection PyTypeChecker
def solve_pool(rhs_):
    m.update()
    m.setAttr("RHS", m.getConstrs()[0], rhs_)
    m.optimize()
    return m.ObjVal


def solve_process(queue_, rhs_):
    m.update()
    m.setAttr("RHS", m.getConstrs()[0], rhs_)
    m.optimize()
    queue_.put(m.ObjVal)


if __name__ == "__main__":
    N = 10
    rhs = np.arange(1, N + 1)

    # works well using pool
    time_start = time.time()
    with multiprocessing.Pool() as pool:
        result = pool.map(solve_pool, rhs)
    time_end = time.time()
    print(f"Time cost for parallel :{time_end - time_start:.4f}s\n")
    print(result)

    # using Process has some issue
    q = multiprocessing.Queue()
    processes = []
    for i in range(1, N):
        p = multiprocessing.Process(target=solve_process, args=(q, i))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
    result = [q.get() for _ in range(10)]
    print(result)

 

Related to

• Parallelism
• Multiple Scenarios

0

• 

  zhen chen

  February 26, 2025 10:36

  • 
  • 

  I find that if define gurobi models inside the target function, there will be no issue running the Process().

  0
• 

  Summer Purschke

  • Gurobi Staff

  February 26, 2025 23:51

  Hi Zhen, 

  Thank you for posting in the community forum. I believe I have a few hints that may help. 

  1. Have you read this article regarding using multiprocessing with Gurobi? It suggests that each process should create its own environment when using multiprocessing - you can do this using pattern below. This pattern automatically discards the model and environment upon leaving the with-block. 

  import multiprocessing as mp
  import gurobipy as gp
  
  def solve_model(input_data):
      with gp.Env() as env, gp.Model(env=env) as model:
          # define model
          model.optimize()
          # retrieve data from model
  
  if __name__ == '__main__':
      with mp.Pool() as pool:
          pool.map(solve_model, [input_data1, input_data2, input_data3])

  
  2. Additionally, I noticed a mismatch in the number of values when utilizing multiprocessing.Process()

  • for i in range(1,N): which is iterating from 1 to N-1
  • result =[q.get() for _ in range(10)] which is expecting exactly 10 items be added to the queue

  This discrepancy can cause the queue to wait indefinitely for the 10th expected item, and for your script to stall.
  
  I hope these suggestions prove beneficial for your model! 

  0
• 

  zhen chen

  February 27, 2025 12:58

  • 
  • 

  Thanks very much Summer for noting one bug in my codes.

  After reading several related posts and test some multiprocessing problem myself, I find that if wanting the multiprocess running successully, it is better to build each solving environmen for  each process. But it seems this way of multiprocessing does not have advantage compared with sequential computing which does not require building new models in each loop iteration and only need to change the coeffficients of the base model.

  From some posts I get to know that Gurobi now has a new feature multiscenario that is specifically to deal with different scenarios of coefficient values. But from some official replies, currently the mulriscenario is MIP and can't output the dual values of the constraint in each scenario.

  Is my understanding correct?

   

   

   

  0
• 

  Erik Halseth

  • Gurobi Staff

  February 28, 2025 19:48

  Hi Chen,

  We appreciate your response. I am a colleague of Summer’s, and wanted to address your question.

  Have you researched this?

  We recommend creating one model process per one environment for achieving parallelism.

  When trying to create multiple model processes in parallel within one Gurobi environment session, it could be dangerous because it impacts thread safety with respect to model optimization.

  In addition, within one Gurobi environment session, running sequentially very fast could often satisfy your use case of changing coefficients over time.

  Please let us know if you have any further questions.

  Warm thanks,

  Erik H.

  0
• 

  Zhen Chen

  March 02, 2025 19:33

  • 
  • 

  Thanks very much Erik for the answer. I am looking forwad for applying different new features of gurobi in the future.

  0

Please sign in to leave a comment.