Parallelized solving of Gurobi models in Python

Hi all,

I want to solve multiple optimization models in parallel using Python's multiprocessing library and Gurobi. I have already learned, that the gurobipy model class is not pickable and thus cannot serve as input of the multiprocessing pool. Based on a community example (Fast creation of Gurobi models – Gurobi Help Center), I have created a function that stores all relevant model components as numpy arrays in a dictionary which I can then pass to the multiprocessing pool. This approach seems to work well as long as I do not make any further adjustments to the model within the multiprocessing function but it throws me errors as soon as I start adjusting the model.

The background of my question is that I want to iteratively solve multiple models in parallel whose objective function is changing in every iteration. The objective function is a quadratic expression which I need to rebuild on every iteration using the model.setObjective() function. Thus I neither need to find a way to pass this new quadratic objective function to the multiprocessing pool or I need to redefine the objective in the parallelized function itself.

Please see below a Minimum Working Example for the case that I would only want to change the RHS of the model which already throws me the error.

My questions would be

1) if there is a way to redefine the objective function of a gurobipy model within a multiprocessing pool or

2) if there is a way to pass some sort of description of the quadratic objective to the pool function in order to define it accordingly when rebuilding the model?

import gurobipy as gp
import numpy as np
from multiprocessing import Pool
import logging


def opt(input_data: dict):
    with gp.Env() as env, gp.Model(env=env) as model:
        logging.info('Rebuild model.')
        model.ModelSense = input_data['model_sense']
        model.addMVar((len(input_data['varnames']),), lb=input_data['lb'], ub=input_data['ub'],
                      obj=input_data['var_obj'], vtype=input_data['vtype'],
                      name=input_data['varnames'])
        model.addMConstr(A=input_data['A'], x=None, sense=input_data['con_sense'], b=input_data['rhs'],
                         name=input_data['connames'])
        logging.info('Update RHS of constraints.')
        for l in [1, 2, 3]:
            con = model.getConstrByName(f'con_{l}')
            con.setAttr(gp.GRB.Attr.RHS, 20.0)
        model.update()
        logging.info('Solve model.')
        model.optimize()

def build_model():
    logging.info('Build model.')
    m = gp.Model()
    x = m.addVars([id for id in [11, 22, 33]], vtype=gp.GRB.CONTINUOUS, lb=0.0, ub=[10, 20, 30],
                   obj=[1, 10, 100], name="con")
    m.addConstr(x[11] <= 10.0, f"con_{1}")
    m.addConstr(x[22] <= 50.0, f"con_{2}")
    m.addConstr(x[33] <= 100.0, f"con_{3}")
    m.addConstr(x[11] + x[22] + x[33] >= 50.0, f"con")
    m.ModelSense = gp.GRB.MINIMIZE
    m.update()
    return m

def get_model_data(model: gp.Model):
    model_data = dict()
    model_data['name'] = model.ModelName
    model_data['A'] = model.getA()
    model_data['model_sense'] = model.ModelSense
    model_data['con_sense'] = np.array(model.getAttr("Sense"))
    model_data['rhs'] = np.array(model.getAttr("rhs"))
    model_data['lb'] = np.array(model.getAttr("LB"))
    model_data['ub'] = np.array(model.getAttr("UB"))
    model_data['vtype'] = np.array(model.getAttr("Vtype"))
    model_data['var_obj'] = np.array(model.getAttr("Obj"))
    model_data['varnames'] = model.getAttr("VarName")
    model_data['connames'] = model.getAttr("ConstrName")
    return model_data


if __name__ == "__main__":

    models = [build_model(), build_model()]
    model_specs = [get_model_data(m) for m in models]

    with Pool(len(models)) as pool:
        pool_results = pool.map(opt, model_specs)

This is the error I get:

David Torres Sanchez

Gurobi Staff

February 09, 2024 12:50 Edited

The error is misleading. Your code doesn't work without the parallel loop (in series).

I recommend debugging that and freeing the models created in \(\texttt{build_model}\) before you enter the parallel part.
Particularly, I think the error is that you need to call \(\texttt{model.update()}\) before \(\texttt{model.getConstrByName}\).

Cheers,
David

February 09, 2024 12:46

Also, as a customer, you can open a ticket for faster 1-1 support.

Lene Grübler

February 14, 2024 10:34

Thanks! I was missing the model.update()

Parallelized solving of Gurobi models in Python

Comments

Didn't find what you were looking for?