Skip to main content

can't pickle pycapsule object multiprocess

Awaiting user input
Reza Mirjalili
Gurobi-versary
First Question

I am trying to implement the branch and price algorithm in python via Gurobi and as you know, in every node of the search tree, the linear relaxation of the master problem (MP) with additional branching constraints must be solved.  

To make the process faster, I am using the Multiprocess library, and I solve multiple nodes in parallel. There are two ways to do that:

  1. At node (i), build the original model from scratch and add the additional cuts/variables which the node demands. This way is working but building the model from scratch is a computational burden. 
  2. At node (i), copy the model from the parent node (p), and add additional cuts/variables. This way is not working, and I received the following error:
"can't pickle pycapsule object multiprocess"

Here is the function solve_dual(II), which will be solved in each node. 

def solve_dual(II):
    mdl_o,new_B,SS,num_node,ub = II
    mdl = mdl_o.copy()
    S = {i:SS[i] for i in SS}
    # add branch as new constraints to primal or a new variable to dual
    B0,B1,D0,D1 = [],[],[],[]
    lm,bt,pi,dl,gm,te,vte,ep,om,mu,ro,sg,ph = Var_name(mdl.getVars())
    for ind,rhs,e in new_B:
        if ind=='xl':
            if rhs ==1:#sg
                D1.append(e)
                sg[e] =mdl.addVars([e],obj=1,vtype =GRB.CONTINUOUS,name='sg')[e]
            else:#ph
                ph[e] =mdl.addVars([e],obj=0,vtype =GRB.CONTINUOUS,name='ph')[e]
                D0.append(e)
        elif ind=='x':#mu
            if rhs ==1:
                B1.append(e)
                mu[e] =mdl.addVars([e],obj=1,vtype =GRB.CONTINUOUS,name='mu')[e]
            else:#ro
                ro[e] =mdl.addVars([e],obj=0,vtype =GRB.CONTINUOUS,name='ro')[e]
                B0.append(e)
    mdl.update()

    ct = {f_cons_name(ctr.ConstrName):ctr for ctr in mdl.getConstrs() if 'f' in ctr.ConstrName }
    # update the current constraints by updating new vars coefficient
    for l,j,r,d in ct:
        edg = edge_dec([l]+S[j]+[r])
        for ind,rhs,e in new_B:
            if e[:2] in edg:
                if ind=='xl' and e[-1]==d:
                    if rhs ==1:#sg
                        mdl.chgCoeff(ct[l,j,r,d], sg[e], 1)
                    else:#ph
                        mdl.chgCoeff(ct[l,j,r,d], ph[e], -1)

                elif ind=='x':#mu
                    if rhs ==1:#mu
                        mdl.chgCoeff(ct[l,j,r,d], mu[e], 1)
                    else:#ro
                        mdl.chgCoeff(ct[l,j,r,d], ro[e], -1)
    mdl.update()
    mdl.optimize()
    fnd_sq = False
    if mdl.status!=GRB.OPTIMAL:
        return mdl,np.inf,S,fnd_sq,([],'',0),(fs,Fs),'i',ub
    if mdl.objVal>=ub:
        return mdl,mdl.objVal,S,fnd_sq,([],'',0),(fs,Fs),'c',ub
    
   

Related to

0

Comments

5 comments

Date Votes
  • Jaromił Najman
    Gurobi Staff Gurobi Staff

    At node (i), build the original model from scratch and add the additional cuts/variables which the node demands. This way is working but building the model from scratch is a computational burden. 

    This is the correct way to go here because the gurobipy module is not thread safe. This means that each model has to have its own environment within a parallel run and you cannot copy models from one environment to another.

    At node (i), copy the model from the parent node (p), and add additional cuts/variables. This way is not working, and I received the following error:

    I assume, you are trying to pickle a model to copy it over. A model object is not picklable, hence the error. Additionally, as mentioned above, copying models from one environment to another is currently not supported.

    Best regards, 
    Jaromił

    2
  • Florian Götz
    • Edited
    Gurobi-versary
    Conversationalist
    First Question

    Hi Jaromił, I'm struggling with the same problem as the post initiator. Silke Horn states on the following website that one can solve different optimization problems in parallel with multiprocessing if you stick with the given code example: https://support.gurobi.com/hc/en-us/articles/360043111231-How-do-I-use-multiprocessing-in-Python-with-Gurobi- So far this implementation works great. But since I want to measure the idealized times of both problems being solved in parallel (see below) and I can/should not make any assumptions about how the tasks are delegated to the cpu cores by the multiprocessing pool, Silkes example is not so useful to me.

    Is there no other way to pass a "solve_model" function to the pool that just solves the optimization problems with the modification that the problems do not need to be defined in "solve_model" itself?

    At the moment I'm trying to produce a minimal example that accomplishes just that:

    import numpy as np
    import gurobipy as gp
    from gurobipy import GRB
    import multiprocessing as mp
    import time

    #First model:
    my_model_1 = gp.Model("m1")
    x1 = my_model_1.addVar(name="x1")
    constr1 = my_model_1.addConstr(x1 <= 17)
    my_model_1.setObjective(2 * x1, GRB.MAXIMIZE)

    #Second model:
    my_model_2 = gp.Model("m2")
    x2 = my_model_2.addVar(name="x2")
    constr2 = my_model_2.addConstr(x2 <= 17)
    my_model_2.setObjective(3 * x2 , GRB.MAXIMIZE)

    def solve(m):
        m.optimize()
        return m

    all_models=[my_model_1,my_model_2]
        
    with mp.Pool() as pool:
        start=time.time()
        result=pool.map(solve,[model for model in all_models]) #Here I get the same error message: "cannot pickle 'PyCapsule' object".
        end=time.time()
        idealized_time=end-start


    0
  • Jaromił Najman
    Gurobi Staff Gurobi Staff

    Hi Florian,

    Is there no other way to pass a "solve_model" function to the pool that just solves the optimization problems with the modification that the problems do not need to be defined in "solve_model" itself?

    Unfortunately, it is currently not possible.

    You could try generating model files (MPS for example) via the write method and then reading them in each solve_model call instead of constructing the models from scratch. You could also measure the time needed to construct/read the model and substract that from the overall time for a solve_model call.

    But since I want to measure the idealized times of both problems being solved in parallel (see below) and I can/should not make any assumptions about how the tasks are delegated to the cpu cores by the multiprocessing pool, Silkes example is not so useful to me.

    If the models are of similar size and your run enough trials for your tests then you should be safe to just ignore the model construction/reading time and still get a valid statement for your parllelization experiments.

    Best regards, 
    Jaromił

    0
  • Florian Götz
    • Edited
    Gurobi-versary
    Conversationalist
    First Question

    Hi Jaromił,

    You could try generating model files (MPS for example) via the write method and then reading them in each solve_model call instead of constructing the models from scratch. 

    Thanks for the insight. I have implemented your idea as indicated. However, it is interesting that I get less accurate results when I proceed as follows:

    Instead of saving the finished but unsolved model as an MPS file, I could save it in a list of "ready to solve" models. Thus no reading/writing to the hard drive would be necessary. Then I pass an index to the solve method that points to the corresponding model in the list. But here I lose accuracy of the solutions.

    Could you please explain to me why this happens?

    Best regards,

    Florian

     

    0
  • Jaromił Najman
    Gurobi Staff Gurobi Staff
    • Edited

    Hi Florian,

    Could you please share a minimal reproducible example of the issue you describe?

    Best regards, 
    Jaromił

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post