Python Multiprocessing process LinExpr objects
Answered
def process_expr(expr: LinExpr):
c = 0
remove_vars = []
for i in range(expr.size()):
var = expr.getVar(i)
if var.VarName.startswith("A"):
c += expr.getCoeff(i) * my_dict[var.VarName]
else: # remove other variables
remove_vars.append(var)
for var in remove_vars:
expr.remove(var)
For a single LinExpr, I want to run the function `process_expr` above. But for a list of LinExprs(maybe hundreds of thousands), I want to use the python multiprocessing module to accelerate this process. But I got a "cannot pickle .... error". How can I let this work?
-
Could you please post a minimal reproducible example together with the full error message you get?
Maybe the Knowledge Base article How do I use multiprocessing in Python with Gurobi? can already help out.
Best regards,
Jaromił0 -
from gurobipy import *
from joblib import Parallel, parallel_backend, delayed
import multiprocessing as mp
def process_expr(expr: LinExpr):
c = 0
remove_vars = []
for i in range(expr.size()):
var = expr.getVar(i)
if var.VarName.startswith("A"):
c += expr.getCoeff(i) * 1
else: # remove other variables
remove_vars.append(var)
for var in remove_vars:
expr.remove(var)
if __name__ == "__main__":
m = Model()
x = m.addVar()
y = m.addVar()
z = m.addVar()
e1 = x + y + z - 1
e2 = 2 * x - y + z - 2
expr_list = [e1, e2]
# method 1 with joblib doesn't work
with parallel_backend("loky", n_jobs=2):
Parallel()(delayed(process_expr)(e) for e in expr_list)
# _pickle.PicklingError: Could not pickle the task to send it to the workers.
# method 2 with multiprocessing doesn't work
with mp.Pool(2) as pool:
pool.map(process_expr, expr_list)
# _pickle.PicklingError: Can't pickle <function process_expr at 0x000001E8DAC14D30>: attribute lookup process_expr on __main__ failedBoth methods don't work. The error is in the comment.
0 -
I had to double check this one. LinExpr objects and other Gurobi modeling objects are not pickleable. Thus, it is not possible to work on multiple LinExpr objects in parallel. This is also not a good approach when constructing model objects. Multiprocessing in Python can be used for isolated, independent environments.
Is your model construction slow? If yes, could you share a minimal working example of slow model construction? The Knowledge base article How do I improve the time to build my model? might be helpful.
Best regards,
Jaromił0 -
My task is to modify some of the constraints(for hundreds of thousands of them) already in the model, not to construct the whole model.
I need to remove the constraints and remove some of the vars in the lhs, modify the constant and add the constraints back. However, this is too slow to sequentially process the expressions like the code above( without parallelism).
In this situation, I have no data to reconstruct the constraints from scratch, I must use getCoeff and getVar to access the data or other ways but the only thing I have is a gurobi model and the name of the constraints.
Is your model construction slow? If yes, could you share a minimal working example of slow model construction? The Knowledge base article How do I improve the time to build my model? might be helpful.
Sorry, I can't give you a minimal example, since for few number of constraints like 100, It takes only 1s. But I have like 50k constraints, It will take 500s to finish the task. If I can modify the expressions in parallel, I think the time will be less than 60s.
0 -
An example with only 100 constraints would already be enough, since it should not take 1 second to modify 100 constraints. If you like, you can share a minimal working example with 100 constraints as described in Posting to the Community Forum.
Best regards,
Jaromił0 -
from gurobipy import *
import numpy as np
import time
def process_expr(expr: LinExpr):
c = 0
remove_vars = []
for i in range(expr.size()):
var = expr.getVar(i)
if var.VarName.startswith("x"):
c += expr.getCoeff(i) * 1
else: # remove other variables
remove_vars.append(var)
for var in remove_vars:
expr.remove(var)
new_expr = expr - c
return new_expr
if __name__ == "__main__":
m = Model()
x = m.addVars(1000, name="x").values()
y = m.addVars(500, name="y").values()
# the model has other constraints, but I eliminate them for simplicity
# add all constraints need to be modified
n_constrs = 100 # change the number to 50000, it will be very slow
for i in range(n_constrs):
x_coeffs = np.random.randint(-5, 6, size=len(x))
y_coeffs = np.random.randint(-5, 6, size=len(y))
const = np.random.randint(100)
m.addConstr(LinExpr(zip(x_coeffs, x)) + LinExpr(zip(y_coeffs, y)) + const >= 0, name="constr[{0}]".format(i))
m.update()
# m.optimize()
# I need to modify the constraints based on the solution
# here I omit the optimization too
# =========================================================================
# My Task here
start_time = time.time()
# get lhs of all constraints I need
expr_list = []
for c in m.getConstrs():
if c.ConstrName.startswith("constr"):
expr_list.append(m.getRow(c))
# My task: remove all y vars and set new constant for each constraint
# I want to speed up the following expression process,
# since it's more about data processing and not related to Constr or LinExpr, but I can't avoid using LinExpr
# It's too slow
new_expr_list = []
for expr in expr_list:
new_expr_list.append(process_expr(expr))
# add new constraint back
m.addConstrs((new_expr_list[i] >= 0 for i in range(len(new_expr_list))), name="new_constr")
elapsed_time = time.time() - start_time
print(f"total elapsed time: {elapsed_time:.2f}s.")
# =========================================================================For this example change n_constrs from 100 to 50000 will be very time consuming.
0 -
You are removing all non-x variables in a \(\texttt{for}\)-loop 1 by 1. You can remove them all at once
expr.remove(remove_vars)
# instead of
# for var in remove_vars:
# expr.remove(var)This already provides a huge speed up for n_constr=100, it brings it from ~10 seconds to 0.4s.
For n_constr = 50000, already the constraint creation takes quite some time. This is because your constraints are very dense. You should consider introducing free auxiliary variables for the
zx = m.addVar(lb=-GRB.INFINITY, name="x_z%d"%(i))
zy = m.addVar(lb=-GRB.INFINITY, name="y_z%d"%(i))
m.addConstr(zx == LinExpr(zip(x_coeffs, x)))
m.addConstr(zy == LinExpr(zip(y_coeffs, y)))
m.addConstr(zx + zy + const >= 0, name="constr[{0}]".format(i))parts. This way, you will only have to remove 1 variable from each constraint to remove the \(y\) variables from a constraint.
for i in range(expr.size()):
var = expr.getVar(i)
if var.VarName.startswith("x"):
c += expr.getCoeff(i) * 1
else: # remove other variables
remove_vars.append(var)
expr.remove(remove_vars)For n_constr=10000, the constraint construction takes ~12s and the modification part takes only 0.5s on my notebook.
Best regards,
Jaromił0 -
You are removing all non-x variables in a for-loop 1 by 1. You can remove them all at once
Your advice seems not working
0 -
You are right, sorry for that. I confused the LinExpr.remove method with the Model.remove method.
In your case, it is actually way faster to construct a new expression instead of removing variables one by one.
def process_expr(expr: LinExpr):
c = 0
remove_vars = []
coeffs = []
xvars = []
for i in range(expr.size()):
var = expr.getVar(i)
if var.VarName.startswith("x"):
c += expr.getCoeff(i) * 1
# save variable of interest and its coefficient
coeffs.append(expr.getCoeff(i))
xvars.append(var)
# use LinExpr constructor to construct the expression fast
expr = LinExpr(coeffs, xvars)
#for var in remove_vars:
# expr.remove(var)
new_expr = expr - c
return new_expr0 -
Sounds great, thanks.
0
Please sign in to leave a comment.
Comments
10 comments