multiprocessing, model updating, iterative process
Answered
Hi,
I am working on solving and updating an optimization model iteratively via Gurobi-Python. Namely, first I generate an optimization model, and then I repeatedly solve and update the model (e.g., add variables or constraints). Please note that as the optimization model decomposes into several smaller subproblems, the solution of the optimization model in each iteration can be obtained by solving the smaller problems in parallel.
I have seen the solution suggested on this page:
https://support.gurobi.com/hc/en-us/articles/360043111231-How-do-I-use-multiprocessing-in-Python-with-Gurobi-
The issue with this solution is that the optimization model needs to be completely regenerated in each iteration which is not time efficient as the model size grows with iterations and regenerating the model consumes time.
Any suggestion or help is appreciated.
best regards
-
Hi,
You could have a copy of the original model which you would have to construct once in the beginning and then extend this one, i.e., add variables and constraints, instead or re-generating it from scratch in each iteration.
Best regards,
Jaromił1 -
Hi Jaromil,
Thanks for the reply. Please note that P1 and P2 need to be solved alternately. Namely, I need to solve P1, terminate the process of P1 (to release the CPU resources), and pass the result to P2 and solve P2. In the next iteration when coming back to P1, how new process can have access to the models of P1?
best,
Ghafour
0 -
Hi Ghafour,
If P2 has to wait for P1 anyway, then I don't see a reason for using multiprocessing. You could then just solve all problems sequentially. If there is a good reason for multiprocessing then please provide more details on your processing. What is running in parallel and what is not?
Best regards,
Jaromił0 -
Sorry for not being clear.
Yes correct. P2 has to wait for P1. The thing is that P1 itself is consists of several small problems that can be solved in parallel. The same thing for P2.
Thanks again,
Ghafour
0 -
Hi Ghafour,
Assuming that you have 4 cores to use. Core 1 could hold an instance of P1 and P2. You then split P1 into 4 subproblems and let each of the 4 cores solve a subproblem. Note that Core 1 still holds an instance of P1 and P2 while solving one of the four subproblems. When all cores finish solving their subproblems, you can dispose of the 4 subproblems. Core 1 then updates P1 and proceeds with P2 in a similar fashion. Would this work?
Best regards,
Jaromił0 -
Hi again,
Thanks for the reply. Yes it should work. I think it is not possible to share a Gurobi model among processors, right?
best,
Ghafour
0 -
Hi Ghafour,
I think it is not possible to share a Gurobi model among processors, right?
Gurobi is not thread safe, thus it is best to pass the required information from one process to another instead of sharing one model/environment.
Best regards,
Jaromił0 -
Dear Jaromił,
i have a similar problem as Ghafour, i.e., many small LP's run in parallel using Python's multiprocessing library. My implementation follows the tutorial posted by Silke. It works fine on my laptop but when i move the problem to the university server that has a Gurobi license through a token server, it often fails. I assume because the token server will be pinged for each small lp i am solving and probably the response time will be occasionally too slow which might cause multiprocessing to fail. Any ideas how to fix that? Is it safe to not use a separate environment for each iteration (small lp)?
Thank you
C.
0 -
Dear C.,
Could you elaborate on the failure you see? What is the exact error message? The response time of the license server should not be a problem, do you limit the number of Threads for each process to ensure that they don't interfere with each other?
Is it safe to not use a separate environment for each iteration (small lp)?
No, it is best to always create a new environment for each process.
Best regards,
Jaromił0 -
Thank you for your reply, Jaromił. Yes, i use 1 thread per process. In the meantime, i think i found the issue: i had requested more licenses than available because i was running too many jobs at once which resulted in an error message from the license server.
Thank you.
C.
0
Please sign in to leave a comment.
Comments
10 comments