Pyomo and DistributedMIP
Answered
I am using my universities Slurm cluster to solve a single pyomo model (python code) through gurobi. I am trying to make gurobi solve the problem using multiple CPUs from multiple nodes through sbatch. I am assigning 4 nodes, each containing 40 cores ( 160 cpus 360 logical threads). I set the gurobi parameter multi-threading to 160. But when I monitor the submitted job, I find that all 160 gurobi threads are being used on a single node. How can I distribute the work across multiple nodes in my case?
-
Hi Ahmed,
Is your university using a cluster manager? I'm guessing not if using slurm. The manager is required for Distributed MIP.
Please see the remote services manual and let us know if you have further questions.
Gurobi Remote Services Manual - Distributed Algorithms- Riley
0 -
Hi Riley,
I am the analyst for the cluster Ahmed is trying to run Gurobi on. I have checked the documentation and after installing the remote services I can start a worker using "grb_rs --worker". Then I have to ssh to the second node and start another remote service there "grb_rs --join ...". Is this how it is supposed to work with the Academic license?
Does the Academic license include the distributed license as well? Can I use the Compute Server with it?
If I arrange multiple workers on different compute nodes, will they be able to speed up the same single optimization problem? Or, two nodes will only be able to run two problems at the same time?
Thank you.
--- Dmitri.
0 -
Hi Dmitri,
I have checked the documentation and after installing the remote services I can start a worker using "grb_rs --worker". Then I have to ssh to the second node and start another remote service there "grb_rs --join ...". Is this how it is supposed to work with the Academic license?
This sounds right, although I would encourage you to use the config file instead (grb_rs.cnf). You can run
grb_rs properties
to see all available config options. Likewise I would use grb_rsm.cnf for the cluster manager.
Does the Academic license include the distributed license as well? Can I use the Compute Server with it?
We have many different license types, some can be used with Compute Servers, some can be used in distributed mode. Each license type can be commercial or academic, with the only difference being Academic = free. You can find out more in the Academic and Research Licenses section of our Knowledge Base. I'm not sure what licenses are currently issued to the institution but if you submit a request through the Gurobi Help Center then our licensing team will be able to assist.
If I arrange multiple workers on different compute nodes, will they be able to speed up the same single optimization problem? Or, two nodes will only be able to run two problems at the same time?
You will definitely be able to run multiple problems on multiple nodes. Whether you can run a single problem across multiple nodes depends on whether the license has distributed computation enabled. Whether multiple nodes can speed up the optimization really depends on the problem. It is likely to help if your problem is requiring a huge amount of nodes in the branch and bound tree. Parallelizing the solve incurs overhead and at some point the overhead outweighs the benefit - for some problems, 2 threads is worse than 1 thread! On a single machine we do not often see benefit in using more than 32 threads, and even above 16 threads can be hit and miss - again it depends on the nature of the problem.
- Riley
0
Please sign in to leave a comment.
Comments
3 comments