Specifying threads to be utilized on a remote machine
Answered
I am trying to run an optimization problem on a HPC cluster. The machine I requested has 24 cores and 1 thread per core. I don't intend to utilize all 24 cores, so I requested the machine using Slurm Manager as follows:
srun -N 1 --ntasks-per-node=10 --mem-per-cpu=1gb -t 00:30:00 -p interactive --pty bash
This command helps me run my code interactively on 1 node, using only 10 of its cores (of the 24 cores it has).
Now, when I run my code, the Gurobi log shows that it is utilizing all 24 cores (same as threads in this case):
Thread count: 24 physical cores, 24 logical processors, using up to 24 threads
Because I only requested 10 cores, I assumed I didn't need to limit the threads by specifying the `Threads` parameter.
So my question is: Why is Gurobi trying to utilize all cores when I only requested a small number of them? Or does it simply display all the cores the machine has (irrespective of what I request), but in reality, it only utilizes what's actually requested through the slurm command (10 in this case)?
I am using Gurobi version 10.0.1. Thanks.
-
To ensure Gurobi only uses the 10 cores you requested, explicitly set the
Threadsparameter in your model to 10. This aligns Gurobi's usage with your Slurm allocation0 -
However, if I only make 10 cores available and Gurobi's default setting of 0 for Threads parameter imposes a limit of 32 cores (which are obviously not available), then it should automatically utilize only 10. Why is there a need to set `Threads` parameter?
0 -
Jaromił Najman, any help/clarifications on this are appreciated. Thanks :)
0 -
I think you're asking a SLURM question, not a Gurobi question. It looks like --ntasks-per-node limits the number of jobs on a node, not the number of threads in a job. To be sure, you should probably ask a SLURM expert.
0 -
The Gurobi process isn't aware Slurm is limiting the number of cores it can use. The Gurobi process sees the machine has 24 physical cores and 24 logical processors, then uses that information to decide how many threads to use. If the number of cores available to Gurobi is externally limited to a number different than what is reported by the machine, you should set the Threads parameter to reflect this.
1 -
Thanks Eli Towle for the clarification! I have some follow-up questions:
- If I set the Threads parameter to 10, I assume it sets an upper limit on how many threads/cores (if single-threaded) Gurobi can use. Is that right? Or does it force Gurobi to use 10?
- I have (say) 10 optimization problems that I am parallelizing on these 10 cores (one on each core). In this case, what do you recommend setting the Threads parameter to for each of these optimization models? Although each optimization problem utilizes one core, can Gurobi access the resources from other cores while they are occupied solving other optimization problems (without slowing down their performance)? If I simply set the Threads parameter to 10 for each model, will Gurobi figure out if it's actually efficient to utilize all threads or just stick to 1 thread (that's why question in point 1.), or maybe I should just set the Threads parameter to 1 for each problem? Again, the machine has more than 10 cores, so I would want to explicitly inform Gurobi about the available cores (which I limit using Slurm).
0 - If I set the Threads parameter to 10, I assume it sets an upper limit on how many threads/cores (if single-threaded) Gurobi can use. Is that right? Or does it force Gurobi to use 10?
-
If I set the Threads parameter to 10, I assume it sets an upper limit on how many threads/cores (if single-threaded) Gurobi can use. Is that right? Or does it force Gurobi to use 10?
Yes, 10 will be an upper bound on the number of threads Gurobi uses. There may be times when Gurobi does not use all 10 threads.
Although each optimization problem utilizes one core, can Gurobi access the resources from other cores while they are occupied solving other optimization problems (without slowing down their performance)? If I simply set the Threads parameter to 10 for each model, will Gurobi figure out if it's actually efficient to utilize all threads or just stick to 1 thread (that's why question in point 1.), or maybe I should just set the Threads parameter to 1 for each problem?
If you solve multiple models in parallel with Threads=10, each Gurobi process will use up to 10 threads. The individual Gurobi processes do not communicate with each other or look at what else is running on the machine to determine thread usage. In general, over-subscribing the machine by launching more threads than cores increases job throughput, though you should test this on your models.
what do you recommend setting the Threads parameter to for each of these optimization models?
I would test different combinations of Threads and number of parallel jobs to see what works best. For example:
- Solve 10 jobs in parallel at a time, each with Threads=10
- Solve 10 jobs in parallel at a time, each with Threads=5
- Solve 10 jobs in parallel at a time, each with Threads=1
- Solve 5 jobs in parallel at a time, each with Threads=2
- Solve 2 jobs in parallel at a time, each with Threads=5
1 -
Perfect, that makes things clearer! Thanks Eli Towle.
0
Please sign in to leave a comment.
Comments
8 comments