Skip to main content

Setting Threads parameter does not work as expected

Awaiting user input
Michel Weitbrecht
  • Edited
Gurobi-versary
First Comment
First Question

Hi,

I use Gurobi 9.1.2 to solve a MIP problem and have a powerful machine (96 threads) at my disposal, but I want to execute many instances at the same time, so I chose to use docker with a CPU limit of 400% (i.e. 4 threads) to solve 24 instances at the same time. The CPU limit works generally, but still leads to a high load, because every instance spawned many threads that couldn't run at 100%.

Therefore, I set the Threads parameter to 4, but Gurobi still spawns more processes than that. The log states that it actually used 32 threads:

Thread count: 48 physical cores, 96 logical processors, using up to 4 threads

[...]

Thread count was 32 (of 96 available processors)

 

How can I get Gurobi to only use the 4 threads I specified using the parameter?

 

Thanks in advance

Michel

0

Comments

7 comments

Date Votes
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    Hi Michel,

    It seems that this is somehow related to the docker environment. How did you limit the number of CPUs within a container?

    There are a few docker parameters that can do this, see here. Have you tried to assign specific cores to each container (--cpuset-cpus), to avoid that some of the cores are used by multiple containers or that some container gets dynamically assigned different cores throughout one particular Gurobi solve?

    What is your docker setup, i.e., host system, Docker (Desktop) version?

    Best regards,
    Mario

    0
  • Michel Weitbrecht
    Gurobi-versary
    First Comment
    First Question

    Hi Mario,

    thank you for your response!

    I'm using Docker 19.03.6 with docker-compose 1.29.2. I limit the CPUs using docker-compose with the key deploy.resources.limits.cpu (https://github.com/compose-spec/compose-spec/blob/master/deploy.md#cpus). I didn't specify the actual CPUs to use, I don't think this is possible for current docker-compose versions without creating a swarm. But I'm confident this isn't a Docker issue, I just wanted to explain why I specified the Gurobi Threads parameter in the first place (because the Docker limitation wasn't enough).

    I can replicate the issue on my local machine without Docker. Locally I have four cores and I set Threads=2 for testing:

    model.set(GRB_IntParam_Threads, 2);

    I uploaded the resulting log file here: https://pastebin.com/Z24eWwiT

    In line four, Gurobi correctly claims to be using two threads: "Thread count: 4 physical cores, 4 logical processors, using up to 2 threads".

    But in line 86 the log states that actually four threads were used: "Thread count was 4 (of 4 available processors)".

    I can corraborate this when I observe the process in htop - it actually spawns and utilizes four threads.

    Thanks and best regards,
    Michel

     

    0
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    Hi Michel,

    This is interesting, I cannot reproduce your issue on my side, both with Gurobi 9.1.2 and 9.5.0, the thread count message at the end states the number of threads that has been set with parameter Threads, as it should be.

    Can you please tell me more about your system (Linux distribution, version, CPU model)?

    Best regards,
    Mario

    0
  • Michel Weitbrecht
    Gurobi-versary
    First Comment
    First Question

    Hi Mario,

    I'm glad that this is not intended behavior :D

    Locally I'm using NixOS 21.11 with Gurobi 9.1.2 as packaged here in nixpkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/science/math/gurobi/default.nix

    My local CPU is a Intel i5-6440HQ, full lscpu output here: https://pastebin.com/Q09vMfVt
    My server has Ubuntu 18.04.5 with two AMD EPYC 7451 CPUs, full lscpu output here: https://pastebin.com/xLvPRNjV

    The docker image I use on the server is based on debian:buster-slim - I uploaded a version here with confidential repositories etc. removed: https://pastebin.com/GYawJbVr (still, all packages I use will be installed with this). Maybe you can reproduce it with this image?

    I don't know whether that's relevant but I use a Tokenserver license for this.

    If there is any other useful information I can provide, just tell me what you need.

    Best regards,
    Michel

    0
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    Hi Michel,

    Thanks, I will try to reproduce it with your information!

    Can you try to run the Gurobi command line on your local NixOS with an included model, just to cross out the C++ API:

    gurobi_cl Threads=2 gurobi912/linux64/examples/data/glass4.mps

    (I am not sure where NixOS installs Gurobi and whether the environment variables are set correctly.)

    I did not try Gurobi on NixOS yet, but it should run as intended on Ubuntu 18.04 with AMD EPYC CPUs. Can you try to run your model natively on your Ubuntu server (outside docker)?

    Best regards,
    Mario

    0
  • Michel Weitbrecht
    Gurobi-versary
    First Comment
    First Question

    Hi Mario,

    thank you, the examples work as intended with two threads on both machines: https://pastebin.com/kgbSF38B / https://pastebin.com/XqnLALtR

    NixOS sets up the environment correctly, I just had to download the examples separately as they are not shipped normally.

    So maybe the fault has to do with the C++ API?

    Best regards,
    Michel

    0
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    Hi Michel,

    Hmm, it seems to be related to the way you are using Gurobi in your C++ application. Could you try to build a minimal working example based on your application and post the source code here?
    You could even read the model file glass4.mps with GRBModel::read() instead of building your own one.
    Do you set any other parameters for your model or in the Gurobi environment?

    Best regards,
    Mario

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post