Skip to main content

Gurobi uses less threads than available after restart

Answered
Lorenzo Moreschini
Edited
  • Gurobi-versary
  • Investigator
  • Conversationalist

I am writing a program than solves many MIP models simultaneously, equally distributing the number of available threads (virtual cores) among all models. When a model completes its threads are reassigned to the remaining running models.

In what follows I will assume to start solving 5 models simultaneously with a total of 30 available threads.
Since the number of threads for a model can only be set when the optimization is not running I do as follows:
 a) Create a shared array of integers called `assignedThreads` with one slot for each model (numbered from 0 to 4).
 b) For each model, create a callback with a reference to the shared array `assignedThreads`.
 c) Simultaneously start each model with `model.optimize()`.
 d) When a model completes I reassign its threads (from an external thread) increasing the values in some slots of the `assignedThreads` array (only for models that are still running).
 e) Inside each callback, each model checks if the number of threads assigned to it is greater than the number he is actually using; if so, I stop the optimization cycle using `model.terminate()`, I set the new number of threads with `model.set(GRB.IntParam.Threads, ...)` (from an external thread) and then continue the optimization with `model.optimize()`. (I'm using Java)

The issue is that, although in the log file of each model I see that the parameter holding the number of available threads is correctly set and increases over time, e.g.

    Set parameter Threads to value 6
    ...
    Set parameter Threads to value 8
    ...
    Set parameter Threads to value 10
    ...
    Set parameter Threads to value 15
    ...
    Set parameter Threads to value 30

when I call model.terminate() Gurobi always prints

    Thread count was 6 (of 32 available processors)
    ...
    Thread count was 6 (of 32 available processors)
    ...
    Thread count was 6 (of 32 available processors)
    ...
    Thread count was 6 (of 32 available processors)
    ...
    Thread count was 6 (of 32 available processors)

where 6 is the initial number of threads when all models were started for the first time.
Is this a misprint or does it mean that each model never actually use the extra threads that it receives over time?

Here you can find the complete log file of the model that completes last in an execution with 5 models and 30 threads: https://drive.google.com/file/d/12niEV0KWMYAWNs7jw3ws5uqlX1xnqX2-/view?usp=sharing

 
0

Comments

9 comments

Date Votes
  • Official comment
    Simranjit Kaur
    • Gurobi Staff
    This post is more than three years old. Some information may not be up to date. For current information, please check the Gurobi Documentation or Knowledge Base. If you need more help, please create a new post in the community forum, or try Gurobot, our chatbot interface offering instant, expert-level support.
  • Jaromił Najman
    • Gurobi Staff

    Hi Lorenzo,

    This behavior is not expected and we are investigating.

    Thank you for reporting this.

    Best regards, 
    Jaromił

    0
  • Jaromił Najman
    • Gurobi Staff
    Edited

    Hi Lorenzo,

    We had some internal discussions about this behavior and it is indeed expected behavior. The reason is that data structures are created for 6 Threads in your first run and when you continue with more threads, these data structures currently cannot be modified for X>6 Threads. Note that 6 is just an arbitrary example number here.

    We added this to our feature request list and will possibly add it in a future release.

    Once again thank you for reporting this.

    Best regards, 
    Jaromił

    0
  • Lorenzo Moreschini
    • Gurobi-versary
    • Investigator
    • Conversationalist

    Hi Jaromił,

    thank you for your quick response. I have a few questions:

     1) In your example you say that "these data structures currently cannot be modified for X>6 Threads", does it mean that I can decrease the number of threads and in this case the new value will be used?
     2) Can you provide an estimate about how long will it take to implement this feature? (something like "not before/within a few weeks/months/years"). It would be really helpful for my purposes.
     3) Is there a way to save the workflow described in my first post? The worst way to patch it would be to rebuild each running model (with performance issues) when new threads become available, set the new number of threads on the clean model and also set to the new model the last incumbent solution found before calling `terminate()`. To avoid building a new model each time, is there a way to reset the internal state of a model for which the `optimize()` function has already been called once? If so, calling `optimize()` after setting the incumbent solution would be "more or less" the same as continuing the optimization on the previous model (hopefully)?

    Finally, I think that the current limitation on setting the number of threads should be clearly stated in the documentation (and model.set(GRB.IntParam.Threads) may even raise an error if called after `model.optimize()`).

    Thank you!

    0
  • Jaromił Najman
    • Gurobi Staff

    HI Lorenzo,

    1) In your example you say that "these data structures currently cannot be modified for X>6 Threads", does it mean that I can decrease the number of threads and in this case the new value will be used?

    No, currently this will not work either.

    2) Can you provide an estimate about how long will it take to implement this feature? (something like "not before/within a few weeks/months/years"). It would be really helpful for my purposes.

    This is hard to say because we currently cannot estimate how difficult it is to implement this feature. If we decide to implement this feature, I would say that it will come with a release some time next year.

    3) Is there a way to save the workflow described in my first post? The worst way to patch it would be to rebuild each running model (with performance issues) when new threads become available, set the new number of threads on the clean model and also set to the new model the last incumbent solution found before calling `terminate()`. To avoid building a new model each time, is there a way to reset the internal state of a model for which the `optimize()` function has already been called once? If so, calling `optimize()` after setting the incumbent solution would be "more or less" the same as continuing the optimization on the previous model (hopefully)?

    You could extract the so far found best feasible solution via reading and storing the X attribute of each variable. You can then use the reset method to reset the current model state. Then, you can set the new Threads parameter, provide the stored feasible solution as MIP start and run optimize().

    Finally, I think that the current limitation on setting the number of threads should be clearly stated in the documentation (and model.set(GRB.IntParam.Threads) may even raise an error if called after `model.optimize()`).

    Yes, I agree. We will update the documentation.

    Best regards, 
    Jaromił

     

    0
  • Lorenzo Moreschini
    • Gurobi-versary
    • Investigator
    • Conversationalist

    Hi Jaromił,

    now that Gurobi 10.0 has been released, have you discussed this issue more deeply?
    Did you reach a decision whether or not you are going to implement this feature?

    0
  • Jaromił Najman
    • Gurobi Staff

    Hi Lorenzo,

    The change would have been too big to make it into v10. We certainly have this on our radar and plan on improving the behavior in the next major release. Please note that I cannot promise anything but you can be sure that we are aware of this issue and discussed this in detail in the dev team.

    Best regards, 
    Jaromił

    0
  • Lorenzo Moreschini
    • Gurobi-versary
    • Investigator
    • Conversationalist

    Hi Jaromił,
    reading the release notes of version 11.0 it seems that this feature has been added, right?

    Change the Threads parameter when resuming optimization If an optimization is interrupted (e.g., due to a time limit), the user can now modify the Threads parameter in order to change the number of threads to be used when the solving process is resumed with another call to optimize.

    So, if at any time during the optimization process I know that more computational resources are available, I can do (pseudocode ahead)

    model.terminate()      # in callback
    model.set(Threads, N)  # in main process
    model.optimize()       # in main process

    and resume the optimization from where I left it, am I correct?

    However, the primary reason I am writing this post is to get a clarification about these lines also contained in the release notes:

    The primary use case for copying models is when performing optimization on multiple, related versions of the same problem on multiple threads. Multi-threading within an environment is not supported. 

    Here the assumption is the the user wants to build a model once, and then solve two or more variations thereof in parallel on different thread; since Gurobi environments are not thread-safe we need to create an environment for each variation of the model and build that variation inside it (and now Gurobi provides a handy way to copy the core model rather than rebuild it). Am I missing something about these lines that can affect my use case (which does not involve multiple models)?

    0
  • Jaromił Najman
    • Gurobi Staff

    Hi Lorenzo,

    and resume the optimization from where I left it, am I correct?

    Yes, you are correct.

    Am I missing something about these lines that can affect my use case (which does not involve multiple models)?

    No, you are not missing anything. This should not affect your use case.

    Thank you once again for reporting this issue.

    Best regards, 
    Jaromił

    0

Post is closed for comments.

Didn't find what you were looking for?

New post