If model construction consumes a significant portion of the overall solution process, determining where the bottlenecks are occurring is vital.
Finding bottlenecks
To find these possible bottlenecks in constructing a model, the first step is to determine the routines in which the time is spent. This can be accomplished in a number of ways, including:
- Third-party modeling tools: Switching to one of Gurobi's native APIs, e.g., the Python API from 3rd party wrappers that are designed for modeling purposes and may significantly slow down model construction
- Profiling: Using a profiler on your application, such as https://docs.spyder-ide.org/3/profiler.html for Python programs. Another example if you are using PyCharm would be Optimize your code using profilers | PyCharm (jetbrains.com). or First look at profiling tools - Visual Studio (Windows) | Microsoft Docs
- Timing: Adding timings to pinpoint which blocks of code are taking longer than expected. For example:
import time
start = time.time()
# code to time goes here
end = time.time()
print(start - end)
Once you've established where the bottlenecks are occurring, the following suggestions may be useful for designing more efficient code.
Inefficiencies in data slicing/queries
For Python programs, the most common reason for bottlenecks in model construction is from using inefficient data queries. This is especially true when using Pandas dataframes and methods such as iterrrows, loc, groupby, etc. as well as when making redundant queries to access the same data instead of storing the needed values in a smaller (temporary) data structure.
Usage of tupledicts
For Python, try using Model.addVars() to create a sparse tupledict of variables, then use the select(), sum(), and prod() methods to iterate over only the matching variables. This is illustrated in the netflow.py example.
Using a for-loop together with the Model.addConstr() / Model.addVar() function is slightly faster than addConstrs() and addVars(), especially if large numbers of objects are added. In order to retain convenient access to all objects, a variable dictionary needs to be constructed. Note that avoiding the retrieval of variables and constraints by name, such as Model.addConstrByName(), can also improve performance.
Efficiency of building expressions
For .NET and Python programs, it is more efficient to build a linear or quadratic expression by modifying an existing expression rather than repeatedly creating new expressions. For .NET, use the AddTerm()/AddTerms() methods instead of the overloaded operators. For Python, use the overloaded += or -= operators and/or add()/addTerms() rather than creating new expressions.
Matrix API
When used properly, the Python Matrix API can be significantly faster than the traditional Model.addConstr() process. When using the Python Matrix API, it is essential to use the matrix in order to model the whole model at once, i.e., state the constraints as \(Ax \leq b\). The matrix API is currently not designed to perform matrix/vector operations efficiently (although it is possible, see the Knowledge Base section on the Python Matrix API).
Additional remarks
- Version: If your application was developed using an older version of Gurobi that required frequent calls to Model.update(), remove those method calls and use the latest version of Gurobi.
- Memory: If your machine doesn't have adequate memory, some disk-swapping may occur, which can lead to slower performance. To monitor memory usage, use the Activity Monitor on a Mac, Task Manager on a Windows machine, or the top command on Linux.
- Helper functions: Sometimes, performance bottlenecks may be caused by user-defined helper functions.
- API: In some rare cases, switching to a lower-level language such as C, C++, or Java may also improve model construction time.