There are many things that we may wish to make comparisons between, including:
- alternate formulations
- potential parameter settings
- different machines and hardware
- competing solvers (including different versions of a solver)
And while any results we collect will be based on model instances that currently exist, we are actually interested in understanding how the solver will perform on similar model instances in the future. However, as discussed in What is performance variability?, a small change in a model could result in very different performance by solver, and we need to account for this. This gives rise to a fundamental rule for comparison:
When making comparisons between competing alternatives it is important to avoid the common mistake of using a single optimization per alternative to make conclusions.
In the absence of many problem instances for the same model, we should repeat optimizations with different values of Seed. The value of Seed influences random processes in Gurobi, which in turn affect the solution path, which in turn affects the results.
The resulting performance variance allows us to compare distributions of metrics against each other, and relevant characteristics such as mean, variance, and skew. Box plots tend to be very useful in making quick visual comparisons, but of course, statistical methods can also be applied.
How many seeds should I use?
The answer is almost always "more than one," and in general, "the more, the better." An exact number will depend on the distributions that are being compared, and this is not something that is specific to solver results. It is both dependent on the variance observed in the resulting distributions and perhaps other statistics such as median, mean and maximum.
This, of course, must be tempered with the cost of solving your models. If your model solves in seconds then running many optimizations is a luxury you have (and should indulge). If your model takes several hours to solve then this may unfortunately limit your ability to perform the number of solves required to derive confident and correct conclusions.
I've run my model with many seeds. What's next?
Repeating optimizations with different values of Seed will only be useful if you are capturing results. If the result of your experiment is a series of log files, then our Python package gurobi-logtools can make quick work of parsing this data, manipulating it, and plotting charts. If you're familiar with data analysis using Pandas then this package will make your life easy. If you are not familiar with Pandas then there are alternative methods such as using gurobi-logtools to produce a CSV which can then be explored with Excel, or perhaps a simple "command line grep" (for Linux and MacOS users) will be sufficient to make a start.
Alternatively, you can loop through Seed values with code and store key results of the optimizations in a data structure. This can avoid the need for tools such as gurobi-logtools when a simple comparison is all that is required.
Above: A boxplot, as produced by gurobi-logtools, which compares the distributions of run times produced by MIPFocus=1, and MIPFocus=3. Note that the fastest time belongs to MIPFocus=1 but in general MIPFocus=3 will produce a better result - it's median value and performance variance are much lower.
Comments
0 comments
Article is closed for comments.