How to debug frozen solver?
Awaiting user inputHi I have a problem with solver getting stuck at Root relaxation - it stays there for 2h without any progress. Any ideas?
m.params.Presolve = 2
m.params.PreSparsify = 1
m.params.NodefileStart = 0.5
m.params.Threads = 8
m.params.LogFile = add_timestamp_to_filename('logs/GRB_opt.txt')
m.params.DegenMoves = 0
CPU model: Intel(R) Xeon(R) CPU @ 2.80GHz, instruction set [SSE2|AVX|AVX2|AVX512] Thread count: 24 physical cores, 48 logical processors, using up to 8 threads WLS license 2543206 - registered to Nexocode Optimize a model with 60561680 rows, 55516429 columns and 132824384 nonzeros Model fingerprint: 0x6eba6ced Model has 14042 general constraints Variable types: 443751 continuous, 55072678 integer (50705781 binary) Coefficient statistics: Matrix range [1e+00, 1e+03] Objective range [1e+00, 1e+00] Bounds range [1e+00, 1e+00] RHS range [1e+00, 2e+04] Found heuristic solution: objective 8.591800e+14 Presolve removed 0 rows and 0 columns (presolve time = 7s) ... Presolve removed 0 rows and 0 columns (presolve time = 11s) ... Presolve removed 722398 rows and 14161 columns (presolve time = 18s) ... Presolve removed 730539 rows and 14280 columns (presolve time = 21s) ... Presolve removed 736001 rows and 310020 columns (presolve time = 25s) ... Presolve removed 736001 rows and 46635743 columns (presolve time = 44s) ... Presolve removed 97753344 rows and 46635743 columns (presolve time = 45s) ... Presolve removed 97753344 rows and 46635743 columns (presolve time = 53s) ... Presolve removed 47061724 rows and 46635743 columns Presolve time: 52.65s Presolved: 13499956 rows, 8880686 columns, 39426392 nonzeros Found heuristic solution: objective 5.002538e+07 Variable types: 133677 continuous, 8747009 integer (7010725 binary) Root relaxation presolve removed 34 rows and 34 columns (presolve time = 5s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 10s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 15s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 20s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 25s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 30s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 35s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 40s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 46s) ... Root relaxation presolve removed 34 rows and 34 columns (presolve time = 50s) ... Root relaxation presolve removed 34 rows and 34 columns Root relaxation presolved: 4629965 rows, 4634825 columns, 17707188 nonzeros Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier Showing barrier log only... Root barrier log... Elapsed ordering time = 19s Elapsed ordering time = 22s Elapsed ordering time = 25s Elapsed ordering time = 30s Elapsed ordering time = 35s Elapsed ordering time = 40s Elapsed ordering time = 45s Elapsed ordering time = 50s Elapsed ordering time = 55s Elapsed ordering time = 60s Elapsed ordering time = 65s Elapsed ordering time = 70s Elapsed ordering time = 75s Elapsed ordering time = 80s Elapsed ordering time = 85s Elapsed ordering time = 90s Elapsed ordering time = 95s Elapsed ordering time = 100s Elapsed ordering time = 105s Elapsed ordering time = 110s Elapsed ordering time = 115s Elapsed ordering time = 120s Elapsed ordering time = 125s Elapsed ordering time = 130s Elapsed ordering time = 135s Elapsed ordering time = 140s Elapsed ordering time = 145s Elapsed ordering time = 150s Elapsed ordering time = 155s Elapsed ordering time = 160s Elapsed ordering time = 165s Elapsed ordering time = 170s Elapsed ordering time = 175s Elapsed ordering time = 180s Elapsed ordering time = 185s Elapsed ordering time = 190s Elapsed ordering time = 195s Elapsed ordering time = 200s Elapsed ordering time = 205s Elapsed ordering time = 210s Elapsed ordering time = 215s Elapsed ordering time = 220s Elapsed ordering time = 225s Elapsed ordering time = 230s Elapsed ordering time = 235s Elapsed ordering time = 240s Elapsed ordering time = 245s Elapsed ordering time = 250s Elapsed ordering time = 255s Elapsed ordering time = 260s Elapsed ordering time = 265s Elapsed ordering time = 270s Elapsed ordering time = 275s Elapsed ordering time = 280s Elapsed ordering time = 313s Elapsed ordering time = 315s Elapsed ordering time = 320s Elapsed ordering time = 325s Ordering time: 838.67s Elapsed ordering time = 839s Elapsed ordering time = 840s Elapsed ordering time = 845s Elapsed ordering time = 850s Elapsed ordering time = 855s Elapsed ordering time = 860s Elapsed ordering time = 865s Elapsed ordering time = 870s Elapsed ordering time = 875s Elapsed ordering time = 880s Elapsed ordering time = 885s Elapsed ordering time = 890s Elapsed ordering time = 895s Elapsed ordering time = 900s Ordering time: 908.07s Barrier statistics: Dense cols : 13225 AA' NZ : 1.742e+07 Factor NZ : 2.565e+08 (roughly 6.0 GB of memory) Factor Ops : 1.079e+12 (roughly 6 seconds per iteration) Threads : 6 Objective Residual Iter Primal Dual Primal Dual Compl Time 0 1.22244919e+09 -6.81588642e+09 5.92e+03 2.65e+02 9.06e+04 1371s 1 1.19094833e+09 -6.93573859e+09 5.71e+03 9.96e+02 8.25e+04 1380s 2 1.12402383e+09 -7.55600633e+09 5.25e+03 1.48e+03 7.84e+04 1389s 3 7.25843525e+08 -8.37559832e+09 3.00e+03 8.54e+02 4.70e+04 1399s 4 1.43538171e+08 -9.05683853e+09 3.44e+02 1.83e+00 6.14e+03 1411s 5 6.60118868e+07 -6.66387784e+09 9.22e+01 1.35e+00 1.82e+03 1425s 6 4.29623838e+07 -3.16812403e+09 1.54e+01 9.70e-01 4.10e+02 1440s 7 3.78905904e+07 -8.18539311e+08 2.11e+00 2.51e-01 7.54e+01 1454s 8 3.53962295e+07 -1.40811914e+08 4.50e-01 3.53e-02 1.37e+01 1466s 9 3.40602119e+07 -3.98061845e+07 2.41e-01 1.31e-02 5.58e+00 1476s 10 3.36071360e+07 -2.64158820e+07 1.88e-01 1.06e-02 4.50e+00 1485s 11 3.33476404e+07 -1.95361227e+07 1.59e-01 9.32e-03 3.94e+00 1493s 12 3.32959745e+07 -1.35761238e+07 1.53e-01 8.13e-03 3.49e+00 1501s 13 3.29926118e+07 -7.13840850e+06 1.22e-01 6.95e-03 2.98e+00 1509s 14 3.27056670e+07 8.52086671e+06 9.57e-02 4.19e-03 1.79e+00 1518s 15 3.26283181e+07 1.14558509e+07 8.94e-02 3.65e-03 1.56e+00 1528s 16 3.24229144e+07 1.74068697e+07 7.49e-02 2.56e-03 1.11e+00 1540s 17 3.23906760e+07 1.76552072e+07 7.15e-02 2.52e-03 1.08e+00 1550s 18 3.23523320e+07 1.92521938e+07 6.77e-02 2.23e-03 9.63e-01 1562s 19 3.21305914e+07 2.32306306e+07 4.94e-02 1.48e-03 6.52e-01 1579s 20 3.20510189e+07 2.45716590e+07 4.15e-02 1.24e-03 5.47e-01 1594s 21 3.19834798e+07 2.63968825e+07 3.35e-02 9.15e-04 4.08e-01 1608s 22 3.19592584e+07 2.71983551e+07 3.09e-02 7.73e-04 3.48e-01 1621s 23 3.19351880e+07 2.78535951e+07 2.84e-02 6.59e-04 2.98e-01 1636s 24 3.18825998e+07 2.92514487e+07 2.32e-02 4.14e-04 1.92e-01 1651s 25 3.18525625e+07 2.98680575e+07 2.02e-02 3.07e-04 1.45e-01 1665s 26 3.18127915e+07 3.00915416e+07 1.56e-02 2.69e-04 1.25e-01 1680s 27 3.17964968e+07 3.04121565e+07 1.39e-02 2.14e-04 1.01e-01 1695s 28 3.17748483e+07 3.06496220e+07 1.12e-02 1.73e-04 8.19e-02 1708s 29 3.17552467e+07 3.10292130e+07 8.87e-03 1.09e-04 5.29e-02 1723s 30 3.17464118e+07 3.12007845e+07 7.75e-03 8.05e-05 3.97e-02 1738s 31 3.17401670e+07 3.13092930e+07 6.91e-03 6.24e-05 3.14e-02 1753s 32 3.17223945e+07 3.15537267e+07 4.38e-03 2.23e-05 1.23e-02 1767s 33 3.17159461e+07 3.15952570e+07 3.35e-03 1.56e-05 8.79e-03 1781s 34 3.17150281e+07 3.16015385e+07 3.22e-03 1.46e-05 8.27e-03 1797s 35 3.17097189e+07 3.16260534e+07 2.36e-03 1.07e-05 6.09e-03 1810s 36 3.17046144e+07 3.16590320e+07 1.53e-03 5.25e-06 3.32e-03 1822s 37 3.16951985e+07 3.16942277e+07 1.86e-06 7.71e-08 7.05e-05 1835s 38 3.16951750e+07 3.16951384e+07 7.40e-11 2.90e-09 2.66e-06 1850s 39 3.16951750e+07 3.16951749e+07 2.58e-11 1.02e-11 2.66e-09 1862s Barrier solved model in 39 iterations and 1861.80 seconds (1370.75 work units) Optimal objective 3.16951750e+07 Root crossover log... 48905 DPushes remaining with DInf 0.0000000e+00 1869s 215 DPushes remaining with DInf 0.0000000e+00 1871s 0 DPushes remaining with DInf 0.0000000e+00 1872s 4341107 PPushes remaining with PInf 0.0000000e+00 1873s 3590623 PPushes remaining with PInf 0.0000000e+00 1875s 2632104 PPushes remaining with PInf 0.0000000e+00 1881s 2174408 PPushes remaining with PInf 0.0000000e+00 1885s 1839326 PPushes remaining with PInf 0.0000000e+00 1890s 1580771 PPushes remaining with PInf 0.0000000e+00 1896s 1357750 PPushes remaining with PInf 0.0000000e+00 1901s 1169415 PPushes remaining with PInf 0.0000000e+00 1905s 1003536 PPushes remaining with PInf 0.0000000e+00 1910s 862037 PPushes remaining with PInf 0.0000000e+00 1915s 717770 PPushes remaining with PInf 0.0000000e+00 1921s 620841 PPushes remaining with PInf 0.0000000e+00 1926s 537886 PPushes remaining with PInf 0.0000000e+00 1930s 451111 PPushes remaining with PInf 0.0000000e+00 1936s 391527 PPushes remaining with PInf 0.0000000e+00 1940s 330227 PPushes remaining with PInf 0.0000000e+00 1946s 288667 PPushes remaining with PInf 0.0000000e+00 1950s 245300 PPushes remaining with PInf 0.0000000e+00 1956s 216869 PPushes remaining with PInf 0.0000000e+00 1960s 186482 PPushes remaining with PInf 0.0000000e+00 1966s 164758 PPushes remaining with PInf 0.0000000e+00 1970s 143325 PPushes remaining with PInf 0.0000000e+00 1976s 129520 PPushes remaining with PInf 0.0000000e+00 1980s 113880 PPushes remaining with PInf 0.0000000e+00 1986s 103456 PPushes remaining with PInf 0.0000000e+00 1990s 91825 PPushes remaining with PInf 0.0000000e+00 1996s 19774 PPushes remaining with PInf 0.0000000e+00 2016s 10779 PPushes remaining with PInf 0.0000000e+00 2033s 9780 PPushes remaining with PInf 0.0000000e+00 2063s 8781 PPushes remaining with PInf 0.0000000e+00 2104s 7782 PPushes remaining with PInf 0.0000000e+00 2152s 6783 PPushes remaining with PInf 0.0000000e+00 2212s 5784 PPushes remaining with PInf 0.0000000e+00 2288s 4785 PPushes remaining with PInf 0.0000000e+00 2376s 3786 PPushes remaining with PInf 0.0000000e+00 2478s 2787 PPushes remaining with PInf 0.0000000e+00 2603s 1788 PPushes remaining with PInf 0.0000000e+00 2748s 789 PPushes remaining with PInf 0.0000000e+00 2923s 0 PPushes remaining with PInf 0.0000000e+00 3080s Push phase complete: Pinf 0.0000000e+00, Dinf 1.2404266e-05 3081s Root simplex log... Iteration Objective Primal Inf. Dual Inf. Time 4383819 3.1695175e+07 0.000000e+00 0.000000e+00 3130s Waiting for other threads to finish... 3195s Waiting for other threads to finish... 3259s Waiting for other threads to finish... 3322s Waiting for other threads to finish... 3386s Waiting for other threads to finish... 3450s Waiting for other threads to finish... 3514s Waiting for other threads to finish... 3578s Waiting for other threads to finish... 3642s Waiting for other threads to finish... 3705s Waiting for other threads to finish... 3769s Waiting for other threads to finish... 3833s Waiting for other threads to finish... 3897s Waiting for other threads to finish... 3961s Waiting for other threads to finish... 4025s Waiting for other threads to finish... 4089s Waiting for other threads to finish... 5429s Waiting for other threads to finish... 5492s Concurrent spin time: 2398.41s (can be avoided by choosing Method=3) Solved with barrier 4383819 3.1695175e+07 0.000000e+00 0.000000e+00 5554s Root relaxation: objective 3.169517e+07, 4383819 iterations, 5231.77 seconds (3666.34 work units)
-
Hi Krzysztof,
Is it possible that this run is hitting your machine's memory limit? It is possible that the OS starts swapping memory to not abort the process which as a result slows down the affected process tremendously.
If it is a memory issue, you might want to have a look at How do I avoid an out-of-memory condition?
Best regards,
Jaromił0 -
I would also add, to what Jaromił wrote, that your model is extremely large. I would suggest you reconsider the formulation of your model.
Judging by the other questions you asked, I believe you are trying to solve an inventory optimization problem. Perhaps this paper could serve as an inspiration.
Best regards
Jonasz0
Please sign in to leave a comment.
Comments
2 comments