Issue recovering good incumbent solution with crossover

Hello to everyone!

I have a (binary) MILP problem on network that is likely totally unimodular. Indeed, empirical evidences when solving small instances show that the objective function of the relaxed root problem coincide with the optimal integer one. See the following log:

Warning: linear constraint 38758 and linear constraint 77517 have the same name "flow_limit"
Start Optimization
Set parameter Heuristics to value 0.1
Set parameter Cuts to value 2
Set parameter Presolve to value 2
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (armlinux64 - "Debian GNU/Linux 11 (bullseye)")
CPU model: ARM64
Thread count: 8 physical cores, 8 logical processors, using up to 8 threads

Non-default parameters:
Heuristics  0.1
Cuts  2
Presolve  2
Optimize a model with 272975 rows, 313398 columns and 1158541 nonzeros
Model fingerprinet: 0xa6bbd8d8
Variable types: 235049 continuous, 78349 integer (78349 binary)

Coefficient statistics:
 Matrix range     [1e+00, 4e+02]
 Objective range  [1e+00, 1e+00]
 Bounds range     [1e+00, 1e+00]
 RHS range        [4e+00, 1e+04]
Found heuristic solution: objective -0.0000000
Presolve removed 53819 rows and 64219 columns (presolve time = 6s)...
Presolve removed 55620 rows and 65543 columns
Presolve time: 9.94s
Presolved: 217355 rows, 247855 columns, 998763 nonzeros
Variable types: 180775 continuous, 67080 integer (67080 binary)
Root relaxation presolve removed 67463 rows and 1539 columns
Root relaxation presolved: 149892 rows, 246316 columns, 714006 nonzeros
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Ordering time: 0.48s
Barrier statistics:
AA' NZ     : 6.263e+05
Factor NZ  : 2.713e+06 (roughly 180 MB of memory)
Factor Ops : 8.766e+07 (less than 1 second per iteration)
Threads    : 6
                 Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
  0   3.24686898e+03  2.61467960e+06  1.54e+05 1.23e-03  6.05e+02    14s
  1   6.63851833e+02  2.57863721e+06  3.11e+04 3.55e-15  1.27e+02    15s
  2   3.26577454e+02  2.33278025e+06  1.51e+04 8.88e-15  6.25e+01    15s
  3   6.36620388e+01  1.79467663e+06  2.77e+03 5.24e-14  1.34e+01    15s
  4   2.36047496e+01  1.05228354e+06  1.02e+03 7.51e-14  5.16e+00    15s
  […]
 24   4.31999937e+01  4.32010215e+01  1.51e-12 1.78e-15  2.08e-09    18s
 25   4.32000000e+01  4.32000000e+01  2.27e-12 1.78e-15  2.20e-15    19s
Barrier solved model in 25 iterations and 18.82 seconds (8.25 work units)
Optimal objective 4.32000000e+01
Root crossover log...
  33984 DPushes remaining with DInf 0.0000000e+00                20s
      0 DPushes remaining with DInf 0.0000000e+00                21s
 131664 PPushes remaining with PInf 0.0000000e+00                21s
Concurrent spin time: 4.16s (can be avoided by choosing Method=3)
Solved with dual simplex

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
   7300    4.3200000e+01   0.000000e+00   0.000000e+00     21s
Root relaxation: objective 4.320000e+01, 7300 iterations, 11.03 seconds (3.75 work units)
   Nodes    |    Current Node    |     Objective Bounds      |     Work
Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time
*    0     0               0      43.2000000   43.20000  0.00%     -   21s
Explored 1 nodes (7300 simplex iterations) in 22.06 seconds (8.60 work units)
Thread count was 8 (of 8 available processors)
Solution count 2: 43.2 -0 
Optimal solution found (tolerance 1.00e-04)
Best objective 4.320000000000e+01, best bound 4.320000000000e+01, gap 0.0000%
Served: 43.2
Cost:3,813,030.98. N enabled links 482.00

Now, I need to really go at a scale. With a larger instance everything seems to work smoothly:

Gurobi Optimizer version 11.0.2 build v11.0.2rc0 (linux64 - "Ubuntu 22.04.4 LTS")
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 128 logical processors, using up to 32 threads
Optimize a model with 2390913 rows, 3518270 columns and 13963165 nonzeros
Model fingerprint: 0x373ed7e2
Variable types: 3408325 continuous, 109945 integer (109945 binary)

Coefficient statistics:
 Matrix range     [4e-05, 8e+02]
 Objective range  [1e+00, 1e+00]
 Bounds range     [1e+00, 1e+00]
 RHS range        [9e+00, 8e+02]

Found heuristic solution: objective -0.0000000

Presolve removed 1119945 rows and 1063183 columns (presolve time = 5s) ...
Presolve removed 1214675 rows and 1315472 columns (presolve time = 10s) ...
[…]
Presolve removed 1718355 rows and 2248898 columns (presolve time = 61s) ...
Presolve removed 1718355 rows and 2248898 columns
Presolve time: 60.74s
Presolved: 672558 rows, 1269372 columns, 5529646 nonzeros
Variable types: 1175317 continuous, 94055 integer (94055 binary)
Deterministic concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...
Root barrier log...
Elapsed ordering time = 6s
Elapsed ordering time = 10s
Ordering time: 14.45s
Barrier statistics:
AA' NZ     : 7.195e+06
Factor NZ  : 1.287e+08 (roughly 2.0 GB of memory)
Factor Ops : 8.458e+10 (less than 1 second per iteration)
Threads    : 30
                 Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
  0   1.87887435e+05  6.05154898e+07  6.74e+05 2.15e-03  4.75e+03    95s
  1   2.04709236e+04  5.88830324e+07  7.29e+04 2.58e-14  5.32e+02    96s
  2   5.80865404e+03  4.86048194e+07  2.03e+04 1.19e-13  1.59e+02    97s
  3   1.66328674e+03  3.40295382e+07  5.57e+03 1.46e-13  4.83e+01    98s
  4   8.71013887e+02  2.03365672e+07  2.79e+03 1.26e-13  2.34e+01   100s
  […]
 29   2.27305745e+02  3.60240133e+02  6.21e-12 2.04e-12  5.27e-05   131s
 30   2.27389414e+02  2.29203986e+02  6.08e-11 2.20e-12  7.19e-07   132s
 31   2.27389999e+02  2.27391814e+02  2.04e-10 1.97e-12  7.19e-10   133s
 32   2.27390000e+02  2.27390000e+02  2.60e-11 2.24e-12  1.42e-14   134s
Barrier solved model in 32 iterations and 134.18 seconds (78.50 work units)
Optimal objective 2.27390000e+02

Root crossover log...
 138314 variables added to crossover basis                      135s
 133553 DPushes remaining with DInf 0.0000000e+00               136s
      0 DPushes remaining with DInf 0.0000000e+00               140s
 598034 PPushes remaining with PInf 0.0000000e+00               140s
 512525 PPushes remaining with PInf 0.0000000e+00               145s
 […]
   4975 PPushes remaining with PInf 0.0000000e+00               176s
      0 PPushes remaining with PInf 0.0000000e+00               177s
 Push phase complete: Pinf 0.0000000e+00, Dinf 0.0000000e+00    177s

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
 730775    2.2739000e+02   0.000000e+00   0.000000e+00    178s
Concurrent spin time: 0.16s
Solved with dual simplex
Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
  67765    2.2739000e+02   0.000000e+00   0.000000e+00    179s
Root relaxation: objective 2.273900e+02, 67765 iterations, 114.97 seconds (62.78 work units)
   Nodes    |    Current Node    |     Objective Bounds      |     Work
Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time
*    0     0               0     227.3900000  227.39000  0.00%     -  179s
Explored 1 nodes (67765 simplex iterations) in 179.43 seconds (91.32 work units)
Thread count was 32 (of 128 available processors)
Solution count 2: 227.39 -0
Optimal solution found (tolerance 1.00e-04)
Best objective 2.273900000000e+02, best bound 2.273900000000e+02, gap 0.0000%
Cost:9,256,535.36

However, reaching a problem size that (for my application of interest) is medium-size, the algorithm (I guess) is not able to retrieve a good incumbent solution (zero is a trivial solution and, as said, the problem is totally unimodular so I should be able to definitely do better). In addition, it goes out of memory (unclear to me why - 256GB available in total). This are the logs:

/var/spool/slurm/slurmd/state/job48802526/slurm_script: line 21: 3610204 Killed                  /cluster/scratch/me/new_interventions/venv/bin/python -m euler_run
slurmstepd: error: Detected 1 oom_kill event in StepId=48802526.batch. Some of the step tasks have been OOM Killed.

Warning: linear constraint 38758 and linear constraint 77517 have the same name "flow_limit"
Start Optimization
Set parameter MIPFocus to value 1
Set parameter Heuristics to value 0.5
Set parameter Method to value 2
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "Ubuntu 22.04.5 LTS")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 128 logical processors, using up to 32 threads

Non-default parameters:
Method  2
Heuristics  0.5
MIPFocus  1

Optimize a model with 8256498 rows, 16610198 columns and 48499468 nonzeros
Model fingerprint: 0x8211abe0
Model has 156698 simple general constraints
 156698 INDICATOR
Variable types: 16531849 continuous, 78349 integer (78349 binary)

Coefficient statistics:
 Matrix range     [1e+00, 4e+02]
 Objective range  [1e+00, 1e+00]
 Bounds range     [1e+00, 1e+00]
 RHS range        [3e-01, 5e+04]
 GenCon rhs range [1e-03, 1e-03]
 GenCon coe range [1e+00, 1e+00]

Found heuristic solution: objective -0.0000000

Presolve removed 0 rows and 0 columns (presolve time = 8s)...
Presolve removed 210 rows and 0 columns (presolve time = 11s)...
Presolve removed 210 rows and 0 columns (presolve time = 15s)...
Presolve removed 78559 rows and 0 columns (presolve time = 21s)...
Presolve removed 78559 rows and 0 columns (presolve time = 31s)...
Presolve removed 78559 rows and 584640 columns (presolve time = 43s)...
Presolve removed 78559 rows and 584640 columns (presolve time = 45s)...
Presolve removed 78559 rows and 584640 columns (presolve time = 54s)...
[…]
Presolve removed 2928946 rows and 4387838 columns (presolve time = 326s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 330s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 335s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 341s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 345s)...
Presolve removed 2542765 rows and 4077222 columns
Presolve time: 345.17s
Presolved: 5713733 rows, 12532976 columns, 37629999 nonzeros
Variable types: 12465869 continuous, 67107 integer (67107 binary)

Root barrier log...

Elapsed ordering time = 7s
Elapsed ordering time = 10s
Elapsed ordering time = 15s
Elapsed ordering time = 20s
[…]
Elapsed ordering time = 190s
Elapsed ordering time = 195s
Elapsed ordering time = 200s
Ordering time: 628.85s

Barrier statistics:
AA' NZ     : 3.326e+07
Factor NZ  : 2.240e+09 (roughly 26.0 GB of memory)
Factor Ops : 2.356e+13 (roughly 50 seconds per iteration)
Threads    : 32
                 Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
  0   4.45546441e+07  1.43067307e+10  2.25e+07 1.13e-01  2.07e+05  1118s
  1   1.33000697e+07  1.53258357e+10  6.75e+06 8.88e-15  6.75e+04  1126s
  2   5.18683538e+06  1.58769797e+10  2.67e+06 2.13e-14  2.87e+04  1132s
  3   2.08792855e+06  1.53995596e+10  1.11e+06 1.14e-13  1.26e+04  1138s
  4   7.76718365e+05  1.32710229e+10  4.40e+05 1.85e-13  5.07e+03  1145s
  5   2.88082824e+05  9.66694151e+09  1.81e+05 5.54e-13  1.99e+03  1150s
  6   1.42796629e+05  6.05332237e+09  9.64e+04 4.83e-13  9.65e+02  1157s
  7   9.21007322e+04  4.46571213e+09  6.40e+04 2.84e-13  6.22e+02  1163s
  8   7.23827018e+04  3.50223317e+09  5.11e+04 2.70e-13  4.78e+02  1169s
  9   5.76106653e+04  2.76878279e+09  4.12e+04 1.99e-13  3.73e+02  1177s
 10   4.37041672e+04  2.19180967e+09  3.18e+04 1.71e-13  2.83e+02  1184s
[…]
 58   1.48260323e+03  3.65314513e+04  1.19e-03 3.55e-15  1.40e-03  4275s
 59   1.48302837e+03  1.15929223e+04  2.80e-04 1.78e-15  4.04e-04  4387s
 60   1.48314372e+03  4.18874255e+03  3.39e-05 1.78e-15  1.08e-04  4502s
 61   1.48315997e+03  1.49443391e+03  3.08e-08 1.78e-15  4.50e-07  4616s
 62   1.48316000e+03  1.48316002e+03  2.72e-09 3.55e-15  8.23e-13  4723s
 63   1.48316000e+03  1.48316000e+03  4.91e-11 3.55e-15  1.51e-18  4834s
Barrier solved model in 63 iterations and 4834.39 seconds (1818.96 work units)
Optimal objective 1.48316000e+03

Root crossover log...
 331032 variables added to crossover basis                     4842s
2106166 DPushes remaining with DInf 0.0000000e+00              4846s
8928934 PPushes remaining with PInf 0.0000000e+00              4850s
8916109 PPushes remaining with PInf 0.0000000e+00              4852s
8912948 PPushes remaining with PInf 0.0000000e+00              4857s
8897318 PPushes remaining with PInf 0.0000000e+00              4892s
8859012 PPushes remaining with PInf 0.0000000e+00              4896s
8819496 PPushes remaining with PInf 0.0000000e+00              4900s
8765784 PPushes remaining with PInf 0.0000000e+00              4906s
8715862 PPushes remaining with PInf 0.0000000e+00              4911s
8677683 PPushes remaining with PInf 0.0000000e+00              4915s
[…]
60073 PPushes remaining with PInf 0.0000000e+00                6445s
35049 PPushes remaining with PInf 0.0000000e+00                6451s
3678 PPushes remaining with PInf 0.0000000e+00                 6461s
0 PPushes remaining with PInf 0.0000000e+00                    6469s

Push phase complete: Pinf 0.0000000e+00, Dinf 0.0000000e+00    6470s

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
8928937    1.4831600e+03   0.000000e+00   0.000000e+00   6473s
8928937    1.4831600e+03   0.000000e+00   0.000000e+00   6480s

Root relaxation: objective 1.483160e+03, 8928937 iterations, 6114.50 seconds (4293.89 work units)
   Nodes    |    Current Node    |     Objective Bounds      |     Work
Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time
    0     0 1483.16000    0 30422   -0.00000 1483.16000      -     - 66133s

I also tried to solve the relaxed problem through the simplex algorithm, but it gets badly stuck “close” to the optimal solution (see primal infeasibility)

Warning: linear constraint 38758 and linear constraint 77517 have the same name "flow_limit"
Start Optimization
Set parameter MIPFocus to value 1
Set parameter Heuristics to value 0.02
Set parameter Method to value 1
Gurobi Optimizer version 12.0.3 build v12.0.3rc0 (linux64 - "Ubuntu 22.04.5 LTS")
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 128 logical processors, using up to 32 threads

Non-default parameters:
Method  1
Heuristics  0.02
MIPFocus  1

Optimize a model with 8256498 rows, 16610198 columns and 48499468 nonzeros
Model fingerprint: 0x8211abe0
Model has 156698 simple general constraints
 156698 INDICATOR
Variable types: 16531849 continuous, 78349 integer (78349 binary)

Coefficient statistics:
 Matrix range     [1e+00, 4e+02]
 Objective range  [1e+00, 1e+00]
 Bounds range     [1e+00, 1e+00]
 RHS range        [3e-01, 5e+04]
 GenCon rhs range [1e-03, 1e-03]
 GenCon coe range [1e+00, 1e+00]

Found heuristic solution: objective -0.0000000

Presolve removed 0 rows and 0 columns (presolve time = 8s)...
Presolve removed 210 rows and 0 columns (presolve time = 11s)...
Presolve removed 210 rows and 0 columns (presolve time = 16s)...
Presolve removed 78559 rows and 0 columns (presolve time = 21s)...
Presolve removed 78559 rows and 0 columns (presolve time = 31s)...
[…]
Presolve removed 2928946 rows and 4387838 columns (presolve time = 320s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 325s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 332s)...
Presolve removed 2928946 rows and 4387838 columns (presolve time = 338s)...
Presolve removed 2542765 rows and 4077222 columns
Presolve time: 338.43s
Presolved: 5713733 rows, 12532976 columns, 37629999 nonzeros
Variable types: 12465869 continuous, 67107 integer (67107 binary)

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
      0    1.4831600e+03   6.531833e+02   0.000000e+00    368s
  22365    1.4831600e+03   1.154735e+04   0.000000e+00    374s
  54071    1.4831600e+03   1.423278e+04   0.000000e+00    387s
  68453    1.4831597e+03   1.255745e+04   0.000000e+00    395s
  82146    1.4831595e+03   1.171588e+04   0.000000e+00    403s
  95487    1.4831593e+03   1.614290e+04   0.000000e+00    412s
 108565    1.4831591e+03   2.102888e+04   0.000000e+00    421s
 121010    1.4831590e+03   2.446216e+04   0.000000e+00    430s
 133162    1.4831589e+03   1.795461e+04   0.000000e+00    440s
 144941    1.4831588e+03   2.247570e+04   0.000000e+00    450s
 156190    1.4831587e+03   2.024730e+04   0.000000e+00    461s
 […]
 607007    1.4831569e+03   9.432153e+04   0.000000e+00   8027s
 607277    1.4831569e+03   5.320955e+04   0.000000e+00   8075s
 607507    1.4831568e+03   4.893755e+04   0.000000e+00   8125s
 607717    1.4831568e+03   1.920370e+05   0.000000e+00   8174s
 607997    1.4831568e+03   6.294248e+04   0.000000e+00   8222s
 608207    1.4831568e+03   1.020343e+05   0.000000e+00   8272s
 608487    1.4831568e+03   2.892431e+04   0.000000e+00   8322s

I'm therefore writing to ask for a feedback from you on what is going on (is it the out of memory issue reasonable? Is it reasonable that it can't find a good incumbent?) and if you have any suggestion on how to improve the current crossover (flag, etc.)

Thanks a lot for your help!!

(P.S.: I slightly changed the MILP parameters during the experiments but everything is reported in the logs)