Gurobi running out of memory in loop, but deleting environment every iteration

回答済み

Tenzin Frijlink

• 2024年03月29日 09:45

Good morning,

It would be greatly appreciated if anyone could help me with the following issue:

I am optimizing a certain problem using a rolling horizon approach. Therefore I need to optimize at each time step, apply the first action and repeat. As such I need to call on Gurobi optimizer roughly 1000 times. However, I run out of memory after a hew hundred iterations, even though I think I am properly deleting the environment and model every iteration (constraints change every iteration so I am not keeping the model), so it should run indefinitely.

I checked every other part of the code, and without Gurobi I have no memory issues at all, so Gurobi must be the issue. For a singular solve Gurobi takes roughly 1 second on average to solve to optimality, so the model itself is also not the issue I think. I use Python (VScode) and Gurobi 11.0.1 on Windows. This is how I call on Gurobi in the loop:

gmodel = Gurmodel()

gmodel.solve_model(instance, current_state, fcperiods, sol_cur)

rt[it] = gmodel.rt

gap[it] = gmodel.gap

gurobj[it] = gmodel.obj

sol_cur = gmodel.sol

improvement[it] = round((gurobj[it] - greobj[it]) * 100 / greobj[it], 4)

gmodel = None

 

So I don't think this should cause the issue. This is the code of the model: 

 

 

import numpy as np

from solution import Solution

# Model class

class Gurmodel:

    # Input is instance, state of the model, run time limit and time horizon

    def __init__(self):

        # Define empty return params

        self.rt = None

        self.gap = None

        self.obj = None

        self.sol = None

    def solve_model(self, instance, current_state, horizon, MIPsol=None, M=2000):

        import gurobipy as gp

        # Define Model

        options = {WLS keys}

       

        with gp.Env(params=options) as env, gp.Model(env=env) as model:

            model.setParam('OutputFlag', 0)

            model.setParam('TimeLimit', 10)

            model.setParam("MIPgap", 0.01)

            # Define sets

           creating sets here, removed for readability

            # create variables

            y = {}

            x = {}

            b = {}

            for i, j in A:

                for t in T:

                    y[i, j, t] = model.addVar(vtype=gp.GRB.BINARY, name="Y[" + str(i) + "][" + str(j) + "][" + str(t) + "]")

                for k in K:

                    for t in T:

                        x[i, j, k, t] = model.addVar(vtype=gp.GRB.INTEGER, lb=0, ub=cap[i, j, k],

                                                    name="X[" + str(i) + "][" + str(j) + "][" + str(k) + "][" + str(

                                                        t) + "]")

            for i in V:

                for k in K:

                    for t in T:

                        if i in O:

                            b[i, k, t] = model.addVar(vtype=gp.GRB.INTEGER, lb=0, ub=ocap[i, k],

                                                    name="B[" + str(i) + "][" + str(k) + "][" + str(t) + "]")

                        else:

                            b[i, k, t] = model.addVar(vtype=gp.GRB.INTEGER, lb=0, ub=(instance.turbs - current_state.installed) * pk[k],

                                                    name="B[" + str(i) + "][" + str(k) + "][" + str(t) + "]")

            # Constraints

            # 1 Arc usage

            for i, j in A:

                for t in T:

                    model.addConstr(M * y[i, j, t] >= sum(x[i, j, k, t] for k in K))

            # 2 Time duration

            for i, j in A:

                for t in range(horizon - 1):

                    max = t + transit[i][j][t]

                    current = min(horizon, max)

                    if t + 1 == current:

                        model.addConstr(y[i, j, t + 1] <= 1 - y[i, j, t])

                    else:

                        subT = range(t + 1, int(current))

                        model.addConstr(sum(y[i, j, z] for z in subT) <= (1 - y[i, j, t]) * M)

           

            # 3 Synchronization

            for t in T:

                for i, j in A:

                    max = t + transit[i][j][t]

                    temp = min(horizon, max)

                    subT = range(t, int(temp))

                    model.addConstr(sum(y[k, l, z] for z in subT for k, l in Adep[i, j]) <= (1 - y[i, j, t]) * M)

           

            # 4 Flow conservation

            for i in V:

                for t in range(horizon - 1):

                    for k in K:

                        model.addConstr(b[i, k, t + 1] == b[i, k, t] + sum(x[n, m, k, z] for n, m, z in Fplus[i, t + 1])

                                        - sum(x[n, m, k, z] for n, m, z in Fmin[i, t + 1]))

                       

            # 5 Loading capacity for transshipment/loading operations

            for t in T:

                for i, j in A:

                    if (i, j) in loadcap:

                        model.addConstr(sum(x[i, j, k, t] for k in K) <= loadcap[i, j])

            # 6 Inventory capacity

            for i in V:

                for t in T:

                    model.addConstr(sum(b[i, k, t] * sk[k] for k in K) <= li[i])

            # 8 Installation

            for t in T:

                for k in K:

                    for (i, j) in Ainst:

                        model.addConstr(x[i, j, k, t] >= pk[k] * y[i, j, t])

                        model.addConstr(x[i, j, k, t] <= pk[k] * y[i, j, t])

            # 9 Initial values

            for i in V:

                for k in K:

                    # Set initial inventories

                    model.addConstr(b[i, k, 0] == ininv[i][k])

            for i in range(len(A)):

                k, l = A[i]

                model.addConstr(y[k, l, 0] == iny[i])

                for c in K:

                    model.addConstr(x[k, l, c, 0] == inx[i][c])

            # If shuttling instance constraint 10 needs to be active

            if instance.shuttle:

                Areturn = instance.Areturn

                # 10 Must return

                for t in T:

                    for (i, j) in Areturn:

                        factor = 0

                        if inback:

                            factor = 1

                        Z = range(t)

                        model.addConstr(sum(y[j, i, z] for z in Z) <= sum(y[i, j, z] for z in Z) + factor)

                        model.addConstr(sum(y[i, j, z] for z in Z) <= sum(y[j, i, z] for z in Z) + 1 - factor)

                        model.addConstr(sum(b[i, k, t] for k in K) + sum(b[j, k, t] for k in K) <=

                                            (1 - y[i, j, t] - y[j, i, t]) * M)

            # objective

            totalcost = gp.quicksum(ct[i, j, t] * y[i, j, t] for t in T for i, j in A) + gp.quicksum(

                ch[i] * sk[k] * b[i, k, t] for k in K for i in V for t in T)

            model.setObjective(totalcost, gp.GRB.MINIMIZE)

            # Provide MIP start if wanted

            if MIPsol is not None:

                model.NumStart = 1

                arcs = len(A)

                for i in range(arcs):

                    k, l = A[i]

                    for t in T:

                        y[k, l, t].Start = int(MIPsol.yval[i, t])

                        for c in K:

                            x[k, l, c, t].Start = int(MIPsol.xval[i, c, t])

                for i in V:

                    for t in T:

                        for k in K:

                            b[i, k, t].Start = int(MIPsol.bval[i, k, t])

            # model.update()

            model.write("model.lp")

            model.optimize()

            # for v in model.getVars():  # print all variables that are greater than 1

            #     if (v.x > 0.1):

            #         print('%s %g' % (v.varName, v.x))

            if model.status == gp.GRB.status.INFEASIBLE:

                print("Gurobi solution is infeasible. Reason:")

                print(model.computeIIS())

                # Print which constraints are violated

                print("Constraints in the Irreducible Inconsistent Subsystem (IIS):")

                for constr in model.getConstrs():

                    if constr.IISConstr:

                        print(f"Constraint {constr.ConstrName} is violated.")

                        # Remove constraints

                model.dispose()

                return 0

            self.rt = round(model.Runtime, 4)

            self.gap = round(model.MIPGap, 4)

            self.obj = round(model.getObjective().getValue())

            print("Took Gurobi", self.rt, "seconds to solve, with gap ", self.gap * 100, " percent and obj",

                self.obj)

            # Save solution

            arcs = len(A)

            bcur = np.zeros((size, components, time))

            xcur = np.zeros((arcs, components, time))

            ycur = np.zeros((arcs, time))

            # Construct bcur directly

            for (i, k, t), var in b.items():

                bcur[i, k, t] = var.X

            # Construct xcur directly

            for (i, j, k, t), var in x.items():

                arcs_index = A.index((i, j))

                xcur[arcs_index, k, t] = var.X

            # Construct ycur directly

            for (i, j, t), var in y.items():

                arcs_index = A.index((i, j))

                ycur[arcs_index, t] = var.X

            # Set solution

            self.sol = Solution(xcur, ycur, bcur, ind, inback)

       

        # Get rid of model and environment

        model.close()

        del model

        env.close()

        del env

        gp.disposeDefaultEnv()

        del gp

        return 1

関連するエンティティ

• Memory

1

• 

  Jaromił Najman

  Gurobi Staff

  • 2024年04月02日 15:47

  Hi Tenzin,

  Is the out-of-memory error happening during model building or during the solution process?

  Could you please try using the dispose() method instead of close()?

  Best regards, 
  Jaromił

  0
• 

  Tenzin Frijlink

  • 2024年04月03日 06:22

  

  

  Hi Jaromil.

  Thank you for the reply and suggestion! I tried using dispose instead of close, but it does not seem to matter. Also for my interest, according to the documentation close and dispose are equal methods, so what is the use of the one over the other?

  After checking I think the out-of-memory error occurs during model building. This is also where it slows down significantly after roughly 200 iterations. I also kept track of the the run times reported by Gurobi and they remain at a few seconds, even before running out of memory, so I am fairly sure now the issue is at the model building. 

  It gets stuck for a few minutes at this stage sometimes (and sometimes recovers and sometimes gives out of memory error):

  Set parameter Username
  Set parameter WLSAccessID
  Set parameter WLSSecret
  Set parameter LicenseID to [redacted]
  Academic license   [redacted]
  Set parameter TimeLimit to value 10
  Set parameter MIPGap to value 0.01

  Whereas in other iterations it goes straight to this and then goes through the solving process:

  Set parameter Username
  Set parameter WLSAccessID
  Set parameter WLSSecret
  Set parameter LicenseID to value  [redacted]
  Academic license [redacted]
  Set parameter TimeLimit to value 10
  Set parameter MIPGap to value 0.01
  Gurobi Optimizer version 11.0.1 build v11.0.1rc0 (win64 - Windows 10.0 (19045.2))

  CPU model: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, instruction set [SSE2|AVX|AVX2]
  Thread count: 4 physical cores, 8 logical processors, using up to 8 threads

   

  Thanks again,

  Tenzin

  0
• 

  Jaromił Najman

  Gurobi Staff

  • 2024年04月03日 07:09

  Hi Tenzin,

  Also for my interest, according to the documentation close and dispose are equal methods, so what is the use of the one over the other?

  This was just to check whether one or the other has a bug, but since both behave the same, it should be alright.

  Could you try computing how many variables and constraints you are trying to add in each iteration? In particular could you please compute the number of variables and constraints you are trying to add in the iteration when you seem to run into the memory issue? It is possible that your machine just runs out of memory due to model size. Could you also please try tracking the memory consumption to see whether the memory drops after an optimization process has finished. This is to make sure that the model and environment are properly disposed.

  Best regards, 
  Jaromił

  0
• 

  Max Berktold

  • 2024年07月16日 15:55

  

  I have exactly the same Problem. This only seems to be the case with the WLS license. On my first laptop i don't have this issue. I would really love to see a solution to this problem.

   

  Here my code to recreate the problem, you can literally see the steady memory increase in the task manager.

  import time
  import gurobipy as gp
  from gurobipy import GRB
  import numpy as np

  def create_large_random_milp(model, num_vars, num_constraints, density=0.1, integer_ratio=0.5, seed=None):

      if seed is not None:
          np.random.seed(seed)

      # Generate random objective coefficients
      obj_coeffs = np.random.rand(num_vars)

      # Generate variable types (binary, integer, or continuous)
      var_types = np.random.choice([GRB.BINARY, GRB.INTEGER, GRB.CONTINUOUS], size=num_vars,
                                   p=[integer_ratio / 2, integer_ratio / 2, 1 - integer_ratio])

      # Create variables
      vars = []
      for i in range(num_vars):
          var = model.addVar(vtype=var_types[i], obj=obj_coeffs[i], name=f'x_{i}')
          vars.append(var)

      # Generate random constraints
      for j in range(num_constraints):
          # Select a random subset of variables for each constraint
          subset = np.random.choice(vars, size=int(density * num_vars), replace=False)

          # Generate random coefficients for the selected variables
          coeffs = np.random.rand(len(subset))

          # Generate a random right-hand side
          rhs = np.random.rand()

          # Add the constraint to the model
          model.addConstr(gp.quicksum(coeffs[k] * subset[k] for k in range(len(subset))) <= rhs, name=f'c_{j}')

      # Set model to maximize or minimize
      model.ModelSense = GRB.MAXIMIZE

      # Integrate new variables into the model
      model.update()
      model.optimize()

  
  # Example usage:
  num_vars = 20000  # Number of variables
  num_constraints = 200  # Number of constraints
  density = 0.05  # Density of each constraint (percentage of variables involved in each constraint)
  integer_ratio = 0.3  # Ratio of integer variables (binary + integer)
  seed = 42  # Random seed for reproducibility

  
  for i in range(1, 5):
      env = gp.Env()
      model = gp.Model(env=env)

      print('Starting:', i)
      print(create_large_random_milp(model, num_vars, num_constraints, density, integer_ratio, seed))
      print('Finished', i)
      time.sleep(5)

      model.close()
      env.dispose()

   

  0
• 

  Max Berktold

  • 2024年07月17日 06:36

  

  For me the problem was python312. When downgrading to python310 this problem did not occur.

  Greetings!

  0
• 

  Can Akkan

  • 2024年07月27日 10:27

  

  I have had a similar problem.  I am solving MIP models repeatedly (hundreds of times) using gurobi (v. 11.01) python . m.dispose() was freeing up only a small fraction of the memory allocated during model creation and solving when I used python312. As suggested above I switched to python 310, and the problem is solved. Thanks.

  0
• 

  Haoxiang Yang

  • 2024年08月01日 22:09

  

  

  Hi, I had the same problem with Python 312. The memory usage seems to pile up even when I tried "del model". The issue was solved when I switched to Python 310. Thanks!

  0
• 

  Simon Bowly

  Gurobi Staff

  • 2024年08月08日 12:18

  Hi all, sorry for the delayed follow-up. Is everyone experiencing this issue using numpy in some way? If anyone has a minimal working example where the memory issue occurs and numpy is *not* involved that would be helpful.

  0
• 

  Can Akkan

  • 2024年08月12日 09:28

  

  Dear Simon

  I am using numpy. So unfortunately I do not have such an example.

  0
• 

  Alex Lusak

  • 2024年08月19日 09:46
  • 編集済み

  

  I'd also like to share that I was having the same issue until I downgraded from 3.12 to 3.10. It seems like model building is consuming infinitely more memory on 3.12. On 3.10, I peaked around maybe 5 GB, while on 3.12 it was climbing all the way to 36 GB then could no longer solve as it exceeded RAM space.

  Is Gurobi actively working to fix this? This seems like a pretty insane difference in consumption just for changing the Python version.

  0
• 

  Mario Ruthmair

  Gurobi Staff

  • 2024年08月19日 10:26

  Our developers narrowed it down to a memory leak related to the Cython version used to build gurobipy 11.0.x. We are using a newer Cython version for the coming 12.0.0 release, so the issue will be resolved there.
  As you have already realized, the issue seems to be triggered by passing a Var or list of Vars to np.array or np.asarray. This can happen in library code like cvxpy, and in some corners of our own matrix-friendly API, as well as directly in user code.
  Unfortunately, the only workaround is to use Python 3.11 or earlier with gurobipy 11.0.x.

  0
• 

  Mario Ruthmair

  Gurobi Staff

  • 2024年08月19日 13:46

  Here is a bug report: https://support.gurobi.com/hc/en-us/articles/27635574446353-Memory-leak-in-gurobipy-with-Python-3-12

  0

サインインしてコメントを残してください。