Skip to main content

Reduce memory usage of gurobipy Model

Answered

Comments

4 comments

  • Jaromił Najman
    Gurobi Staff Gurobi Staff

    Hi Christian,

    Could you please provide a minimal reproducible example? Which Gurobi version are you using (please use the latest version if you are not already using it)?

    Best regards, 
    Jaromił

    0
  • Christian Perau
    Gurobi-versary
    Conversationalist
    First Question

    Hi Jaromil,
    sure! I was working with gurobi 9.5.2. Updating to 10.0 did however only slightly increase the memory usage.
    Here is an example. It definetly does not make any sense but will show the memory allocation.

    import numpy as np
    import gurobipy as gp
    from gurobipy import GRB
    import scipy.sparse as sp
    import guppy
    n_var = 90000000
    n_constraints = 40000000
    model = gp.Model()
    lb = np.zeros(n_var, dtype=float)
    ub = np.full(n_var, 100, dtype=float)
    obj = np.ones(n_var, dtype=float)
    vtypes = np.full(n_var, GRB.CONTINUOUS)
    gp_vars = model.addMVar(n_var, lb, ub, obj, vtypes)
    rows = np.arange(n_constraints)
    cols = np.arange(n_constraints)
    val = np.ones(n_constraints, dtype=float)
    A=sp.csr_matrix((val, (rows, cols)), shape=(n_constraints, n_var))
    sense = np.full(n_constraints, '>')
    rhs = np.ones(n_constraints, dtype=float)
    model.addMConstr(A, gp_vars, sense, rhs)
    h=guppy.hpy()
    print(h.heap())

    Output



    Partition of a set of 390325818 objects. Total size = 28287024970 bytes.
     Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
         0 90000000  23 9360000000  33 9360000000  33 dict of gurobipy.Var
         1 90000000  23 4320000000  15 13680000000  48 gurobipy.Var
         2 40000000  10 4160000000  15 17840000000  63 dict of gurobipy.Constr
         3     81   0 3800008386  13 21640008386  77 numpy.ndarray
         4 130009159  33 3640271656  13 25280280042  89 int
         5 40000000  10 1920000000   7 27200280042  96 gurobipy.Constr
         6   3960   0 1045994480   4 28246274522 100 list
         7 103196   0 13057412   0 28259331934 100 str
         8  74405   0  5307456   0 28264639390 100 tuple
         9  21552   0  3825727   0 28268465117 100 types.CodeType
    <1004 more rows. Type e.g. '_.more' to view.>

    The increase in memory happens when the variables are created (model.addMVar). And mostly the dictionaries of gurobi variables are using most (~33%) of the storage.

     

    0
  • Jaromił Najman
    Gurobi Staff Gurobi Staff

    Hi Christian,

    The memory consumption you see is currently expected. While our APIs are designed to be only a thin layer on top of the C code, special objects such as MVars require additional handling on the Python layer. These objects are pretty heavy because they are already a dictionary of their attributes. Then, we have to also store additional pointers to vars contained in the MVar object. This all comes on top of the actual values, variable types, lower and upper bounds, and names of each variable.

    We are actively working on reducing the memory requirements in these cases.

    Best regards, 
    Jaromił

    0
  • Christian Perau
    Gurobi-versary
    Conversationalist
    First Question

    Hi Jaromil,
    That´s what I expected. I guess that´s another downside of using python. Thank you very much for your explanation!
    Best regards,
    Christian

    0

Please sign in to leave a comment.