Skip to main content

Problems adding columns "Only linear constraints allowed"

Answered

Comments

88 comments

  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    The following version does at least something

    def addColumn(self, newSchedule, iter, index):
            self.newvar = {}
            colName = f"ScheduleUsed[{index},{iter}]"
            newScheduleList = []
            cons_demandList = []
            for i, t, s, r in newSchedule:
                newScheduleList.append(newSchedule[i, t, s, r])
            rounded_ScheduleList = ['%.2f' % elem for elem in newScheduleList]
            Column = gu.Column([], [])
            self.newvar = self.model.addVar(vtype=gu.GRB.CONTINUOUS, lb=0, column=Column, name=colName)
            self.model.update()
    ...

    def modifyConstraint(self):
            for t in self.days:
                for s in self.shifts:
                    self.newcoef = 1.0
                    current_cons = self.cons_demand[t, s]
                    qexpr = self.model.getQCRow(current_cons)
                    new_var = self.newvar
                    new_coef = self.newcoef
                    qexpr.add(new_var, new_coef)
                    rhs = current_cons.getAttr('QCRHS')
                    sense = current_cons.getAttr('QCSense')
                    name = current_cons.getAttr('QCName')
                    newcon = self.model.addQConstr(qexpr, sense, rhs, name)
                    self.model.remove(current_cons)
                    self.cons_demand[t, s] = newcon
                    return newcon
    ...


    if reducedCost < -1e-6:
                ScheduleCuts = subproblem.getNewSchedule()
                master.addColumn(ScheduleCuts, itr, index)
                master.modifyConstraint()
                master.updateModel()
                modelImprovable = True

    Note that in \(\texttt{addColumn}\) the \(\texttt{newvar}\) is not added to any linear constraint currently. It is only created and added to the model. It is then used in \(\texttt{modifyConstraint}\). If you have to add this variable to linear constraints as well, you can do this in \(\texttt{addColumn}\) by providing a list of coefficients and linear constraints.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Ah okay. How would I implement your first "idea"?

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Ah okay. How would I implement your first "idea"?

    In addColumn, you have to collect only linear constraints together with the respective coefficients and construct the column using this information.

    def addColumn(self, newSchedule, iter, index):
            self.newvar = {}
            colName = f"ScheduleUsed[{index},{iter}]"
            newVarCoefficients = []
            linearConstraints = []
            # somehow fill linearConstraints and newVarCoefficients
            Column = gu.Column(newVarCoefficients, linearConstraints)
            self.newvar = self.model.addVar(vtype=gu.GRB.CONTINUOUS, lb=0, column=Column, name=colName)
            self.model.update()

    This way, even if \(\texttt{newvar}\) does not have to be added to any linear constraint, the code would work, because you would still add \(\texttt{newvar}\) to the model.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman 

    Thank you so much. Now the reduced costs also change from iteration to iteration, as columns with values \(\neq0\) are also added. Unfortunately, the target function value does not change. Here I assume that this is because the column only has the indexes [t,s]. This is how it looks in the LP:

    qc20: slack[1,1] + ScheduleUsed[1,1] + ScheduleUsed[2,1]
    + ScheduleUsed[3,1] + ScheduleUsed[1,2] + ScheduleUsed[2,2]
    + ScheduleUsed[3,2] + [ motivation_i[1,1,1,1] * lmbda[1,1]
    + motivation_i[2,1,1,1] * lmbda[2,1]
    + motivation_i[3,1,1,1] * lmbda[3,1] ] >= 2

    However, [i, t, s, r] is used in the query level condition. How is it possible to add these indices to the COlumns of the individual subproblems? So the \(i\) of the respective nurse and the \(r\) as the respective iteration \(+1\). The code in the OP is up to date.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    However, [i, t, s, r] is used in the query level condition. How is it possible to add these indices to the COlumns of the individual subproblems? So the of the respective nurse and the as the respective iteration . The code in the OP is up to date.

     I am not sure I understand the problem. You are currently adding a Column based on two indices \(\texttt{iter}\) and \(\texttt{index}\)

    def addColumn(self, newSchedule, iter, index):
            self.newvar = {}
            colName = f"ScheduleUsed[{index},{iter}]"
    ...

    Can't you also find out the other indices? And are you sure that the other indices play any role?

    You should try to make clear what the constraint \(\texttt{qc20}\) in your comment should look like in a working case. Then, you can start thinking about how to achieve this.

    I have another question. In the \(\texttt{addColumn}\) function you construct an empty Column object which you use, when adding the variable \(\texttt{newvar}\). What is the purpose of this function? Since the Column object is empty, you could just add \(\texttt{newvar}\) in the \(\texttt{modifyConstraints}\) function.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman 

    Ah, I see. The information about the indices \(t,s\) is not relevant, since each of the (in this case) 21 demand constraints is just such a combination. In this case, only the index \(index=i\quad \text{and} \quad iter = r\) is relevant. But what I don't understand now is why the columns are only added to qc20 in my concrete example. Actually, all optimal values of \(motivation^*\) from the respective SPs (21 in number) should be added as a column to the MP, provided that the column has negative reduced costs. If, for example, I now display the optimal values from the SPs after the first iteration, I get this list(for nurse 3):

    {(3, 1, 1, 2): 0.5, (3, 1, 2, 2): 0.0, (3, 1, 3, 2): 0.0, (3, 2, 1, 2): 0.0, (3, 2, 2, 2): 0.5, (3, 2, 3, 2): 0.0, (3, 3, 1, 2): 0.0, (3, 3, 2, 2): 0.0, (3, 3, 3, 2): 0.0, (3, 4, 1, 2): 0.0, (3, 4, 2, 2): 0.5, (3, 4, 3, 2): 0.0, (3, 5, 1, 2): 0.5, (3, 5, 2, 2): 0.0, (3, 5, 3, 2): 0.0, (3, 6, 1, 2): 0.0, (3, 6, 2, 2): 0.0, (3, 6, 3, 2): 0.0, (3, 7, 1, 2): 0.0, (3, 7, 2, 2): 0.5, (3, 7, 3, 2): 0.0}

    According to my understanding, the first value of 0.5 will then be added to constraint qc20 \([t,s]=[1,1]\), the second of 0.0 to qc0 \([t,s]=[1,2]\) and so on. Strangely enough, the columns are only added to qc20, as already mentioned. Why is that the case? In my opinion, this is the first problem: not all optimal values are added as columns, as seen in the master.lp:

    qc0: slack[1,2] + [ motivation_i[1,1,2,1] * lmbda[1,1]
    + motivation_i[2,1,2,1] * lmbda[2,1]
    + motivation_i[3,1,2,1] * lmbda[3,1] ] >= 1
    qc1: slack[1,3] + [ motivation_i[1,1,3,1] * lmbda[1,1]
    + motivation_i[2,1,3,1] * lmbda[2,1]
    + motivation_i[3,1,3,1] * lmbda[3,1] ] >= 0
    ......
    qc19: slack[7,3] + [ motivation_i[1,7,3,1] * lmbda[1,1]
    + motivation_i[2,7,3,1] * lmbda[2,1]
    + motivation_i[3,7,3,1] * lmbda[3,1] ] >= 0
    qc20: slack[1,1] + Schedule[1,1] + Schedule[2,1] + Schedule[3,1]
    + Schedule[1,2] + Schedule[2,2] + Schedule[3,2] + [
    motivation_i[1,1,1,1] * lmbda[1,1] + motivation_i[2,1,1,1] * lmbda[2,1]
    + motivation_i[3,1,1,1] * lmbda[3,1] ] >= 2

     

    The second problem that prevents a successful implementation is the missing assignment of the \(\lambda\)'s. This means that \(motivation_{112}^2\), for example, is never fixed to zero, which would then also change the IF and thus the dual values of the MP. Columns are currently being added, but they are not fixed to zero by multiplication with \(\lambda\), which means that the solver in the MP always sets \(motivation\) to 1 and thus the target coefficient always remains at zero. 

    How can I overcome these two problems?

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    You have a

    return newcon

    in \(\texttt{modifyConstraints}\) which exits the function after the first loop iteration. You have to remove this.

    The second problem that prevents a successful implementation is the missing assignment of the lambdas's. This means that motivation2_112, for example, is never fixed to zero, which would then also change the IF and thus the dual values of the MP. Columns are currently being added, but they are not fixed to zero by multiplication with lambda, which means that the solver in the MP always sets motivation to 1 and thus the target coefficient always remains at zero. 

    I am not sure what you mean. Could you please provide an explicit example of what you think is wrong and how the model should look like? Do you need to add a multiplication of a \(\lambda\) and another variable?

    -------------------------------------------

    There is another question from my previous comment. In the \(\texttt{addColumn}\) function you construct an empty Column object which you use, when adding the variable \(\texttt{newvar}\). What is the purpose of this function \(\texttt{addColumn}\)? Since the Column object is empty, you could just add \(\texttt{newvar}\) in the \(\texttt{modifyConstraints}\) function before going into the loops. I see that you create \(\texttt{rounded_ScheduleList}\) in \(\texttt{addColumn}\) but do nothing with it. Is it on purpose? Is it just old code?

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman I guess thats just some old code:

     

    Regarding your first part:

     

    Okay , itry to give a proper example. Let's take a look at the constraints qc0 and qc1 as well as R0, R1 and R2. Let's assume that the initial solution of the MP, which we define in the code, is that all motivation values are set to zero (at this point I don't know whether this has been implemented correctly in the code, as an objective of zero is always set). This results in an objective value of 21, so the constraints look like this:

     

    qc0: slack[1,2] + [ motivation_i[1,1,2,1] * lmbda[1,1]

       + motivation_i[2,1,2,1] * lmbda[2,1]

       + motivation_i[3,1,2,1] * lmbda[3,1] ] >= 1

    qc0: slack[1,2]  + 0*1+0*1+0*1  >= 1

     qc1: slack[1,3] + [ motivation_i[1,1,3,1] * lmbda[1,1]

       + motivation_i[2,1,3,1] * lmbda[2,1]

       + motivation_i[3,1,3,1] * lmbda[3,1] ] >= 0

    qc1: slack[1,3]  + 0*1+0*1+0*1  >= 1

    R0: lmbda[1,1] = 1

    R0: 1 = 1

    R1: lmbda[2,1] = 1

    R1: 1 = 1

    R2: lmbda[3,1] = 1

    R2: 1 = 1

     

    Now we calculate the dual values and pass them to the symmetric SPs. This results in the following optimal solution for \(motivation^*={(3, 1, 1, 2): 0.5, (3, 1, 2, 2): 0.0, (3, 1, 3, 2): 0.0}\). This results in the following in the MP, for example:

    qc0: slack[1,2] + [ motivation_i[1,1,2,1] * lmbda[1,1]

       + motivation_i[2,1,2,1] * lmbda[2,1]

       + motivation_i[3,1,2,1] * lmbda[3,1] ] + schedule[1,2]* lmbda[1,2] + schedule[2,2]* lmbda[2,2]+  schedule[3,2]* lmbda[3,2]>= 1

    qc0: slack[1,2]  + 0*0.5+0*0.5+0*0.5 + 0.5*0.5 + 0.5*0.5 + 0.5*0.5  >= 1

     qc1: slack[1,3] + [ motivation_i[1,1,3,1] * lmbda[1,1]

       + motivation_i[2,1,3,1] * lmbda[2,1]

       + motivation_i[3,1,3,1] * lmbda[3,1] ] + schedule[1,2]* lmbda[1,2] + schedule[2,2]* lmbda[2,2]+ schedule[3,2]* lmbda[3,2]>= 0

    qc0: slack[1,3]  + 0*0.5+0*0.5+0*0.5 + 0*0.5+0*0.5+0*0.5   >= 1

    R0: lmbda[1,1]+  lmbda[1,2]= 1

    R0: 0,5 + 0,5 = 1

    R1: lmbda[2,1] + lmbda[2, 2] = 1

    R1: 0,5 + 0,5  = 1

    R2: lmbda[3,1] + lmbda[3,2]= 1

    R2: 0,5 + 0,5 = 1

    This no longer results in an objective value of 21 but lower because the \(motivation\) is now \(\ge0\) in some shifts. Since \(\lambda\) is now also relaxed, there can also be fractional allocations. It should be noted in the example that I do not know exactly whether the solver now sets \(\lambda\) to 0.5 in iteration 1 of the CG or possibly to another value. But I hope it is now clear what should happen. The current problem is that in the start solution the values of \(motvation\) are not zero, but 1 everywhere. And the second problem is that no new \(\lambda\)'s are added to the new columns in the iteration. Maybe this post will also help you to understand. Thank you very much for your help.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Thanks for the explanation.

    So you don't want to add \(\texttt{newvar}\) in a linear fashion to the quadratic constraint but you want to add it to the quadratic constraint by multiplying it with a \(\lambda\) variable.

    You can achieve that by

    qexpr.add(new_var * self.lmbda[index1, index2], new_coef)

    where \(\texttt{index1}\) and \(\texttt{index2}\) are the respective correct indices that you would have to somehow determine in your code.

    What confuses me is that when you add the new variable, you name it

    colName = f"ScheduleUsed[{index},{iter}]"

    so you use different indices from the ones you used to define the \(\lambda\) variables where you use \(\texttt{nurses}\) and \(\texttt{roster}\) indices

    self.lmbda = self.model.addVars(self.nurses, self.roster, vtype=gu.GRB.BINARY, lb=0, name='lmbda')

    but from your last comment, it looks like the \(\texttt{ScheduleUsed}\) and \(\lambda\) variables should use the same indices.

    I think that you need to figure which indices you have to use for \(\texttt{newvar}\) (called "ScheduleUsed") and then you can multiply it with the corresponding \(\lambda\) variables as stated above.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman

    Thank you very much, it's good that we are now on the same page. Unfortunately, I don't know where in the COde I need to implement your suggestion. I have put my current code in the OP but it cannot be executed without errors. What exactly do I have to adapt? I get this error:

        qexpr.add(new_var * self.lmbda[self.nurses, self.roster], new_coef)
                            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: unhashable type: 'list'

    And of course you are right about the indexes. I have replaced it in my code. However, the current problem is that a new column has \(r=1\) as the roster index after the first iteration, but it is actually the second roster, so it should be \(r=2\). With

    colName = f "Schedule[{self.nurses},{self.roster+1}]"

    it obviously does not work.
    Thank you very much

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    The issue I see is the following:

    In \(\texttt{addColumn}\) you call

    colName = f"Schedule[{self.nurses},{self.roster}]"

    However, \(\texttt{self.nurses}\) and \(\texttt{self.roster}\) are lists of indices. So to me it looks like at this point you need to somehow determine a valid index \(\texttt{nurseindex}\) and \(\texttt{rosterindex}\) out of the \(\texttt{self.nurses}\) and \(\texttt{self.roster}\) lists. Once you have these indices you can save them

     self.newvar = self.model.addVar(vtype=gu.GRB.CONTINUOUS, lb=0, column=Column, name=colName)
    self.nurseindex = nurseindex
    self.rosterindex = rosterindex

    and use them in \(\texttt{modifyConstraints}\)

    qexpr.add(new_var * self.lmbda[self.nurseindex, self.rosterindex], new_coef)

    However, I don't know how you have to determine the nurse and roster index. This is something you have to find out.

    Currently you are passing \(\texttt{itr}\) and \(\texttt{index}\) to \(\texttt{addColumn}\) which is not used at all. Maybe you can determine the nurse and roster index before calling \(\texttt{addColumn}\) and pass it as arguments instead of \(\texttt{itr}\) and \(\texttt{index}\).

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman I tried something out (see my code), but sadly it yields this error:

        qexpr.add(new_var * self.lmbda[nurseindex, rosterindex], new_coef)
                            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
    KeyError: (0, 0)

    Any idea why?

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    A KeyError means that you are trying to access the \(\texttt{self.lmbda}\) tupledict at an index that is not present, namely

    self.lmbda[0, 0]

    but according to the code posted in your original post, the lists \(\texttt{self.nurses}\) and \(\texttt{self.roster}\) are given as

    [1, 2, 3]
    [1]

    In this Knowledge Base article we discuss how to debug KeyErrors.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    @Jaromił Najman I thought I had it figured out with the two indices nursesIndex and rosterIndex, but now I have this KeyError: 

     

     

    qexpr.add(new_var * self.lmbda[self.nurseIndex, self.rosterIndex + 1], new_coef)

                            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    KeyError: (1, 2)

     

    I realize at this point that the problem is that according to def __init__(self):

    self.roster intitally is only [1]. How do I get this index to dynamically adjust with the number of iterations without calling master.buildModel() every time, which would delete previous columns. I tried to pass the current itr in the addColumn() function, but somehow this value is not passed to __init__ (see output of the print() function).

     My idea would be something like this. I would create this function:

     

    class Masterproblem()
    ....
    def updateRoster(self):
    self.roster = list(range(1,self.current_iteration + 2))

     

    Which I would than call after itr += 1.

    Would that work? Currently I can't test the code, earliest tomorrow.

     

    And another question remains unanswered, on how I set the initial rosters, rather than setting a start solution. Although I transfer a start solution with .start, I actually want to initialise the model with initial columns where the motivation is 0 for all nurses. How do I have to adapt my SetStartSolution()?

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Would that work? Currently I can't test the code, earliest tomorrow.

    No, this would not work, because at this point you already created the optimization variables with the old \(\texttt{roster}\) list. To make it work, you would also have to add a new variable to each variable tupledict which has a \(\texttt{roster}\) index.

    One easier way would be to create too many variables a priori, i.e., set the initial \(\texttt{self.roster}\) list to something already quite big, e.g., \(1,\dots,10\). Here, I assumed that \(\texttt{itr}\) will not be bigger than \(10\). This way, you would create too many variable of which many would be unused. This is not an issue performance wise even if you have 100 unused variables (it may become an issue if you have MANY more). Since all variables are already created, you can then access the respective tupledicts and add the variables to affected constraints. During model creation you would have to adjust the constraint construction to only use the first entry of the \(\texttt{roster}\) list to keep the current code valid, i.e., create an initial \(\texttt{rosterintial}\) list which would only hold \([1]\).

    A different way would be to re-write the whole code and take this design flaw into account when structuring your code. This would be quite a bit of work but with all the knowledge you have now about the encountered issues, it might be the best long term way. If you plan to use this code for a bigger long term project, then I would recommend considering this option.

    And another question remains unanswered, on how I set the initial rosters, rather than setting a start solution. Although I transfer a start solution with .start, I actually want to initialise the model with initial columns where the motivation is 0 for all nurses. How do I have to adapt my SetStartSolution()?

    You don't have to set any start solution. This is an optional feature and not needed to guarantee convergence. You can even also set a partial start solution, i.e., provide start values only for a subset of variables, see How do I use MIP start?

     

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman 

    Thanks for the answer. I also came across the idea of defining a "maximum" index self.roster. For this I now pass the max_itr as a value to the MasterProblem class. Unfortunately, I then always get this problem.

    line 74, in solveRelaxModel
        self.model.optimize()
      File "src\\gurobipy\\model.pxi", line 893, in gurobipy.Model.optimize
    gurobipy.GurobiError: Constraint Q not PSD (diagonal adjustment of 1.0e+00 would be required). Set NonConvex parameter to -1 or 2 to solve model.

    I suspect this is due to the point you also mentioned:

    Since all variables are already created, you can then access the respective tupledicts and add the variables to affected constraints. During model creation you would have to adjust the constraint construction to only use the first entry of the roster list to keep the current code valid, i.e., create an initial rosterintial list which would only hold [1].

    Unfortunately, I don't know how to modify the code.

     

    My second question: Isn't it a prerequisite for column generation to initialize the model with a solution in the pre-iteration that is feasible but not "optimal", f.e. too high? If I now set motivation to 1 everywhere and add columns, then the model will never choose these columns via \(\lambda\), because with \(motivation=1\) an objective of zero is always set, which cannot be improved in any case. However, if I initialize the model with \(I\) columns with \(motivation=0\), then the objective value is 21 in my example and can then be improved by the new columns. With .start, however, I only define a start, which is then solved again when master.solveRelaxModel() is called for the first time so that a target function value of 0 results. Or am I misunderstanding this? How can I add these initial columns to the code?

     

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Unfortunately, I don't know how to modify the code.

    You will need to add another list

    # should hold all entries up to max_itr
    self.roster = list(range(1, self.iteration + 2))
    # should be the original roster list that you used before
    self.rosterinitial = list(range(1, self.iteration + 2))

    Then use \(\texttt{self.roster}\) when constructing variables, but use \(\texttt{self.rosterinitial}\) when constructing constraints.

    The error states that some of your quadratic constraints are nonconvex. I think this might happen, because you keep adding more and more quadratic terms. I don't know whether this is expected (and correct) for your application and model. You need to find out whether this behavior is actually expected.
    As is stated in the Error message, you can just set the NonConvex parameter to make Gurobi solve the nonconvex model.

    Isn't it a prerequisite for column generation to initialize the model with a solution in the pre-iteration that is feasible but not "optimal", f.e. too high?

    I don't remember enough about column generation to state something useful here. From my understanding, you always solve your model to global optimality, so it should not matter whether you provide a starting solution or not. This is again something, you might want to figure out before proceeding.

    If I now set motivation to 1 everywhere

    Note that you do not fix the motivation variables to \(1\). You only provide a starting solution. This is a big difference. A starting solution just provides a possibly feasible point to the solver, it does not fix any variables to the given values. This means, that if there is a better solution with motivation set to \(0\) then Gurobi will find it independent of whether you provided a start solution with value \(1\).

    However, if I initialize the model with columns with , then the objective value is 21 in my example and can then be improved by the new columns.

    This sounds to me a bit like if you would like to FIX variables to a value and not just provide a starting solution. Please double check whether this is true.

    You can fix a variable to a value by adding a constraint.

    model.addConstr(var == value)

    or by setting its LB and UB attributes.

    Note that if you want to unfix a variable, you will have to remove the equality constraint and/or reset the LB and UB values.

    With .start, however, I only define a start, which is then solved again when master.solveRelaxModel() is called for the first time so that a target function value of 0 results.

    As described above, this is the correct behavior, because a MIP start is only an optional help for the solver, see Documentation of MIP starts.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman Thank you. That did the trick. The model is now running smoothly. However, one thing remains strange. Somehow the columns are only added to qc20, although the values should be added to every "qc.." constraint. Why is this the case? From my understanding, the columns should be added everywhere. Also lmbda[] doesnt not increase with every iteration.

     R0: lmbda[1,1] = 1
     R1: lmbda[2,1] = 1
     R2: lmbda[3,1] = 1
     qc0: slack[1,2] + [ motivation_i[1,1,2,1] * lmbda[1,1]
       + motivation_i[2,1,2,1] * lmbda[2,1]
       + motivation_i[3,1,2,1] * lmbda[3,1] ] >= 1
    ....
     qc20: slack[1,1] + [ motivation_i[1,1,1,1] * lmbda[1,1]
       + motivation_i[2,1,1,1] * lmbda[2,1]
       + motivation_i[3,1,1,1] * lmbda[3,1] + lmbda[1,2] * Schedule[1,2]
       + lmbda[1,3] * Schedule[1,3] + lmbda[2,2] * Schedule[2,2]
       + lmbda[2,3] * Schedule[2,3] + lmbda[3,2] * Schedule[3,2]
       + lmbda[3,3] * Schedule[3,3] ] >= 2


    Secondly: As I understand it, it is fixed, see https://or.stackexchange.com/a/11620/11914. I have now changed setStartSolution() to 

    def setStartSolution(self):
        for i in self.nurses:
            for t in self.days:
                for s in self.shifts:
                    self.model.addConstr(self.motivation_i[i ,t, s, 1] == 0)

    changed. Now the dual values are logically different. But now there is this problem. I have changed the  NonConvex parameter to both -1 and 2, but unfortunately, neither helped.

      File "G:\...\test.py", line 267, in <module>
        master.solveRelaxModel()
    File "G:\...\test.py", line 76, in solveRelaxModel
        self.model.optimize()
      File "src\\gurobipy\\model.pxi", line 893, in gurobipy.Model.optimize
    gurobipy.GurobiError: Constraint Q not PSD (diagonal adjustment of 1.0e+00 would be required). Set NonConvex parameter to -1 or 2 to solve model.
    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Somehow the columns are only added to qc20, although the values should be added to every "qc.." constraint. Why is this the case?

    We discussed this already a few posts before. In your \(\texttt{modifyConstraints}\) code, you still have

    return newcon

    which aborts the function early.

    I have changed the  NonConvex parameter to both -1 and 2, but unfortunately, neither helped.

    Where and how did you set the NonConvex parameter? I see that you have set it in one of the 3 solve functions that you have. Are you sure that it put it at the right spot? Looks like it is missing in \(\texttt{solveRelaxModel}\).

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman 

    Sorry, i have must overlooked that. Looks good now regarding the qc constraints. However is still dont know how to add the new \(\lambda\)'s of each new iteration to R0, R1 and R2. Do you know how?

    I feel very stupid. The output even tells me where to set the parameter. Now I have set the NonConvex parameter in solveRelaxModel() to 2 (-1 yields the same error as before). Unfortunately, I can no longer calculate the duals. But I am surprised that the duals can be calculated in iteration 1, but not in two. What could be the reason for this?

        Pi_cons_lmbda = self.model.getAttr("Pi", self.cons_lmbda)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "src\\gurobipy\\model.pxi", line 1912, in gurobipy.Model.getAttr
      File "src\\gurobipy\\attrutil.pxi", line 151, in gurobipy._gettypedattrlist
    gurobipy.GurobiError: Unable to retrieve attribute 'Pi'

    I really appreciate your help.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Sorry, i have must overlooked that.

    No problem, the thread got really long, so that's understandable.

    However is still dont know how to add the new \(\lambda\)'s of each new iteration to R0, R1 and R2. Do you know how?

    What are R0, R1, and R2? Are those linear constraints? If yes, you can determine linear constraints in the \(\texttt{addColumn}\) function and directly add the new variables to the linear constraints.

    def addColumn(self, newSchedule):
            self.nurseIndex = index
            self.rosterIndex = itr + 1
            self.newvar = {}
            colName = f"Schedule[{self.nurseIndex},{self.rosterIndex}]"
            newScheduleList = []
            for i, t, s, r in newSchedule:
                newScheduleList.append(newSchedule[i, t, s, r])
    ############################
    # here you pass empty lists when generating the Column object
    # you could pass gu.Column([1,1,1], [R0, R1, R2])
    # where you determined the LINEAR constraints R0, R1, R2 before
    ############################ Column = gu.Column([], []) self.newvar = self.model.addVar(vtype=gu.GRB.CONTINUOUS, lb=0, column=Column, name=colName) self.current_iteration = itr self.model.update()

    Unfortunately, I can no longer calculate the duals. But I am surprised that the duals can be calculated in iteration 1, but not in two. What could be the reason for this?

    The reason is that in iteration 1, your model is convex and it is possible for Gurobi to compute duals for convex models. It is however not possible to compute duals for a nonconvex model. Since, your models becomes nonconvex after first iteration, it is no longer possible to retrieve the duals.

    What you might do is to fix one of the product variables \(\lambda \cdot \text{schedule}\) to their optimal solution value and re-solve the model. Since you fix one of the product variables, the model becomes linear and you can retrieve the duals again. However, before doing that, you should double check that your model is indeed intended to become nonconvex and whether you really need the Dual values. Maybe it is possible for you to somehow compute the dual values or maybe the reduced cost values by hand from the information you have?

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman

    Is that what you mean?

    def addColumn(self, newSchedule):
    self.nurseIndex = index
    self.rosterIndex = itr + 1
    self.newvar = {}
    colName = f"Schedule[{self.nurseIndex},{self.rosterIndex}]"
    newScheduleList = []
    gu.Column([1, 1, 1], ['lmb0', 'lmb1', 'lmb2'])
    for i, t, s, r in newSchedule:
    newScheduleList.append(newSchedule[i, t, s, r])
    Column = gu.Column([], [])
    self.newvar = self.model.addVar(vtype=gu.GRB.CONTINUOUS, lb=0, column=Column, name=colName)
    self.current_iteration = itr
    self.model.update()


    Unfortunately, I don't quite understand why the model was previously solvable with the assumed parameters over many iterations if nothing has changed in the objective value, but now this is no longer the case by fixing the first column. Could you explain this in more detail?

    I also don't understand in the current code, the columns are initialized empty, are the optimal values from ScheduleCuts passed to the model at all? Is there something like model.write("test.lp") where you can also see the values of the variables?

    And why are the lmb-constrains enumerated but the demand one not? Whenever i call self.model.getConstrs() i am not able to retrieve those variables.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Is that what you mean? 

    No, there are multiple issues with this function as you posted.

    Values for \(\texttt{index}\) and \(\texttt{itr}\) are not passed as arguments. This means that these values will be just some values that previous happened to have this name.

    You create a Column object

    gu.Column([1, 1, 1], ['lmb0', 'lmb1', 'lmb2'])

    but you pass some variable names as constraint list. Please have a look at the Column object documentation to understand the constructor. The idea is to retrieve linear constraints (if they are linear), put them in a list, and construct the Column object.

    Then, you are not using the Column you construct, because you use the empty column

    Column = gu.Column([], [])

    Unfortunately, I don't quite understand why the model was previously solvable with the assumed parameters over many iterations if nothing has changed in the objective value, but now this is no longer the case by fixing the first column. Could you explain this in more detail?

    What do you mean by not solvable? Do you mean that the model is declared infeasible? In this case, please refer to How do I determine why my model is infeasible?

    I also don't understand in the current code, the columns are initialized empty, are the optimal values from ScheduleCuts passed to the model at all?

    Do you mean in the code \(\texttt{addColumn}\)? In \(\texttt{addColumn}\), you are just creating a new variable \(\texttt{newvar}\) and add it to no linear constraint or set any values for it.

    Is there something like model.write("test.lp") where you can also see the values of the variables?

    Yes, you can use the .sol format to write a solution file

    model.write("solution.sol")

    Note that this is only possible when at least one feasible solution is available.

    And why are the lmb-constrains enumerated but the demand one not?

    When you take a look at your code

    def generateConstraints(self):
            for i in self.nurses:
                self.cons_lmbda[i] = self.model.addConstr(gu.quicksum(self.lmbda[i, r] for r in self.rosterinitial) == 1, name = "lmb")
            for t in self.days:
                for s in self.shifts:
                    self.cons_demand[t, s] = self.model.addConstr(
                        gu.quicksum(self.motivation_i[i, t, s, r]*self.lmbda[i, r] for i in self.nurses for r in self.rosterinitial) +
                        self.slack[t, s] >= self.demand[t, s], name = "demand")
            return self.cons_lmbda, self.cons_demand

    you can see that all constraints are called the same. If you want unique names you have to make sure to provide those

    ...,name = "lmb_"+str(i))
    ...,name = "demand_"+str(t)+"_"+str(s))

    Whenever i call self.model.getConstrs() i am not able to retrieve those variables.

    How are you trying to retrieve variables when you are calling the getConstrs() method?

    I think at this point, it might make sense for you to have a look (if you didn't already) at our Advanced Python Modelling Webinar (and maybe other webinars) to strengthen the understanding of the objects like tupledicts, Columns, constraints etc. I think this might save you quite a lot of time waiting for an answer in this thread.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman Thanks for the clarification. I have now decided to add the \(\lambda\)'s via expr.add and it "works" too, although only 90%. I have added this function:

    def addLambda(self, index, itr):
    self.nurseIndex = index
    self.rosterIndex = itr + 1
    for i in self.nurseIndex:
    self.newlmbcoef = 1.0
    current_lmb_cons = self.cons_lmbda[i]
    expr = self.model.getRow(current_lmb_cons)
    new_lmbcoef = self.newlmbcoef
    expr.add(self.lmbda[self.nurseIndex, self.rosterIndex], new_lmbcoef)
    rhs_lmb = current_lmb_cons.getAttr('RHS')
    sense_lmb = current_lmb_cons.getAttr('Sense')
    name_lmb = current_lmb_cons.getAttr('ConstrName')
    newconlmb = self.model.addConstr(expr, sense_lmb, rhs_lmb, name_lmb)
    self.model.remove(current_lmb_cons)
    self.cons_lmbda[i] = newconlmb

    Now, after the zeroth iteration, i.e. the initialization of the RMP:

    lmb(1): lmbda[1,1] = 1
    lmb(2): lmbda[2,1] = 1
    lmb(3): lmbda[3,1] = 1

    The following constraints:

    lmb(1): 2 lmbda[1,1] + lmbda[1,2] + lmbda[2,1] + lmbda[2,2] + lmbda[3,1] + lmbda[3,2] = 1
    lmb(2): lmbda[1,1] + lmbda[1,2] + 2 lmbda[2,1] + lmbda[2,2] + lmbda[3,1] + lmbda[3,2] = 1
    lmb(3): lmbda[1,1] + lmbda[1,2] + lmbda[2,1] + lmbda[2,2] + 2 lmbda[3,1] + lmbda[3,2] = 1

    Where does the 2 in lmbda[1,1] come from, for example? The coeff in the constraint of every \(\lambda\) should be one accros the board. And why are there so many lmbda[]? Actually, lmb(1) should look like this:

    lmb(1): lmbda[1,1] + lmbda[1,2] == 1

     

    However, I am still facing the following problem:

    gurobipy.GurobiError: Unable to retrieve attribute 'Pi'

    However, i dont know why, because the \(\lambda\) constraints lmb(..) are linear, and they should remain linear, even after calling addLambda(). I understand that all demand(t,s) constraints are quadratic and therefore "QCPI" is necessary as atrribute, but with lmb(), respectively self.cons.lmbda I don't understand it. If this problem can be solved, then I can check if the columns in the solution (with the ".sol" files in the code) are used correctly.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    here does the 2 in lmbda[1,1] come from, for example? The coeff in the constraint of every should be one accros the board. And why are there so many lmbda[]? Actually, lmb(1) should look like this:

    You are using variable \(\texttt{i}\) to loop over \(\texttt{self.nurseIndex}\) but then you use

    expr.add(self.lmbda[self.nurseIndex, self.rosterIndex], new_lmbcoef)

    to add \(\texttt{lmda}\) to the expression. So you add the whole \(\texttt{self.nurseIndex}\) list in every loop iteration.

    However, i dont know why, because the constraints lmb(..) are linear, and they should remain linear, even after calling addLambda(). I understand that all demand(t,s) constraints are quadratic and therefore "QCPI" is necessary as atrribute, but with lmb(), respectively self.cons.lmbda I don't understand it. If this problem can be solved, then I can check if the columns in the solution (with the ".sol" files in the code) are used correctly.

    If the whole model is a nonconvex one, you will not be able to retrieve any Pi values. Pi values are only available for continuous convex model. Is your model nonconvex?

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman

    Thanks, I could have figured it out myself. I fixed it.

    Yes, I think the model is currently non-convex, because of the quadratic formulation of \(\lambda \cdot motivation\), so the parameter model.Params.NonConvex is also set to 2. But I'm also sure that it shouldn't actually be "non-convex". This has also been confirmed by the great Rob Pratt on or.stackexchange.com. In response to my question

    So, I just read somewhere, that this specific MP is not linear but rather a MIQCP, because the multiplication of \(\lambda \cdot motivation\). Is that true or is the model above still linear?

    he said:


    The model is linear. When motivation is multiplied by \(\lambda\), it is not a variable but rather the fixed value of motivation in column \(r\).

    However, because a new variable is currently created in the code with the addColumn() function and then multiplied by \(\lambda\) with modifyConstraint(), this logically becomes a multiplication of two variables and the model is therefore non-convex. This means that no dual values of the updated model can be calculated after iteration 1.

    How do I have to adapt the code so that the optimal values of \(motivation^*\) from the SPs are now multiplied with the \(\lambda\)'s instead of defining a new variable? So if, for example, the optimal value in iteration 1 from the SP(1) returns the vector \(motivation_{1ts}^*=[0,1,0,0.9,0,0,0.9,....]\), then the constraints demand(1,1)-demand(2,1) should look like this:

    demand(1,1): slack[1,1] + [ motivation_i[1,1,1,1] * lmbda[1,1]
       + motivation_i[2,1,1,1] * lmbda[2,1]
       + motivation_i[3,1,1,1] * lmbda[3,1] + lmbda[1,2] * 0....] >= 2
     demand(1,2): slack[1,2] + [ motivation_i[1,1,2,1] * lmbda[1,1]
       + motivation_i[2,1,2,1] * lmbda[2,1]
       + motivation_i[3,1,2,1] * lmbda[3,1] + lmbda[1,2] * 1 ....] >= 1
     demand(1,3): slack[1,3] + [ motivation_i[1,1,3,1] * lmbda[1,1]
       + motivation_i[2,1,3,1] * lmbda[2,1]
       + motivation_i[3,1,3,1] * lmbda[3,1] + lmbda[1,2] * 0 .... ] >= 0
     demand(2,1): slack[2,1] + [ motivation_i[1,2,1,1] * lmbda[1,1]
       + motivation_i[2,2,1,1] * lmbda[2,1]
     + motivation_i[3,2,1,1] * lmbda[3,1] + lmbda[1,2] * 0.9 .... ] >= 1

    Note that the values for i=2 and i=3 are dropped.

    How do I have to adapt my code for this? I think this will bring us a lot closer to making the column generation work.

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    After solving a subproblem, you can access the solution point via, e.g.,

    motivation_i[2,1,1,1].X

    see the X attribute.

    From your explanation, it sounds like in your \(\texttt{addColumn}\) function, you actually do not have to add a new variable but rather save the solution point values for \(\texttt{motivation_i}\) variables. You can first save these values to some dictionary, e.g.,

    self.motivation_i_solution[2,1,1,1] = self.motivation_i[2,1,1,1].X

    You can then use these values in \(\texttt{modifyConstraint}\)

    qexpr.add(self.motivation_i_solution[i,j,k,l] * self.lmbda[self.nurseIndex, self.rosterIndex], new_coef)

    Of course you will have to generalize the above snippets and possibly adjust the indices, but this should give you an idea of how it should be tackled.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman Would something like this work? Sadly my PC with PyCharm broke down today and i cant test it right now.

    def addColumn(self, newSchedule):
    self.newvar["motivation_i"] = {}
    for i, t, s, r in newSchedule:
    self.newvar["motivation_i"][(i, t, s, r)] = newSchedule[i, t, s, r]
    self.model.update()
    def modifyConstraint(self, index, itr):
    self.nurseIndex = index
    self.rosterIndex = itr + 1
    for t in self.days:
    for s in self.shifts:
    qexpr = self.model.getQCRow(self.cons_demand[t, s])
    for i, j, k, l in self.newvar["motivation_i"]:
    if i != self.nurseIndex:
    new_coef = self.newvar["motivation_i"][(i, j, k, l)] * self.newvar["motivation_i"][
    (self.nurseIndex, t, s, self.rosterIndex)]
    qexpr.add(self.motivation_i[i, j, k, l] * self.lmbda[self.nurseIndex, self.rosterIndex],
    new_coef)
    rhs = self.cons_demand[t, s].getAttr('QCRHS')
    sense = self.cons_demand[t, s].getAttr('QCSense')
    name = self.cons_demand[t, s].getAttr('QCName')
    newcon = self.model.addQConstr(qexpr, sense, rhs, name)
    self.model.remove(self.cons_demand[t, s])
    self.cons_demand[t, s] = newcon

    Does this code also output the values and the respective \(\lambda\)'s in the .lp files?

    0
  • Jaromił Najman
    • Gurobi Staff Gurobi Staff

    Would something like this work?

    At a quick glance it looks OK. Please test it once your PC works again.

    Does this code also output the values and the respective \(\lambda\)'s in the .lp files?

    Yes, you should see the difference in the generated LP files.

    0
  • Carl Baier
    • Curious
    • Gurobi-versary
    • Thought Leader

    Jaromił Najman I was just able to test it and I observed the following:


    1) If I increase max_itr, then the reduced cost of the subproblem also changes, albeit marginally. I think this is because the \(\lambda\)'s contraints change, which also changes \(\mu_i\). Could that be the reason?

    2) A lot of \(motivation_{its}^r\) values are added to the objective function, as in this example. Is this normal?

    slack[1,1] + slack[1,2] + slack[1,3] + slack[2,1] + slack[2,2]
    + slack[2,3] + slack[3,1] + slack[3,2] + slack[3,3] + slack[4,1]
    .....
    + 0 motivation_i[3,1,1,16] + 0 motivation_i[3,1,1,17]
    + 0 motivation_i[3,1,1,18] + 0 motivation_i[3,1,1,19]
    + 0 motivation_i[3,1,1,20] + 0 motivation_i[3,1,1,21]
    ....

    3) Unfortunately, the demand(t,s) constraints remain unchanged, no matter how many iterations I perform. As a result, the dual values \(\pi_{ts}\) and also the objective function of the master problem do not change, since \(motivation_{its}^1=0\) still applies and thus the objective function value remains unchanged at 21. Why were the quadratic terms from \(\lambda \cdot\) newSchedule added in the old code, but not anymore? Perhaps because they are no longer quadratic terms? See the .lp here:

    lmb(3): lmbda[3,1] + lmbda[3,2] + lmbda[3,3] + lmbda[3,4] + lmbda[3,5]
    + lmbda[3,6] + lmbda[3,7] + lmbda[3,8] + lmbda[3,9] + lmbda[3,10]
    + lmbda[3,11] + lmbda[3,12] + lmbda[3,13] + lmbda[3,14] + lmbda[3,15]
    + lmbda[3,16] + lmbda[3,17] + lmbda[3,18] + lmbda[3,19] + lmbda[3,20]
    + lmbda[3,21] = 1
    demand(1,1): slack[1,1] + [ motivation_i[1,1,1,1] * lmbda[1,1]
    + motivation_i[2,1,1,1] * lmbda[2,1]
    + motivation_i[3,1,1,1] * lmbda[3,1] ] >= 2

     

    Any idea why that is?

    0

Please sign in to leave a comment.