gurobipy model.addConstrs implementation issues
Answered
I discovered that `model.addConstrs` does introspection on the provided generator frame.
Example:
>>> model.addConstrs((10 < 20 for i in range(3, 5)), name="smth")
{3: ..., 4: ...}
The sole fact that you see keys "3" and "4" in the output is alarming. The generator does not output anything like that according to python standards: it is simply a black box yielding the value True twice. However, if I define an equivalent version of this generator
def some_generator():
yield True
yield True
it will not work with addConstrs and will fail with a non-descriptive error (tuple index out of range on python 3.9.9). Things become evident if you just make addConstrs fail by providing anything but a generator as an input:
model.addConstrs({}, name="smth")
Traceback (most recent call last):
File "src/gurobipy/model.pxi", line 3751, in gurobipy.Model.addConstrs
File "src/gurobipy/model.pxi", line 224, in gurobipy.Model.__genexpr_key
AttributeError: 'dict' object has no attribute 'gi_frame'
There is no reason to look into the gi_frame for a package like gurobipy. gi_frame is not even a part of python standard.
That said, it is really difficult to understand or predict or to use addConstrs: I do not even know now if the generator frame is being interpreted by cPython or gurobipy. Based on the observations above I would like to raise two issues with this function.
1. API doc should clearly state what does addConstrs do under the hood and what it accepts as an input
2. If the input is a generator type then it should support standard generators as defined in python language (period). If the input is something else having its own language then it should come with its own specs
-
Hi Artem,
I discovered that `model.addConstrs` does introspection on the provided generator frame.
Indeed it does. This is really the only way to pull off this particular bit of magic; as you correctly pointed out the Python standard doesn’t allow the introspection needed to extract the keys. So, gurobipy relies on a CPython implementation detail here.
That said, it is really difficult to understand or predict or to use addConstrs: I do not even know now if the generator frame is being interpreted by cPython or gurobipy. Based on the observations above I would like to raise two issues with this function.
I tend to agree, this method is not so easy to understand. To give some context: gurobipy is now ~15 years old; some features are there to support compact modelling statements in older versions of Python, and are maintained for users who rely on them. I would say that this particular method is now redundant, since f-strings arrived in Python 3.6 and made producing name strings very easy. For example this code from the documentation:
constrs = m.addConstrs(
(x[i,j] == 0 for i in range(4) for j in range(4) if i != j),
name='c'
)is exactly equivalent to this:
constrs = gp.tupledict({
(i, j): m.addConstr(x[i, j] == 0, name=f"c[{i},{j}]")
for i in range(4)
for j in range(4)
if i != j
})or this, if you don’t really need to keep track of the constraint objects returned:
for i in range(4):
for j in range(4):
if i != j:
m.addConstr(x[i, j] == 0, name=f"c[{i},{j}]")or this, to make better use of the Python standard library for this specific case:
for i, j in itertools.permutations(range(4), r=2):
m.addConstr(x[i, j] == 0, name=f"c[{i},{j}]")To me the latter examples are preferable; they are self-documenting and typically faster. I recommend using that style instead of addConstrs.
Regarding the docs: we’ll certainly take a look at what we can clarify, but the documentation does state that “The first argument to addConstrs is a Python generator expression”. There is a distinction between a generator expression (PEP289) and a generator (PEP255). addConstrs only accepts the former.
0 -
Thanks for a prompt response! I will use addConstr for now.
Saying that you accept "generator expression" as opposed to the "generator" is still confusing for me. Python does not have "expression" or "generator expression" type. So it raises those exact questions I posted above. If gbpy looks into expression only does this mean that it also interprets it? If no, does it look into AST or bytecode? Why this does not work?
def gen():
for i in range(3, 5):
yield True
addConstr(gen(), name="smth")0 -
Hi Artem,
The syntax of a generator expression in Python is defined here: https://docs.python.org/3/reference/expressions.html#generator-expressions. It's syntactically very similar to a list comprehension (though it's lazily evaluated) and is part of the Python language standard.
By contrast, what you’ve given in your example is a generator, which returns a generator iterator when called.
> If gbpy looks into expression only does this mean that it also interprets it? If no, does it look into AST or bytecode?
No, the Python interpreter interprets the expression, gurobipy only sees the resulting generator expression object which is passed to addConstrs. Extracting the keys correctly requires looking both at the local scope variables and the bytecode of the generator expression, which is why this only works in CPython. Inspecting the AST may also work, but at that point you're starting to implement a Python compiler ...
> Why this does not work?
It may be possible to make a generator work with a different implementation. gi_frame is specific to generator expressions so I guess it would need separate handling. I haven't looked into it. I would argue though that handling a generator here doesn't add much in terms of helping users of gurobipy write clear and concise code for mathematical programming models. This:
model.addConstrs(
gp.quicksum(x[i] for i in I[j]) <= 1
for j in J
)is a nice way to represent this mathematical construct:
\sum_{I_j} x_i \le 1 \forall j \in Jwhich is why addConstrs targets generator expressions. However, this:
def gen(x, I, J):
for j in J:
yield gp.quicksum(x[i] for i in I[j]) <= 1
model.addConstrs(gen(x, I, J))doesn't seem to me to have a lot of readability benefits over a plain for loop?
0
Please sign in to leave a comment.
Comments
3 comments