• Gurobi Staff

Hi Vignesh,

I am trying to understand what you want to do but it is not clear to me.

In your first comment, you said that you want to run the "model" 10 times but then you say you want to embed 10 neural networks.

So to understand better:

- How many different neural networks do you have? (i.e. do you have one h5 file or several?)

- How many optimization models do you want to solve in total: 1 or 10?

Mayybe what you are looking for is some sort of multi-objective formulation but it is not clear to me. If you have a full description of the mathematical model you want to solve, it would be easier to help you.

• Gurobi Staff

Hi Vignesh,

We actually have released a Python package: gurobi-machinelearning which you can use for this.
You can find the GitHub here: Gurobi/gurobi-machinelearning
And documentation here: Gurobi Machine Learning (particularly see the installation instructions)

Here's the translated code (edit: with 10 output variables instead):

import gurobipy as gpfrom gurobi_ml import add_predictor_constrimport tensorflow as tftf_model = tf.keras.models.load_model("peaks_3x10.h5")print("loaded ML model")with gp.Model() as m:    input_vars = m.addVars(2, lb=-3, ub=3, vtype=gp.GRB.CONTINUOUS)    output_vars = m.addVars(        10,        lb=-gp.GRB.INFINITY,        ub=gp.GRB.INFINITY,        vtype=gp.GRB.CONTINUOUS,        name="output",    )    m.update()    pred_constr = add_predictor_constr(m, tf_model, input_vars, output_vars)    pred_constr.print_stats()    m.setObjective(output_vars.sum(), gp.GRB.MINIMIZE)    m.optimize()

Cheers,
David

Hi David,

Thanks for your response. Basically I am looking for AnnModel lib function to leverage for my problem. The code you mentioned top is using the general ml. Kindly can you specify if the AnnModel lib function is still being used separately.? Also can you confirm with the below line, can I loop over to generate 10 diff 10 embed_network_formulations..? It would be great if we can connect on call to resolve.?

with gp.Model() as m:

Just to add to my above comment, looks like the "with gp. model() as m" is performing the optimization separately  instead I need to optimize the 10 different models at a time.

Lets say : I have the obj as

max : c1*x1 + c2*x2 + h1*y1 + h2*y2

const-1: f(x1) = y1

const-2 : f(x2) = y2

x, y belongs to X lets say.

Here f(x1) and f(x2) are my ann models which are supposed to be in one optimization run instead of 2. So basically my f(x1) and f(x2) are not separable

• Gurobi Staff

Hi Vignesh,

Unfortunately, I'm not familiar with the library's source code.
It looks like the latest release was a while ago.

Please check out the library I linked above, as (I think) does the same thing and is frequently maintained.

Just to add to my above comment, looks like the "with gp. model() as m" is performing the optimization separately  instead I need to optimize the 10 different models at a time.

No, this creates a single gurobipy model for all 10 ML-models (see my updated example); hence they are all optimized simultaneously.

Cheers,
David

Hi David,

In your updated code, it looks like you're optimizing the output_vars.sum() with output_vars being considered as 10 diff ML-models outputs and finally optimizing the sum of these 10. Rather I am wondering if we can do it individually and access the individually optimized model.

For example: Lets say I have supply-demand constraint problem. Here is how I modeled it

input_vars = m.addVars(2, lb=-3, ub=3, vtype=gp.GRB.CONTINUOUS)output_vars = m.addVars(    10,    lb=-gp.GRB.INFINITY,    ub=gp.GRB.INFINITY,    vtype=gp.GRB.CONTINUOUS,    name="output",)m.update()pred_constr = gurobi_ml.keras.add_keras_constr(m, tf_model, input_vars, output_vars)pred_constr.print_stats()m.setObjective(    (        gp.quicksum(            demandProb["p"][p] * dcost * distanceCF[c, f] * x[p, c, f] for c, f in cf        )        + gp.quicksum(tfcost * (1 - setup[current, f]) * y[f] for f in facility)        + output_vars[p]        for p in range(10)    ),    GRB.MINIMIZE,)# x[p,c,f] <= bpi   #supply constraints# x[p,c,f] >= apj   # demand constraintsm.optimize()

Comments: As shown above this is what I am expecting to be done. If you see my overall model is getting optimized for every ML models individually with output_vars[p] for every p in range of 10.

The question here is : Since output_vars is being created 10 times by considering 10 ML models. However I am not able to optimize at p level(I mean individual level) in one gurobi model.

Also is there a way to set up my supply-demand constraints to be considered in the ML model.? or the way I have written is enough to be considered at every node in the ML model).?

Hi David,

Any idea about my above query.?

Thanks!

Vignesh

• Gurobi Staff

Hi Vignesh,

I guess you can add a model with the supply-demand constraints and variables as well, just normally using standard gurobipy (see Python Examples for simple examples and modeling examples for more complex supply network design).
I don't know that much on this topic so I'm not sure how you can only optimise each level. Perhaps there is a miscommunication between our terms. The package translates the keras model to a mixed integer program (MIP) this is the single $$\texttt{gp.Model}$$. The translation depends on how many input variables and output variables you choose (of course the keras model doesn't allow any number of these). The objective of this MIP can be changed but the constraints represent the keras model in the variables you define (not optimised just satisfied as constraints).

Cheers,
David

• Gurobi Staff

Hi Pierre,

Thanks for taking my question.

1. I have got 1 neural network i.e. one h5 file.

2. Basically I will have to run this NN model n times(per say in this case 10 times) and optimize every model in the loop.

Here is my code which is currently running. But I do see my model running into infeasibility.

import gurobipy as gpfrom gurobi_ml import add_predictor_constrimport tensorflow as tffrom gurobipy import GRBopt_model = gp.Model()tf_model = {}for p in range(10):    tf_model[p] = tf.keras.models.load_model("model.h5")    print(tf_model[p])state = 0pred_constr = {}facilities = [*range(1, 24)]def input_features():    input_vars1 = []    f = opt_model.addVars(facilities, lb=50, ub=300, vtype=GRB.CONTINUOUS)    y = opt_model.addVars(facilities, lb=0, ub=1, vtype=GRB.BINARY)    # x = opt_model.addVars(facilities, lb =0, ub = 300,vtype=GRB.CONTINUOUS)    # w = opt_model.addVars(cf, lb =0, ub = 300,vtype=GRB.CONTINUOUS)    input_vars1.extend(f.values())    input_vars1.extend(y.values())    opt_model.update()    return input_vars1output_vars = opt_model.addVars(    10,    lb=-gp.GRB.INFINITY,    ub=gp.GRB.INFINITY,    vtype=gp.GRB.CONTINUOUS,    name="output",)opt_model.update()for p in range(10):    input_vars1 = input_features()    # output_vars = output_response()    pred_constr[p] = add_predictor_constr(        opt_model, tf_model[p], input_vars1, output_vars[p]    )    pred_constr[p].print_stats()opt_model.setObjective(gp.quicksum(output_vars[p] for p in range(10)), gp.GRB.MINIMIZE)opt_model.optimize()

This above code constructs 10 diff embed networks and solve for every network in a loop. I am kind of wondering why model is running into infeasibility few nodes and taking ever and ever. Ofcourse my h5 file is appropriate and with r2 value ~65%

Let me know your thoughts on my understanding and working functionality.

Thanks!
Vignesh
• Gurobi Staff

Hi Vignesh,

I don't think that the model above completely matches what you describe.

In the model I see that 10 networks would be added to the optimization problem and then you minimize the sum of the output of these networks.

What are the sizes of the networks that you are adding?

Indeed I don't see why this model would be infeasible. Could you please copy the log of Gurobi that you get?

Thanks,

Pierre

Here is the screenshot of my nn model-9, like-wise as you rightly pointed I have got 10 such models every model is the same with 46 feature variables and one output variable.

Also Here is the log of the model: This is taking ever and ever, hence I terminated after an hour or so

• Gurobi Staff

From the log it looks that the resulting model is very hard to optimize.

Unfortunately formulation with neural networks can be very challenging.

What you could try first is adding only one network and see how it goes.

Then adding 2, probably adding each network will make the optimization exponentially harder.

If you can and want to share the model I could have a look at it to see if there is something that can be done to solve it faster (since we have the python code, we would only need the h5 file).

I won't be able to look at it before the middle of next week though.

Best,

Pierre

Hi Pierre,

I am not sure how to upload the .h5 file here. Kindly can you share your email-id or any link to share the file.

Thanks!

Vignesh

• Gurobi Staff

Hi Vignesh

You can upload the file to a file-sharing platform of your choosing (e.g. Dropbox or Drive) and share the link with us here.

Cheers,
David

Here is my H5 model file.

https://www.dropbox.com/scl/fi/52nxrfbptpah0nyhub4cw/model_actual.h5?rlkey=w9luj2m5hfrtq8dgi3vqkgu9t&dl=0

Let me know if you can access this file.

Thanks!

Vignesh

Just to add to my above comments, I was thinking to do warm start my problem by providing a few known solutions to the nn model. But I am thinking what exactly the below line is doing in general.? does it constructs constraints at every node of the neural net.? kindly can you shed some light here.?

pred_constr = add_predictor_constr(m, tf_model, input_vars, output_vars)
• Gurobi Staff

Hi Vignesh,

It's indeed very tough to solve.

I could at least get some solutions by changing some parameters of Gurobi, namely with

opt_model.params.PreSOS1BigM = 1e8

opt_model.params.OBBT=3
opt_model.params.MIRCuts = 2

The first parameter is to be more aggressive in the formulation of the neural network the two others are to be very aggressive with some techniques that are usually useful for neural networks formulation.

With those it took me some minutes but I got some solutions.

I'll try to see if something else can be done but for some reason the problem is very tough even if I only add one neural network.

To answer your second question, yes we do a full model of the neural network with constraints representing each node of the neural network. You can look at "further reading" in our documentation for papers giving more details on how the networks can be formulated: https://gurobi-machinelearning.readthedocs.io/en/stable/mlm-further-reading.html

Our own formulation is described in the documentation also.

For reference, I use a slight variant of your code using Gurobi's matrix notations to avoid calling several times add_predictor_constr, this is the full code:

import gurobipy as gp
import tensorflow as tf
from gurobipy import GRB

print(tf_model)

# Parameters of the optimization model
facilities = 23
n_pred_constr = 10

opt_model = gp.Model()

(n_pred_constr,1),
lb=-gp.GRB.INFINITY,
ub=gp.GRB.INFINITY,
vtype=gp.GRB.CONTINUOUS,
name="output",
)

opt_model.update()
# First group of variables in the input matrix are continuous between 50 and 300
f = input_vars[:, :facilities]
f.LB = 50
f.UB = 300

# Second groupd of variables in the input matrix are binary
y = input_vars[:, facilities:]
y.UB = 1.0
y.VType = GRB.BINARY

opt_model.update()

opt_model, tf_model, input_vars, output_vars
)
pred_constr.print_stats()

# Setup objective
opt_model.setObjective(output_vars.sum(), gp.GRB.MINIMIZE)
opt_model.params.OBBT=3
opt_model.params.PreSOS1BigM = 1e8
opt_model.params.MIRCuts = 2
opt_model.optimize()

Thanks, Team @Pierre Bonami and @ David Torres Sanchez these are really helpful. Appreciate your help.

Hi Pierre Bonami, just a quick question: Basically I have trained my model after transforming my feature variables via z-transformation, the same .h5 model read into the optimization. I would like to understand how would gurobi know the model that is read into optimization was trained after feature variable transformation.? Do I need to perform any inverse-transformation before I feed the h5 model into the optimization.? Thanks for your valuable responses.

• Gurobi Staff

Hi Vignesh,

I am not so familiar with how things are stored in the .h5 format. But if those transformation are not part of the neural network and the Keras model that is created from reading the h5 file, Gurobi wouldn't be able to know about it. So you would have to do the transformations.

Note that for Keras, we don't currently support many types of layers. In particular, we don't support any of the preprocessing layers. There is an issue in Gurobi ML about adding support for it:

https://github.com/Gurobi/gurobi-machinelearning/issues/10

If there are some particular layers you would be interested in, you can add them there. And if you have some example, that would motivate us to implement them (not every layer can be implemented though).

(For scikit-learn we implement a number of preprocessing layers, then Gurobi ML can handle directly the transformation).