Need urgent help as Gurobi gives optimal weights value 0 for SVM mathematical model, Code is correct and not giving error , but not getting optimal value
Answered
Hi,
I am working on SVM optimization model for uncertainty in label. For the following equation:
Please find attached code:
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score
import gurobipy as gp
from gurobipy import GRB
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(X)
print(y)
# Step 2: Train the Robust SVM Model
def train_robust_svm(X, y, M, Gamma):
n = len(X)
p = len(X[0])
print(n)
print(p)
# Create a new Gurobi model
model = gp.Model("Robust_SVM_Model")
model.setParam('NumericFocus', 3)
# Add variables
w = model.addMVar(p, name="w", lb=-GRB.INFINITY)
b = model.addVar(name="b", lb=-GRB.INFINITY)
q = model.addVar(name="q", lb=0.0)
xi = model.addMVar(n, lb=0.0, name="xi")
phi = model.addMVar(n, lb=0.0, name="phi")
ri = model.addMVar(n, lb=0.0, name="ri")
si = model.addMVar(n, vtype=GRB.BINARY, name="si")
ti = model.addMVar(n, vtype=GRB.BINARY, name="ti")
# Add constraints
for i in range(n):
model.addConstr(q + ri[i] >= phi[i] - xi[i])
model.addConstr(xi[i] >= 1 - y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b))
model.addConstr(xi[i] <= 1 - y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b) + M * (1 - si[i]))
model.addConstr(xi[i] <= M * si[i])
model.addConstr(phi[i] >= 1 + y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b))
model.addConstr(phi[i] <= 1 + y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b) + M * (1 - ti[i]))
model.addConstr(phi[i] <= M * ti[i])
# Set objective function
model.setObjective(gp.quicksum(xi[i] for i in range(n)) + Gamma * q + gp.quicksum(ri[i] for i in range(n)), sense=GRB.MINIMIZE)
# Optimize the model
model.optimize()
model.write("C:/Users/manju/uncertaintylabelmodel_liverdisease.lp")
model.write("C:/Users/manju/uncertaintylabelmodel_liverdisease.mps")
# Retrieve the optimal values
weights = np.array([w[i].x for i in range(p)])
bias = b.x
return {'w': weights, 'b': bias, 'q': q.x, 'xi': np.array([xi[i].x for i in range(n)]),
'ri': np.array([ri[i].x for i in range(n)]), 'si': np.array([si[i].x for i in range(n)]),
'ti': np.array([ti[i].x for i in range(n)]), 'phi': np.array([phi[i].x for i in range(n)])}
# Train the robust SVM model
results = train_robust_svm(X_train, y_train, M=100000, Gamma=40)
# Step 3: Predict on the Testing Set
def predict(X, params):
decision_values = np.dot(X, params['w']) + params['b']
return np.sign(decision_values)
# Evaluate on the testing set
y_pred = predict(X_test, results)
# Evaluate Accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy on the test set: {accuracy}")
# Display the optimal values
print("Optimal Weight vector (w):", results['w'])
print("Optimal Bias (b):", results['b'])
print("Optimal q:", results['q'])
print("Optimal xi:", results['xi'])
print("Optimal ri:", results['ri'])
print("Optimal si:", results['si'])
print("Optimal ti:", results['ti'])
print("Optimal phi:", results['phi'])
I am getting below output:
Set parameter NumericFocus to value 3 Gurobi Optimizer version 10.0.2 build v10.0.2rc0 (win64) CPU model: 12th Gen Intel(R) Core(TM) i7-1250U, instruction set [SSE2|AVX|AVX2] Thread count: 10 physical cores, 12 logical processors, using up to 12 threads Optimize a model with 560 rows, 412 columns and 4640 nonzeros Model fingerprint: 0x47b9c1f5 Variable types: 252 continuous, 160 integer (160 binary) Coefficient statistics: Matrix range [4e-03, 1e+04] Objective range [1e+00, 1e+01] Bounds range [1e+00, 1e+00] RHS range [1e+00, 1e+04] Found heuristic solution: objective 393.1283257 Presolve time: 0.01s Presolved: 560 rows, 412 columns, 4640 nonzeros Variable types: 252 continuous, 160 integer (160 binary) Root relaxation: objective 5.800000e+01, 253 iterations, 0.00 seconds (0.01 work units) Nodes | Current Node | Objective Bounds | Work Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time 0 0 58.00000 0 4 393.12833 58.00000 85.2% - 0s H 0 0 58.0000000 58.00000 0.00% - 0s 0 0 58.00000 0 4 58.00000 58.00000 0.00% - 0s Explored 1 nodes (253 simplex iterations) in 0.11 seconds (0.01 work units) Thread count was 12 (of 12 available processors) Solution count 2: 58 393.128 Optimal solution found (tolerance 1.00e-04) Best objective 5.800000000000e+01, best bound 5.800000000000e+01, gap 0.0000% Accuracy on the test set: 0.2 Optimal Weight vector (w): [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] Optimal Bias (b): -1.0 Optimal q: 2.0 Optimal xi: [0. 0. 0. 2. 2. 2. 0. 0. 0. 0. 0. 0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 2. 0. 2. 0. 2. 2. 0. 0. 2. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 2. 0. 0. 0. 2. 2. 2. 0. 2. 0. 0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 2. 2. 0. 0. 0. 0. 0. 0.] Optimal ri: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] Optimal si: [0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 0. 1. 1. 0. 0. 1. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 1. 1. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 0. 0. 0. 0. 0.] Optimal ti: [1. 1. 1. 0. 0. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 0. 1. 0. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 0. 0. 0. 1. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 0. 0. 1. 1. 1. 1. 1. 1.] Optimal phi: [2. 2. 2. 0. 0. 0. 2. 2. 2. 2. 2. 2. 2. 0. 2. 2. 2. 2. 2. 2. 2. 0. 2. 2. 2. 2. 2. 2. 2. 2. 0. 2. 0. 2. 0. 0. 2. 2. 0. 2. 0. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 0. 2. 2. 2. 0. 0. 0. 2. 0. 2. 2. 2. 0. 2. 2. 2. 2. 2. 2. 2. 2. 0. 0. 2. 2. 2. 2. 2. 2.]
Code is correct but always gives optimal weight 0 which is not right. In SVM i need optimal weights for hyperplane.
I want to know where is the mistake. I request you to kindly help me as i am not a maths expert and i am using Gurobi as a beginner. I am stuck now in this . Please help. Thanks a lot. I am sending LP file in attachment. Please find.https://drive.google.com/file/d/1P3UVBaVFTWjOeIh-olcrO-ceN9i7lxHL/view?usp=drive_link
-
Thanks . But given post is also from me . I debugged it on handcrafted small data. Equation is right and objective and constraints are right but still gives 0 solution.
0 -
Our article How do I diagnose a wrong result? may be a good starting place to figuring out where the issue lies. -G
0 -
Hi Manju,
I think there are 4 possibilities you should consider:
1) The formulation is wrong
2) The code is wrong
3) The formulation and code are correct, the solution is wrong
4) The formulation and code are correct, the solution is correctYou're confident the code is correct. I've looked at it and can't see an error. So let's rule out 2. How about 3? The solver doesn't report any numerical issues, so the possibility that the solver is producing a non-optimal solution is tiny. I ran your model through another solver and it yielded the exact same result with w=0. So let's rule out 3. This leaves 1 and 4. You mentioned that the model gave correct results on a small instance, so this leans towards ruling out 1. Why can't w=0 in the optimal solution? What evidence do you have that allows you to rule out 4 as a possibility?
- Riley
0 -
I tried to use all parameters which are given to diagnose the problem. Issue is , this is working for small number of data points , if its is 10 to 12 data points. But for real datasets , its thousands of rows and its not working there and gives 0 value. Ideally if model is right , it should handle all datasets. Can you please help me how can i make this work? Gurobi is not solving this model for realdatasets.
0 -
From 1 to 4, If formulation and code both are correct then it should provide us weight vector for atleast any real datasets. I tried to run this for 4 real datasets and i am getting weight vector 0. I don't think it is possible that weight vector is 0 for all real datasets.
Another point is , i tried to run machine learning model for SVM using SVC , by adding uncertainty in labels by myself. There its providing weight vectors for real datasets if i am using directly SVC library and run the model. If that is working , then it should work here also. That's my assumption.
Last point is , real datasets i am working, some authors have already worked on that and created accuracy of model list in their paper. SO i am trying to replicate that. If weight vector is 0 then they would not have obtained those results which means something is wrong here.
0 -
Ok, so you rule out 4, I guess that leaves 1... The formulation is wrong?
0 -
I tried to reformulate the equation:
# Create a new Gurobi model
model = gp.Model("Robust_SVM_Model")
# Add variables
w = model.addMVar(p, name="w", lb=-GRB.INFINITY)
b = model.addVar(name="b", lb=-GRB.INFINITY)
q = model.addVar(name="q", lb=0.0)
xi = model.addMVar(n, lb=0.0, name="xi")
phi = model.addMVar(n, lb=0.0, name="phi")
ri = model.addMVar(n, lb=0.0, name="ri")
si = model.addMVar(n, vtype=GRB.BINARY, name="si")
ti = model.addMVar(n, vtype=GRB.BINARY, name="ti")
# Add constraints
for i in range(n):
expr = gp.quicksum(w[j] * X[i, j] for j in range(p)) - b
model.addConstr(q + ri[i] >= phi[i] - xi[i])
model.addConstr(phi[i] >= 1 + y[i] * expr)
model.addConstr(phi[i] <= 1 + y[i] * expr + M * (1 - ti[i]))
model.addConstr(phi[i] <= M * ti[i])
model.addConstr(xi[i] >= 1 - y[i] * expr)
model.addConstr(xi[i] <= 1 - y[i] * expr + M * (1 - si[i]))
model.addConstr(xi[i] <= M * si[i])
# Create the objective function expression
obj_expr = gp.LinExpr()
# Add the terms for xi
for i in range(n):
obj_expr += (xi[i])
# Add the terms for ri
for i in range(n):
obj_expr += (ri[i])
# Add the term for Gamma * q
obj_expr += Gamma* q
# Set the objective function
model.setObjective(obj_expr, sense=GRB.MINIMIZE)
# Optimize the model
model.optimize()
# Retrieve the optimal values
weights = np.array([w[i].x for i in range(p)])
bias = b.x
This also provides same result.0 -
Ok, so out of 1,2,3, and 4, what do you currently think is the most likely explanation?
0 -
From these, possibility that formulation is wrong but in that case it should not have worked on small set of handcrafted data also. I tried to reformulate also, it also worked on small set of data but not on real data.
another point is i am scaling the real datasets by using standard scalar from scikit. One possibility that datapoints are really small in real datasets and then its giving 0 as weight vector. But the problem is even i am not scaling then also getting this. In SVM we need to normalize or scale the data.
I am confused about how should i proceed in this. I dont have any idea what is going wrong here.
0 -
Can you please help me in reformulate model or change it if something is wrong in formulation according to the equation given in my post?
0 -
Can you paste the y vector?
0 -
[ 1 1 1 1 1 1 1 1 -1 1 1 1 -1 1 1 -1 1 -1 1 1 1 1 1 1 -1 1 1 1 -1 -1 1 1 -1 -1 -1 1 -1 1 1 1 1 -1 -1 1 -1 -1 1 1 1 1 1 1 1 1 1 1 -1 -1 1 -1 1 1 1 1 1 1 1 1 1 -1 1 1 1 1 1 -1 1 1 -1 1 1 1 -1 1 1 1 -1 1 1 1 1 1 1 1 1 1 1 1 1 1]
0 -
X is :
[[ 1.05192795e+00 -1.77951304e+00 -5.99803909e-01 -6.49629966e-01 -4.93421482e-01 -4.03336400e-01 -5.75771538e-01 6.40637530e-01 4.48832808e-01 -1.48304120e-01] [ 8.73231242e-01 5.61951487e-01 2.80475918e+00 2.83398654e+00 2.14255305e+00 -2.19775698e-01 -2.27616785e-01 1.35132893e+00 3.02633196e-01 -7.07942308e-01] [ 8.73231242e-01 5.61951487e-01 1.60314868e+00 1.93082670e+00 1.06654001e+00 -2.35072423e-01 -3.63482054e-01 8.43692215e-01 4.48832808e-01 -1.83281506e-01] [ 6.34968966e-01 5.61951487e-01 -4.99669701e-01 -4.56095716e-01 -5.19163421e-01 -4.10984762e-01 -5.67279959e-01 6.40637530e-01 5.95032419e-01 2.01469748e-01] [ 1.46888693e+00 5.61951487e-01 4.68294315e-01 5.76086951e-01 -4.52234380e-01 -3.61270406e-01 -4.01694161e-01 1.14827424e+00 -8.66963697e-01 -1.89717346e+00] [-7.98178625e-02 5.61951487e-01 -2.32645145e-01 -2.62561466e-01 -3.85305339e-01 -3.91863856e-01 -5.92754697e-01 1.45285627e+00 2.05702854e+00 1.25079135e+00] [-1.27112924e+00 -1.77951304e+00 -5.33047770e-01 -5.85118550e-01 -6.63318278e-01 -4.03336400e-01 -6.01246276e-01 8.43692215e-01 7.41232031e-01 2.01469748e-01] [-1.09243254e+00 -1.77951304e+00 -5.33047770e-01 -5.20607133e-01 -4.16195666e-01 -4.10984762e-01 -6.05492066e-01 5.39110188e-01 8.87431643e-01 5.51243615e-01] [-1.80721936e+00 5.61951487e-01 -5.33047770e-01 -5.20607133e-01 -4.16195666e-01 -3.80391312e-01 -5.71525749e-01 1.24980158e+00 1.61842970e+00 9.01017482e-01] [ 4.56272259e-01 5.61951487e-01 -5.99803909e-01 -5.85118550e-01 3.68624563e-02 -2.61841692e-01 -4.05939951e-01 6.40637530e-01 5.95032419e-01 2.01469748e-01] [ 5.75403397e-01 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -3.75008564e-01 -2.69490055e-01 -4.01694161e-01 -2.73108551e-01 -4.28364862e-01 -4.98077987e-01] [ 1.46888693e+00 5.61951487e-01 6.77574811e-02 1.24507034e-01 -1.17589176e-01 -3.45973681e-01 -4.14431531e-01 1.24980158e+00 1.02339728e-02 -1.19762572e+00] [ 9.92362380e-01 5.61951487e-01 -5.33047770e-01 -5.20607133e-01 1.39830211e-01 -2.31248242e-01 -4.05939951e-01 8.43692215e-01 5.95032419e-01 -1.48304120e-01] [ 1.58801807e+00 -1.77951304e+00 -4.66291631e-01 -4.56095716e-01 -3.54415013e-01 -3.80391312e-01 -5.24822062e-01 1.96049298e+00 1.61842970e+00 2.01469748e-01] [ 8.13665673e-01 5.61951487e-01 -5.99803909e-01 -5.85118550e-01 -7.09653768e-01 -2.61841692e-01 -4.78118376e-01 -3.74635893e-01 -4.28364862e-01 -2.53236280e-01] [-1.33069481e+00 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -5.14015033e-01 -1.16522804e-01 -4.27168900e-01 -6.79217921e-01 -1.01316331e+00 -8.47851854e-01] [-5.56342415e-01 5.61951487e-01 -2.32645145e-01 -1.98050049e-01 3.04578619e-01 1.77939155e-01 1.22019750e+00 1.45285627e+00 2.05702854e+00 1.25079135e+00] [-8.54170260e-01 5.61951487e-01 -2.99401284e-01 -3.91584299e-01 -6.06686013e-01 -4.07160581e-01 -5.54542590e-01 1.14827424e+00 7.41232031e-01 -7.83493463e-02] [-4.37211277e-01 -1.77951304e+00 -5.33047770e-01 -5.20607133e-01 5.23076196e-02 4.22686757e-01 3.88022719e-01 6.40637530e-01 1.56433584e-01 -4.98077987e-01] [-4.37211277e-01 -1.77951304e+00 -5.33047770e-01 -5.20607133e-01 5.23076196e-02 4.22686757e-01 3.88022719e-01 6.40637530e-01 1.56433584e-01 -4.98077987e-01] [ 2.18009983e-01 5.61951487e-01 -9.91328665e-02 -6.90272159e-02 1.68434654e+00 -3.99512219e-01 -5.33313642e-01 1.14827424e+00 -5.74564474e-01 -1.37251266e+00] [ 2.18009983e-01 5.61951487e-01 1.34513620e-01 1.24507034e-01 1.02535290e+00 -3.80391312e-01 -5.07838903e-01 8.43692215e-01 -8.66963697e-01 -1.54739959e+00] [ 8.73231242e-01 5.61951487e-01 1.43625833e+00 1.22120112e+00 1.33425617e+00 -2.09182716e-02 -3.71973634e-01 2.34528161e-01 1.56433584e-01 -1.48304120e-01] [-4.37211277e-01 5.61951487e-01 -1.99267075e-01 -6.90272159e-02 -2.66892421e-01 -4.03336400e-01 -4.18677320e-01 -1.89754603e+00 -2.03656059e+00 -1.19762572e+00] [ 9.32796811e-01 5.61951487e-01 -5.33047770e-01 -5.85118550e-01 -4.57382768e-01 -2.65665874e-01 -4.61135217e-01 -1.71581209e-01 1.32603048e+00 3.17454762e+00] [-7.94604691e-01 5.61951487e-01 5.35050455e-01 5.76086951e-01 3.17140686e-02 2.88163532e+00 2.45147650e+00 -1.18685463e+00 -4.28364862e-01 5.51243615e-01] [-7.94604691e-01 5.61951487e-01 5.35050455e-01 5.76086951e-01 3.17140686e-02 2.88163532e+00 2.45147650e+00 -1.18685463e+00 -4.28364862e-01 5.51243615e-01] [-7.94604691e-01 5.61951487e-01 1.23598991e+00 1.22120112e+00 -2.20556931e-01 5.96010125e+00 2.95672547e+00 1.04674690e+00 1.47223009e+00 9.01017482e-01] [-1.62852266e+00 5.61951487e-01 -4.66291631e-01 -3.91584299e-01 -7.97176360e-01 -3.88039675e-01 -5.24822062e-01 -2.30365540e+00 -1.59796176e+00 2.65828139e-02] [ 2.18367376e+00 -1.77951304e+00 -5.99803909e-01 -5.85118550e-01 -4.88273095e-01 -4.14808944e-01 -5.63034169e-01 -1.71581209e-01 3.02633196e-01 5.51243615e-01] [ 5.75403397e-01 5.61951487e-01 5.01672385e-01 5.11575535e-01 -4.77976319e-01 -2.92435143e-01 -1.80913098e-01 -9.83799948e-01 -2.18276020e+00 -1.89717346e+00] [ 2.77575552e-01 5.61951487e-01 -5.33047770e-01 -5.85118550e-01 -6.53021503e-01 -3.30676955e-01 -4.65381007e-01 -1.28838197e+00 -1.35965639e-01 1.60056522e+00] [ 5.75403397e-01 5.61951487e-01 -4.99669701e-01 -5.20607133e-01 -4.93421482e-01 -3.91863856e-01 -5.54542590e-01 -9.83799948e-01 -1.35965639e-01 9.01017482e-01] [-5.56342415e-01 -1.77951304e+00 3.43794116e-02 5.99956175e-02 6.54668987e-01 -2.38896605e-01 -4.10185741e-01 -5.77690578e-01 1.02339728e-02 -4.98077987e-01] [-5.56342415e-01 -1.77951304e+00 3.43794116e-02 5.99956175e-02 6.54668987e-01 -2.38896605e-01 -4.10185741e-01 -5.77690578e-01 1.02339728e-02 -4.98077987e-01] [-1.03286697e+00 5.61951487e-01 -3.99535492e-01 -4.56095716e-01 1.02535290e+00 -7.44568096e-02 -3.12532578e-01 7.42164873e-01 4.48832808e-01 -1.48304120e-01] [-1.80721936e+00 -1.77951304e+00 -5.99803909e-01 -5.85118550e-01 -7.09653768e-01 -3.95688037e-01 -4.99347324e-01 1.04674690e+00 1.32603048e+00 8.31062709e-01] [-7.98178625e-02 -1.77951304e+00 3.90623548e+00 4.31774912e+00 4.69327028e-01 -3.19204412e-01 -3.25269947e-01 -1.89754603e+00 -1.45176214e+00 -4.98077987e-01] [ 3.93132756e-02 5.61951487e-01 -3.66157423e-01 -3.27072883e-01 -1.02144013e-01 -3.19204412e-01 -3.71973634e-01 -3.74635893e-01 -1.15936292e+00 -1.16264834e+00] [-2.02522935e-02 5.61951487e-01 6.77574811e-02 1.24507034e-01 -4.03633600e-02 5.85099736e-03 -3.42253106e-01 3.14734761e-02 4.48832808e-01 5.51243615e-01] [-1.39383432e-01 5.61951487e-01 -3.23767274e-02 -4.51579917e-03 -5.91240850e-01 -3.38325318e-01 -4.39906269e-01 -1.08532729e+00 -5.74564474e-01 2.01469748e-01] [ 8.73231242e-01 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -6.32427952e-01 -3.03907686e-01 -1.85158888e-01 -1.28838197e+00 -5.74564474e-01 5.51243615e-01] [-3.18080139e-01 5.61951487e-01 1.43625833e+00 1.35022395e+00 1.78731429e+00 -3.68918768e-01 -4.52643638e-01 -7.00538663e-02 -1.01316331e+00 -1.19762572e+00] [ 1.58444414e-01 5.61951487e-01 3.43794116e-02 5.99956175e-02 6.80410925e-01 1.09191848e+00 1.79337910e+00 2.34528161e-01 3.02633196e-01 2.01469748e-01] [ 2.24323933e+00 -1.77951304e+00 -4.99669701e-01 -5.20607133e-01 -3.85305339e-01 -3.99512219e-01 -5.88508907e-01 8.43692215e-01 8.87431643e-01 2.01469748e-01] [-7.35039122e-01 5.61951487e-01 -2.32645145e-01 -3.27072883e-01 -4.03633600e-02 -2.80962599e-01 1.03554810e-01 3.36055503e-01 3.02633196e-01 -1.48304120e-01] [-1.56895709e+00 5.61951487e-01 4.68294315e-01 4.47064118e-01 -6.83911829e-01 -3.26852774e-01 -5.37559431e-01 6.40637530e-01 1.32603048e+00 1.39070090e+00] [-4.37211277e-01 5.61951487e-01 -4.66291631e-01 -5.20607133e-01 -2.72040809e-01 5.76889219e+00 3.42376234e+00 -1.28838197e+00 -2.82165250e-01 1.25079135e+00] [-9.13735829e-01 -1.77951304e+00 -6.33181979e-01 -6.49629966e-01 -5.50053748e-01 -3.15380230e-01 -5.33313642e-01 -1.71581209e-01 1.02339728e-02 2.01469748e-01] [ 4.56272259e-01 5.61951487e-01 5.30811440e+00 4.96286329e+00 -3.95602115e-01 -2.19775698e-01 1.03554810e-01 3.14734761e-02 -1.74416137e+00 -1.89717346e+00] [-1.39383432e-01 -1.77951304e+00 -5.99803909e-01 -5.85118550e-01 -5.80944074e-01 -3.84215493e-01 -5.92754697e-01 -4.76163236e-01 -7.20764085e-01 -8.47851854e-01] [-7.94604691e-01 -1.77951304e+00 -6.33181979e-01 -6.49629966e-01 -6.27279564e-01 -4.07160581e-01 -5.71525749e-01 4.37582846e-01 5.95032419e-01 2.01469748e-01] [-5.56342415e-01 5.61951487e-01 2.01269759e-01 3.18041284e-01 -1.53627891e-01 -1.58588798e-01 1.07159486e+00 6.40637530e-01 1.32603048e+00 1.25079135e+00] [-5.56342415e-01 5.61951487e-01 -4.66291631e-01 -5.20607133e-01 -4.36789217e-01 -1.35643710e-01 -1.53273007e-02 1.33000819e-01 7.41232031e-01 9.01017482e-01] [-3.18080139e-01 5.61951487e-01 2.13719779e+00 2.18887237e+00 -5.58085232e-02 -3.45973681e-01 -3.93202582e-01 -3.74635893e-01 -1.45176214e+00 -1.54739959e+00] [-3.18080139e-01 5.61951487e-01 2.13719779e+00 2.18887237e+00 -5.58085232e-02 -3.45973681e-01 -3.93202582e-01 -3.74635893e-01 -1.45176214e+00 -1.54739959e+00] [-8.54170260e-01 5.61951487e-01 -5.66425840e-01 -5.85118550e-01 -4.36789217e-01 -3.65094587e-01 -5.54542590e-01 1.85896564e+00 1.47223009e+00 2.01469748e-01] [ 3.93132756e-02 -1.77951304e+00 -5.33047770e-01 -5.85118550e-01 -5.55202135e-01 -3.72742950e-01 -4.22923110e-01 -6.79217921e-01 -4.28364862e-01 -1.48304120e-01] [ 2.18009983e-01 5.61951487e-01 -5.66425840e-01 -5.85118550e-01 4.33288313e-01 -3.03907686e-01 -5.75771538e-01 -9.83799948e-01 -1.45176214e+00 -1.19762572e+00] [ 9.92362380e-01 5.61951487e-01 -4.66291631e-01 -3.91584299e-01 -7.09653768e-01 -3.88039675e-01 -5.50296800e-01 -6.79217921e-01 3.02633196e-01 1.56558783e+00] [-9.73301398e-01 -1.77951304e+00 -5.66425840e-01 -5.85118550e-01 -6.42724727e-01 -3.84215493e-01 -5.84263118e-01 -1.71581209e-01 1.02339728e-02 2.01469748e-01] [ 6.34968966e-01 5.61951487e-01 -4.99669701e-01 -3.91584299e-01 -6.42724727e-01 -3.23028593e-01 -4.69626796e-01 1.04674690e+00 8.87431643e-01 2.01469748e-01] [ 6.34968966e-01 5.61951487e-01 -4.99669701e-01 -3.91584299e-01 -6.42724727e-01 -3.23028593e-01 -4.69626796e-01 1.04674690e+00 8.87431643e-01 2.01469748e-01] [ 5.75403397e-01 5.61951487e-01 -5.99803909e-01 -5.85118550e-01 -3.85305339e-01 -3.30676955e-01 -2.40354154e-01 -1.08532729e+00 -1.30556253e+00 -8.47851854e-01] [ 5.75403397e-01 5.61951487e-01 -3.99535492e-01 -4.56095716e-01 -1.22737564e-01 -3.11556049e-01 -2.87057840e-01 3.36055503e-01 -7.20764085e-01 -1.19762572e+00] [ 5.75403397e-01 5.61951487e-01 -3.66157423e-01 -2.62561466e-01 9.63572252e-01 -2.27424061e-01 -2.78566261e-01 -5.77690578e-01 -7.20764085e-01 -4.98077987e-01] [ 3.96706690e-01 5.61951487e-01 -9.91328665e-02 5.99956175e-02 -4.52234380e-01 -2.54193330e-01 -2.48845733e-01 -1.71581209e-01 1.03363125e+00 2.30011295e+00] [-6.15907984e-01 5.61951487e-01 -2.32645145e-01 -1.98050049e-01 -3.49266625e-01 -2.61841692e-01 -4.05939951e-01 2.34528161e-01 1.17983087e+00 1.60056522e+00] [ 1.11149352e+00 5.61951487e-01 -5.99803909e-01 -5.85118550e-01 -2.25705319e-01 -3.61270406e-01 -5.41805221e-01 1.33000819e-01 1.03363125e+00 1.60056522e+00] [ 7.54100104e-01 5.61951487e-01 -5.66425840e-01 -5.85118550e-01 -3.49266625e-01 -3.72742950e-01 -5.80017328e-01 1.33000819e-01 1.02339728e-02 -1.48304120e-01] [-1.68808823e+00 -1.77951304e+00 -5.99803909e-01 -5.85118550e-01 -4.98569870e-01 1.70290792e-01 1.03338275e+00 -6.79217921e-01 1.02339728e-02 9.01017482e-01] [ 1.64758364e+00 -1.77951304e+00 -5.66425840e-01 -5.85118550e-01 -4.88273095e-01 -3.88039675e-01 -5.29067852e-01 -1.79601869e+00 -1.74416137e+00 -1.19762572e+00] [ 1.64758364e+00 -1.77951304e+00 -5.66425840e-01 -5.85118550e-01 -4.00750503e-01 -3.61270406e-01 -5.50296800e-01 -1.79601869e+00 -1.45176214e+00 -4.98077987e-01] [ 2.77575552e-01 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -5.75795686e-01 -3.80391312e-01 -5.84263118e-01 4.37582846e-01 8.87431643e-01 9.01017482e-01] [ 1.23062466e+00 5.61951487e-01 -5.99803909e-01 -6.49629966e-01 -7.09653768e-01 -3.88039675e-01 -5.58788380e-01 -3.74635893e-01 -1.35965639e-01 2.01469748e-01] [-1.09243254e+00 -1.77951304e+00 -5.99803909e-01 -6.49629966e-01 -6.22131176e-01 -2.65665874e-01 -4.78118376e-01 -9.83799948e-01 -7.20764085e-01 -1.48304120e-01] [-9.73301398e-01 5.61951487e-01 -5.33047770e-01 -5.85118550e-01 1.21069486e+00 2.58246962e-01 -5.80017328e-01 -8.82272605e-01 -1.01316331e+00 -8.47851854e-01] [ 1.23062466e+00 -1.77951304e+00 -6.33181979e-01 -6.49629966e-01 6.88421817e+00 -1.01226079e-01 -1.12980463e-01 -1.59296400e+00 -1.30556253e+00 -4.98077987e-01] [ 1.34975579e+00 5.61951487e-01 -3.66157423e-01 -3.27072883e-01 -7.04505380e-01 -4.18633125e-01 -5.50296800e-01 3.14734761e-02 1.17983087e+00 2.23015818e+00] [ 6.34968966e-01 -1.77951304e+00 1.01135551e-01 1.24507034e-01 1.99324980e+00 -2.80962599e-01 -3.16778368e-01 -1.49143666e+00 -2.03656059e+00 -1.54739959e+00] [ 6.34968966e-01 -1.77951304e+00 -3.23767274e-02 -4.51579917e-03 3.25460480e+00 -2.35072423e-01 -4.92936182e-02 -1.49143666e+00 -1.74416137e+00 -1.19762572e+00] [-1.09243254e+00 5.61951487e-01 -4.99669701e-01 -5.20607133e-01 -1.07004091e+00 -3.68918768e-01 -5.41805221e-01 -1.08532729e+00 -1.35965639e-01 1.25079135e+00] [ 9.88788446e-02 5.61951487e-01 -5.99803909e-01 -6.49629966e-01 -6.94208605e-01 -4.10984762e-01 -6.01246276e-01 -7.80745263e-01 -2.82165250e-01 2.01469748e-01] [-8.54170260e-01 5.61951487e-01 -1.65889006e-01 -6.90272159e-02 -1.27885952e-01 2.77367868e-01 -6.83572138e-03 -7.80745263e-01 1.02339728e-02 1.07590442e+00] [-9.13735829e-01 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -2.36002095e-01 -2.92435143e-01 -5.20576273e-01 1.35132893e+00 1.91082892e+00 1.39070090e+00] [-1.98591607e+00 5.61951487e-01 -3.66157423e-01 -3.91584299e-01 -7.12536865e-02 -2.42720786e-01 -4.61135217e-01 5.39110188e-01 1.32603048e+00 1.60056522e+00] [-2.04548164e+00 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 1.91314089e-01 -3.57446224e-01 -4.14431531e-01 1.04674690e+00 8.87431643e-01 2.01469748e-01] [ 6.34968966e-01 5.61951487e-01 -5.66425840e-01 -5.85118550e-01 7.80495583e-02 -3.38325318e-01 -4.01694161e-01 3.14734761e-02 1.56433584e-01 2.01469748e-01] [-1.74765380e+00 5.61951487e-01 -6.33181979e-01 -5.85118550e-01 1.31366262e+00 -3.38325318e-01 -5.07838903e-01 1.35132893e+00 3.02633196e-01 -8.47851854e-01] [ 7.54100104e-01 5.61951487e-01 5.01672385e-01 5.11575535e-01 -2.30853707e-01 -9.44572777e-03 8.33830635e-01 9.45219558e-01 4.48832808e-01 -4.98077987e-01] [ 7.54100104e-01 5.61951487e-01 1.06909957e+00 1.09217828e+00 -3.54415013e-01 1.11103939e+00 2.95672547e+00 1.14827424e+00 3.02633196e-01 -5.68032761e-01] [ 7.54100104e-01 5.61951487e-01 1.43625833e+00 1.35022395e+00 1.29533436e-01 1.08044594e+00 2.71896125e+00 6.40637530e-01 1.02339728e-02 -8.47851854e-01] [ 7.54100104e-01 5.61951487e-01 2.03706358e+00 1.86631529e+00 7.80495583e-02 1.11103939e+00 2.95672547e+00 1.24980158e+00 1.02339728e-02 -1.19762572e+00] [ 7.54100104e-01 5.61951487e-01 1.10247764e+00 1.02766687e+00 -4.05898890e-01 3.76796582e-01 1.04612012e+00 8.43692215e-01 1.02339728e-02 -8.47851854e-01] [ 7.54100104e-01 5.61951487e-01 9.02209219e-01 8.34132618e-01 -5.91240850e-01 1.73235412e-02 2.05453763e-01 6.40637530e-01 -1.35965639e-01 -8.47851854e-01] [ 1.64758364e+00 5.61951487e-01 -5.33047770e-01 -5.85118550e-01 -4.32464571e-03 -3.68918768e-01 -5.54542590e-01 -1.79601869e+00 -1.15936292e+00 2.01469748e-01] [-4.96776846e-01 5.61951487e-01 4.34916246e-01 2.53529868e-01 7.80495583e-02 -7.44568096e-02 2.02265175e+00 9.45219558e-01 4.48832808e-01 -4.98077987e-01] [-4.96776846e-01 5.61951487e-01 1.36950219e+00 1.22120112e+00 -3.49266625e-01 2.62071143e-01 3.38130444e+00 -2.20212806e+00 -1.89036098e+00 -8.47851854e-01] [-1.74765380e+00 5.61951487e-01 -6.33181979e-01 -6.49629966e-01 -9.18472375e-02 -9.35777160e-02 3.13763857e-02 -2.73108551e-01 1.56433584e-01 5.51243615e-01] [-1.74765380e+00 5.61951487e-01 -5.99803909e-01 -6.49629966e-01 1.50126987e-01 7.13324535e-01 1.06734907e+00 7.42164873e-01 1.03363125e+00 5.51243615e-01]]
0 -
Using a scikit-learn dataset I don't get a zero w solution:
import gurobipy as gp
from gurobipy import GRB
import numpy as np
from sklearn import datasets, metrics
from sklearn.model_selection import train_test_split
cancer = datasets.load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, test_size=0.3,random_state=109) # 70% training and 30% test
#map 0 -> -1
y_test = np.where(y_test, 1,-1)
y_train = np.where(y_train, 1,-1)
# Step 2: Train the Robust SVM Model
def train_robust_svm(X, y, M, Gamma):
n = len(X)
p = len(X[0])
print(n)
print(p)
# Create a new Gurobi model
model = gp.Model("Robust_SVM_Model")
model.setParam('NumericFocus', 3)
# Add variables
w = model.addMVar(p, name="w", lb=-GRB.INFINITY)
b = model.addVar(name="b", lb=-GRB.INFINITY)
q = model.addVar(name="q", lb=0.0)
xi = model.addMVar(n, lb=0.0, name="xi")
phi = model.addMVar(n, lb=0.0, name="phi")
ri = model.addMVar(n, lb=0.0, name="ri")
si = model.addMVar(n, vtype=GRB.BINARY, name="si")
ti = model.addMVar(n, vtype=GRB.BINARY, name="ti")
# Add constraints
for i in range(n):
model.addConstr(q + ri[i] >= phi[i] - xi[i])
model.addConstr(xi[i] >= 1 - y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b))
model.addConstr(xi[i] <= 1 - y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b) + M * (1 - si[i]))
model.addConstr(xi[i] <= M * si[i])
model.addConstr(phi[i] >= 1 + y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b))
model.addConstr(phi[i] <= 1 + y[i] * (gp.quicksum(w[j] * X[i, j] for j in range(p)) - b) + M * (1 - ti[i]))
model.addConstr(phi[i] <= M * ti[i])
# Set objective function
model.setObjective(gp.quicksum(xi[i] for i in range(n)) + Gamma * q + gp.quicksum(ri[i] for i in range(n)), sense=GRB.MINIMIZE)
# Optimize the model
model.optimize()
# Retrieve the optimal values
weights = np.array([w[i].x for i in range(p)])
bias = b.x
return {'w': weights, 'b': bias, 'q': q.x, 'xi': np.array([xi[i].x for i in range(n)]),
'ri': np.array([ri[i].x for i in range(n)]), 'si': np.array([si[i].x for i in range(n)]),
'ti': np.array([ti[i].x for i in range(n)]), 'phi': np.array([phi[i].x for i in range(n)])}
# Train the robust SVM model
results = train_robust_svm(X_train, y_train, M=100000, Gamma=40)
# Step 3: Predict on the Testing Set
def predict(X, params):
decision_values = np.dot(X, params['w']) + params['b']
return np.sign(decision_values)
# Evaluate on the testing set
y_pred = predict(X_test, results)
# Evaluate Accuracy
accuracy = metrics.accuracy_score(y_test, y_pred)
print(f"Accuracy on the test set: {accuracy}")Optimal Weight vector (w): [ 4.27420607e-01 -3.82393344e-02 -2.97599013e-02 -1.26282100e-03
4.54603522e+00 1.52733564e+01 -6.27400722e+00 -7.75194505e+00
-4.58946623e+00 -1.07178125e+00 -9.87668218e-01 -1.84690585e-01
1.39751328e-01 -4.56206749e-04 -9.55333547e+00 -2.34038325e+00
1.35220191e+01 -5.05290770e+01 -2.31278917e+00 3.36014477e+01
-3.87803194e-01 1.11789443e-02 -2.46573661e-02 3.09436140e-03
-6.30821599e+00 -1.66628842e-01 -1.23835022e+00 7.17691280e-01
-1.19578101e+00 -1.59087955e+01]Also try experimenting with other values of Gamma (including 0).
- Riley
0 -
Did you try this model on different dataset?
Why this is giving 0 in datasets which i am trying? I am getting those real datasets from UCI website which are listed in paper.
If i put Gamma as 0 , it means there is no uncertainty in label. I have to model the uncertainty in label, this is the what i want to do in this model. SO i need to put some gamma values.
0 -
another point is, accuracy which mentioned in paper is very high for eg 87 percent . Even i try in handcrafted dataset my accuracy is around 50 percent not more that that. How can i achieve this much high accuracy?
0 -
I don't know. Maybe there is some data preprocessing required.
0 -
Ok, Thanks. I can say that model formulation and code is right, right?
0 -
The code looks to be accurate to the formulation. Is the formulation correct? I am not so sure.
0 -
Try this dataset:
https://archive.ics.uci.edu/dataset/174/parkinsonsI think you have a typo in your predict method, you want to subtract the bias, not add it right?
With a gamma value of 1, I get an 86% accuracy which is close to what is reported in the paper for this dataset (I didn't preprocess the data either). The weight vector is non-zero. If I use gamma=40 (as per the code above) then the weight vector goes to zero. Note that in the paper they are doing parameter tuning to find the best value of gamma for each dataset.
0 -
When predicting using a support vector machine (SVM) model, need to use the formula y(w⋅x+b).
So according to that its always additive.
I tried it with this dataset. Its giving 87 percent accuracy which is same as paper.
Yes, you are right, In paper may be they are doing parameter tuning and find best value of gamma.
Is there any way how can i find best value of gamma? Because for real big datasets its difficult to try each value of gamma.
0 -
Hi Manju,
When predicting using a support vector machine (SVM) model, need to use the formula y(w⋅x+b)
This isn't necessarily true. In the model b has no bounds, so you could switch the sign of the coefficients and the resulting formula would change. The reason I questioned it was the presence of wx - b in the model, and the fact I seemed to get better accuracy using wx - b in the prediction function.
Is there any way how can i find best value of gamma?
You could use Optuna for something like this. The other obstacle to reproducing the results from the paper is that they clearly are doing some preprocessing of the data, and it's not clear what this is.
- Riley
0 -
Hi,
Thanks , i got your point and i changed sign for y(w.x + b) to y(w.x-b).
Model is working for gamma = 0. In my above comments , i mentioned that if gamma is 0 then it means there is no uncertainty , means it should behave like nominal model and its given in research paper also .But when i run actual nominal model for SVM , it provides different weight vectors as compared to when i get weight vectors with gamma = 0. This is for smae X and y given in comments. Here i am using Gamma = 0, M = 1000000000000000000000
result for this robust model is :
Optimal Weight vector (w): [ 0.55926838 -0.01151814 -0.04787238 0.47622849 0.19034862 0.94500694 0.17726415 0.79670367 -1.01948092 0.52285992] Optimal Bias (b): -1.0121027258626611
result for nominal model is :
weight vector: {'w': array([ 0.00261057, -0.00046048, 0.0004205 , 0.00371566, 0.0048371 ,
0.00610579, 0.00476305, 0.00553666, -0.0076985 , 0.00404158]),
bias: 'b': 1.0036750651811581}0 -
Hi Manju,
But when i run actual nominal model for SVM , it provides different weight vectors as compared to when i get weight vectors with gamma = 0.
This is expected right?
Here i am using Gamma = 0, M = 1000000000000000000000
This value of M is waaaaaay too high. Consider a big M constraint with y continuous and x binary:
y <= Mx
Solutions to MIPs are often not purely integral. Computers use floating point arithmetic and the consequences are that integer variables can take values which are not precisely integral in valid solutions. The default IntFeasTol is 1e-5. Let's say the x variable takes a value of 0.000001. Ideally we'd like the y variable to be 0, this is the point of the big M constraint, however with a big M value of 1000000000000000000000, the y variable can take a value up to 1000000000000000. Even if you tighten IntFeasTol you could still end up with a very large y value. You can try to mitigate this with IntegralityFocus but the best fix is either to limit the big M value or use indicator constraints. See these links for more info:
https://www.gurobi.com/documentation/current/refman/dealing_with_big_m_constra.html
https://or.stackexchange.com/questions/231/when-to-use-indicator-constraints-versus-big-m-approaches-in-solving-mixed-int/348#348- Riley
0 -
Thanks. I got your point.
I tried for Parkinsons dataset from UCI and tried with M=10000, Gamma=0. In this according to my opinion, M is small , so it should work like nominal model of SVM. But with this also weight vector solution is different than weights what ia m getting with gamma = 0 in robust solution.
I tried with different params also which was suggested in last comment, which were
model.setParam('NumericFocus', 3)
model.setParam('IntegralityFocus',0)
model.setParam('IntFeasTol',1e-9)
model.setParam('FeasibilityTol',1e-9)But there is huge difference in weights solutions:
weight according to nominal model:
weights{'w': array([-0.34318071, -0.07749391, -0.18092597, -0.83382874, -0.32532997, 0.37532182, -0.28987008, 0.37253385, 0.15736407, 0.00595781, -0.10172352, 0.28033579, 0.46224634, -0.10228992, 0.16956159, 0.17820232, -0.4482566 , 0.41657873, 0.98374453, 0.40014912, 0.09749479, 0.54775984]),
'b': 1.7043663830233236}weight with Gamma = 0 in unecratinty in label model:
weights: [-8.36845077e-01 -3.67010639e-01 1.78810398e-01 -6.29931120e+00 -2.37937375e+00 -3.42119518e+01 1.38809467e+00 4.07941037e+01 2.38949117e+01 5.53790674e-01 1.22706784e+03 -6.12983259e+00 -4.14820412e+00 -1.24149060e+03 2.69841482e-01 -1.83733951e-01 -3.06528864e-01 2.12750575e-01 -6.67001462e-01 4.56505719e-01 3.98259191e-01 3.09971385e+00] Optimal Bias (b): -2.4437203457953434
These two are very different. This should not be possible.0
Please sign in to leave a comment.
Comments
26 comments