Question regarding using Gurobi with Python's multiprocessing library

Hi,

I am encountering an issue when using Gurobi with Python's multiprocessing library.

I get the following error:

MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x15525b1e5700>'.
Reason: 'TypeError("cannot pickle 'PyCapsule' object")'

The problem seems to arise when Gurobi tries to pass the model between processes during the optimization. However, each process is supposed to be independent. No information should be passed from one to the other.

This error only occurs when the number of binary variables increases beyond a certain threshold (approximately 2,500 variables). Below this threshold, the code runs without any issues.

Could anyone help me with this please? Thank you in advance!

Below is an example:

import random
import numpy as np
from gurobipy import *
from multiprocessing import Pool

def knapsack(pb_input):
    
    values, weights, capacity, num_threads = pb_input
    
    n = len(values)
    
    # Create a new model
    with Env() as env, Model(env=env) as m:
        m = Model("knapsack")

        m.setParam('Threads', num_threads)

        # Create variables
        x = m.addVars(n, vtype=GRB.BINARY, name="item")

        # Set objective
        m.setObjective(quicksum(values[i] * x[i] for i in range(n)), GRB.MAXIMIZE)

        # Add constraint
        m.addConstr(quicksum(weights[i] * x[i] for i in range(n)) <= capacity, "weight_constraint")

        # Optimize model
        m.optimize()

        # Print solution
        if m.status == GRB.OPTIMAL:
            print('Optimal objective: %g' % m.objVal)
            print('Solution:')
            c = 0
            for i in range(n):
                if x[i].x > 0.5:
                    c += 1
            print(c, 'items in solution')
        else:
            print('No solution')

        m.setParam('Threads', num_threads)

values = [random.randint(1, 1000) for _ in range(150*8*2)]
weights = [random.randint(1, 1000) for _ in range(150*8*2)]
capacity = 150*8*500

num_threads = 2

pb_input = (values, weights, capacity, num_threads)

%time
if __name__ == '__main__':
    # Define knapsack problems
    args_list = [(values, weights, capacity, num_threads) for day in range(10)]

    # Create a pool with 4 processes (one for each problem)
    with Pool(processes=4) as pool:
        results = pool.map(knapsack, args_list)

The full output printing is:

CPU times: user 2 µs, sys: 1e+03 ns, total: 3 µs
Wall time: 3.34 µs
Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Set parameter Threads to value 2
Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")


Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads

Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads


Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads

Restricted license - for non-production use only - expires 2025-11-24
Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Set parameter Threads to value 2
Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")


CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads


Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads

Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Restricted license - for non-production use only - expires 2025-11-24
Set parameter Threads to value 2
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")
Restricted license - for non-production use only - expires 2025-11-24
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads


Set parameter Threads to value 2
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Red Hat Enterprise Linux 8.5 (Ootpa)")

CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]
Thread count: 128 physical cores, 256 logical processors, using up to 2 threads
CPU model: AMD EPYC 7742 64-Core Processor, instruction set [SSE2|AVX|AVX2]

Thread count: 128 physical cores, 256 logical processors, using up to 2 threads

 

---------------------------------------------------------------------------
MaybeEncodingError                        Traceback (most recent call last)
Cell In[10], line 8
      6 # Create a pool with 4 processes (one for each problem)
      7 with Pool(processes=4) as pool:
----> 8     results = pool.map(knapsack, args_list)

File ~/anaconda3/envs/cbssim/lib/python3.9/multiprocessing/pool.py:364, in Pool.map(self, func, iterable, chunksize)
    359 def map(self, func, iterable, chunksize=None):
    360     '''
    361     Apply `func` to each element in `iterable`, collecting the results
    362     in a list that is returned.
    363     '''
--> 364     return self._map_async(func, iterable, mapstar, chunksize).get()

File ~/anaconda3/envs/cbssim/lib/python3.9/multiprocessing/pool.py:771, in ApplyResult.get(self, timeout)
    769     return self._value
    770 else:
--> 771     raise self._value

MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x148f3d65bd90>'. Reason: 'TypeError("cannot pickle 'PyCapsule' object")'