Gurobi parallel load COBRA (.mat) model
Hi,
I am trying to load about 300 .mat model of COBRA parallel, I used the below function:
def get_capacity_matrix_parallel(model_path, target_rxns, AD_samples, HT_samples,
n_workers=16):
all_fnames = sorted(os.listdir(model_path))
all_fnames = [f for f in all_fnames if f.endswith('.mat')]
n_models = len(all_fnames)
if n_models == 0:
raise RuntimeError(f"No .mat files found in {model_path}")
n_workers = min(n_workers, n_models)
AD_set = set(AD_samples)
HT_set = set(HT_samples)
matrix = np.zeros((n_models, len(target_rxns)))
phenotypes = [None] * n_models
model_dict = {'AD': {}, 'HT': {}}
fname_to_idx = {f: i for i, f in enumerate(all_fnames)}
args_list = [
(fname, model_path, target_rxns, AD_set, HT_set)
for fname in all_fnames
]
errors = []
print(f"Loading {n_models} models with {n_workers} workers "
f"(each worker: own Gurobi Env, Threads=1)")
t0 = time.time()
with Pool(processes=n_workers, initializer=_pool_initializer) as pool:
for result in tqdm(
pool.imap_unordered(_load_and_compute_one, args_list, chunksize=1),
total=n_models, desc="Models",
):
idx = fname_to_idx[result['fname']]
matrix[idx] = result['v_max_row']
phenotypes[idx] = result['phenotype']
if result['error']:
errors.append((result['fname'], result['error']))
continue
if result['phenotype'] in ('AD', 'HT') and result['model'] is not None:
model_dict[result['phenotype']][result['sample_id']] = result['model']
elapsed = time.time() - t0
print(f"\nFinished in {elapsed:.1f}s ({elapsed/n_models:.2f}s per model avg)")
if errors:
print(f"\n{len(errors)} models failed:")
for fname, err in errors[:10]:
print(f" {fname}: {err}")
if len(errors) > 10:
print(f" ... and {len(errors) - 10} more")
return matrix, model_dict, all_fnames, phenotypes, n_models But it failed to have parallel loading processes, still load the .mat model serially. Could you please give me some guidance on what should I do? Thanks!
-
The bottleneck is very likely not Gurobi but what you send back through the Pool.
1) You return result['model'], a live Gurobi model object. Gurobi Model/Env objects are not picklable, and even when a wrapper makes them serializable, each one must be pickled in the worker, streamed over a pipe, and unpickled in the parent. The parent unpickles one result at a time, so it becomes the serial bottleneck and the whole run looks serial even though workers do run in parallel. Return only numbers you need (v_max_row, phenotype, sample_id, fname) and rebuild/re-read models in the parent only for the few you actually need.
2) chunksize=1 with large payloads maximizes IPC overhead. If tasks are short, raise chunksize. If each task is dominated by the .mat read, that is disk I/O and 16 workers on one disk serialize on it regardless of Gurobi. Time one worker call to see how much is scipy.io.loadmat vs the LP solve.
3) Make sure _pool_initializer creates one gurobipy.Env per process, and that no Env is created at module import before the fork. An env created pre-fork and inherited by children is a classic cause of contention. Create it in the initializer with Threads=1 and OutputFlag=0 and reuse it for all models in that worker.
4) On Linux the default start method is fork; if any parent-level Gurobi env exists, use multiprocessing.set_start_method("spawn") so workers start clean.
5) Verify parallelism independently of Gurobi: print os.getpid() and a timestamp at the top of _load_and_compute_one. If 16 distinct PIDs start at once, the workers are fine and the problem is the result-handling path in point 1.
0
Please sign in to leave a comment.
Comments
1 comment