gurobipy routine killed on macOS even though VM available
回答済みI am running a gurobipy MIP routine for nonconvex QCQP that gets killed on macOS even though there is still tons of virtual memory available, with little RAM left. Since MacOS handles memory in such a way that it dynamically allocates new swapfiles to the process as long as there is free space in the startup disk, I wonder if there is a known issue with gurobipy and what are the causes for the OS killing the process when it has only used 40GB of VM, with hundreds of GBs of VM still available. More technical details on the machine at https://discussions.apple.com/thread/256204863?sortBy=rank&answerId=261726728022 where I also reported the issue. This does not happen with Linux, where the process is killed only when all available swap is used, for example, so is it a known issue with macOS? The only difference between the macOS and Linux implementation is that in macOS I am doing this in a conda environment, in Linux in a regular python environment.
-
-
Gurobi Staff
Hi Giordano,
Can you investigate whether this is specific to gurobipy with something like the following?
import time import numpy as np arrays = [] size_mb = 200 n = (size_mb * 1024 * 1024) // 8 i = 0 while True: print(f"{i=}") a = np.ones(n, dtype=np.float64) arrays.append(a) i += 1 time.sleep(0.1)What version of gurobipy and Python are you using?
- Riley
0 -
-
Thank you, it is python 3.12 and gurobipy12.03 all in a conda environment. Your script gets killed by zsh at about 9GB of (compressed) virtual memory occupied (the parameter I=438 at this point). Does this mean that the issue is with the conda environment? Perhaps it interferes with the VMM in macOS…
By the way it does not seems that, after zsh kills your script, the VM gets released. Could you share a way to do that?
0 -
-
-
Hi Giordano,
My guess is that it is not conda and just how MacOS handles memory and processes.
I also guess that although it appears that virtual memory is being consumed that this is not really an issue and it will be reclaimed when needed.
- Riley
0 -
-
Thanks,
that is what is odd though: macOS is supposed to use virtual memory until there is free space in disc, and on my disk there are still hundreds of GBs available as it is an almost empty 500GB ssd, with df -h displaying 360 GB available in the VM volume. Further, a similar code to yours with C++ can use far more virtual memory with macOS (130GB). If conda environments do not differ fundamentally from standard python venvs, could the problem be python itself then, and limitations within its garbage collector? I also find odd that while the gurobi code can reclaim up to 40GB of virtual memory, the script you wrote is killed after only 9GB of virtual memory…
EDIT: it indeed macOS VMM, not conda, causing this, as you predicted. Essentially the documentation on macOS VMM does not tell the full story. I have been in conversation with engineers on the apple developer forum and I received confirmation that in addition to the vague description that the whole ssd can be used by swapfiles, the VMM manager put caps on each process available total memory. These caps are far less than the total available. I am now looking into how to bypass the VMM automatic swapping behaviour in order to allocate a swapfile to the process I want to be unconstrained by nothing other than my swapfile. I am not sure how to do this yet, as I am only familiar with Linux.
0 -
-
-
Thanks for posting the update Giordano!
0 -
-
In conclusion, the only way to do this is to create a file of desired size (dd from dev/zero) and then use mmap to back the memory-consuming processes with this file, rather than leaving it to the VMM as by default. Is there a way in gurobipy, before doing model.optimize(), to set some parameters so that this mmapping is possible? Note that this may be different from saving nodefiles to disk by setting the corresponding parameter, since I think that way is cumulative only because it runs out of space faster, whereas the standard behaviour of gurobi is to save memory by deleting nodes that are fathomed as the bounds improve (I noticed many times memory freeing up by a couple of GBs as the optimisation proceeds).
0 -
-
-
Hi Giordano,
Is there a way in gurobipy, before doing model.optimize(), to set some parameters so that this mmapping is possible?
There isn't (or any of our other APIs). If there was some way of doing this it would have to be done outside of Gurobi somehow.
- Riley
0 -
サインインしてコメントを残してください。
コメント
7件のコメント