Memory leak with multi threaded application
Answered
Hi,
I'm trying to write a program (using the C++ API) where I copy a model on several different threads to do some independent processing of several variations of a problem. When I do, I get memory leaks that seem to stem from GRBEnv::GRBEnv(bool).
Below is a MWE:
#include <iostream>
#include <thread>
#include <mutex>
#include "gurobi_c++.h"
static std::mutex mtx;
void foo(GRBModel model)
{
GRBEnv env2 = GRBEnv();
mtx.lock();
GRBModel model2 = GRBModel(model, env2);
mtx.unlock();
model2.set(GRB_IntParam_Threads, 1);
model2.optimize();
}
void bar(GRBModel model)
{
GRBEnv env2 = GRBEnv();
mtx.lock();
GRBModel model2 = GRBModel(model, env2);
mtx.unlock();
model2.set(GRB_IntParam_Threads, 1);
model2.optimize();
}
int main()
{
GRBEnv env = GRBEnv();
GRBModel model = GRBModel(env);
model.set(GRB_IntParam_Threads, 1);
GRBVar* x = model.addVars(2, GRB_BINARY);
model.addQConstr(x[0] + x[1], GRB_EQUAL, 0.0);
GRBLinExpr obj = x[0];
model.setObjective(obj);
std::thread first (foo, model);
std::thread second (bar, model);
first.join();
second.join();
std::cout << "foo and bar completed.\n";
return 0;
}
____________________________
When I use valgrind to check for memory leaks, I receive:
==297419== HEAP SUMMARY:
==297419== in use at exit: 1,801 bytes in 56 blocks
==297419== total heap usage: 987 allocs, 931 frees, 2,181,350 bytes allocated
==297419==
==297419== 16 bytes in 1 blocks are definitely lost in loss record 3 of 11
==297419== at 0x484A2F3: operator new[](unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==297419== by 0x122D8C: GRBModel::addVars(int, char) (in /home/adelgren/Insync/Google_Drive/USNA/Scholarship/Lakmali/QMOSCP/Cpp_implementation/Thread_Example/solver)
==297419== by 0x11AF8D: main (threaded.cpp:34)
==297419==
==297419== 1,785 (40 direct, 1,745 indirect) bytes in 1 blocks are definitely lost in loss record 11 of 11
==297419== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==297419== by 0x861CA35: ???
==297419== by 0x8616C1B: ???
==297419== by 0x8604FBA: ???
==297419== by 0x860C068: ???
==297419== by 0x79B6D9E: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:273)
==297419== by 0x5270108: GRBgetusername (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==297419== by 0x526B885: PRIVATE00000000009da1bf (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==297419== by 0x526A6A0: PRIVATE00000000009d60cc (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==297419== by 0x52695E8: PRIVATE00000000009d75e8 (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==297419== by 0x522A51F: GRBloadenvadv (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==297419== by 0x11C5E1: GRBEnv::GRBEnv(bool) (in /home/adelgren/Insync/Google_Drive/USNA/Scholarship/Lakmali/QMOSCP/Cpp_implementation/Thread_Example/solver)
==297419==
==297419== LEAK SUMMARY:
==297419== definitely lost: 56 bytes in 2 blocks
==297419== indirectly lost: 1,745 bytes in 54 blocks
==297419== possibly lost: 0 bytes in 0 blocks
==297419== still reachable: 0 bytes in 0 blocks
==297419== suppressed: 0 bytes in 0 blocks
==297419==
==297419== For lists of detected and suppressed errors, rerun with: -s
==297419== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
____________________________________________
Can you please advise, and let me know if I'm doing something incorrectly.
Best Regards,
-Nate
-
Hi Nate,
Thank you for the reproducible example, I was able to whittle it down to the following:
#include "gurobi_c++.h"
int main()
{
GRBEnv env = GRBEnv();
GRBModel model = GRBModel(env);
GRBVar* x = model.addVars(2, GRB_BINARY);
return 0;
}It looks like the issue is produced by not freeing up the memory allocated to the x variables, and can be resolved by adding
delete[] x;
- Riley
0 -
Hi Riley,
Thanks very much for your help on this!
Unfortunately, it didn't solve my problem -- at least not entirely.
Certainly, I was too hasty in making my MWE and definitely should have added "delete[] x", as you recommend.
When I saw your comment last night, I checked my original program to see if I had forgotten a similar statement, but I hadn't. Interestingly, I ran the program using valgrind and received no memory leaks. I didn't change anything in the code; the only difference was that I was using my personal laptop, rather than my work machine. When I arrived at work this morning, I added the delete statement that you suggested to my MWE, and I'm still getting the memory leak. I tried placing the delete statement before the calls to std::thread and after the join statements, but I get the same result either way.
I'm not sure if you can help me further, since I know it will now be difficult to reproduce the issue, but any advice you can give would be much appreciated. I'm happy to provide any additional information you think may be helpful. For now, I can tell you that both machines are using Gurobi 11.0.0 build v11.0.0rc2. Additionally, my personal machine -- the one that does NOT produce a leak -- is running Linux Mint 20 with kernel 5.4.0-169-generic and the valgrind version is 3.15.0, while my work machine -- the one that produces the leak -- is running Ubuntu 22.04.3 LTS with kernel 6.2.0-39-generic and the valgrind version is 3.18.1.
0 -
Hi Nate,
Can you see if you get a memory leak with the following modification to the main function?
int main()
{
GRBEnv env = GRBEnv();
GRBModel model = GRBModel(env);
model.set(GRB_IntParam_Threads, 1);
GRBVar x = model.addVar(0.0, 1.0, 0.0, GRB_BINARY, "x");
model.addConstr(x == 0.0, "c1");
GRBLinExpr obj = x;
model.setObjective(obj);
std::thread first (foo, model);
std::thread second (bar, model);
first.join();
second.join();
std::cout << "foo and bar completed.\n";
return 0;
}- Riley
0 -
Hi Riley,
Unfortunately, I still get the leak even with the modified main function.
I did also notice that my personal machine has an Intel CPU whereas my work machine has an AMD. Not sure if it helps ...
Thanks again!
-Nate
0 -
Thanks Nate,
Can you post the Valgrind output from this latest run?
We may need to swing this over to our dev team, but I will see if my teammates have any ideas.
- Riley
0 -
Sure Riley,
Here it is:
==348169==
==348169== HEAP SUMMARY:
==348169== in use at exit: 1,785 bytes in 55 blocks
==348169== total heap usage: 985 allocs, 930 frees, 2,194,508 bytes allocated
==348169==
==348169== 1,785 (40 direct, 1,745 indirect) bytes in 1 blocks are definitely lost in loss record 10 of 10
==348169== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==348169== by 0x861CA35: ???
==348169== by 0x8616C1B: ???
==348169== by 0x8604FBA: ???
==348169== by 0x860C068: ???
==348169== by 0x79B6D9E: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:273)
==348169== by 0x5270108: GRBgetusername (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==348169== by 0x526B885: PRIVATE00000000009da1bf (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==348169== by 0x526A6A0: PRIVATE00000000009d60cc (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==348169== by 0x52695E8: PRIVATE00000000009d75e8 (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==348169== by 0x522A51F: GRBloadenvadv (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==348169== by 0x11C6C1: GRBEnv::GRBEnv(bool) (in /home/adelgren/Insync/Google_Drive/USNA/Scholarship/Lakmali/QMOSCP/Cpp_implementation/Thread_Example/solver)
==348169==
==348169== LEAK SUMMARY:
==348169== definitely lost: 40 bytes in 1 blocks
==348169== indirectly lost: 1,745 bytes in 54 blocks
==348169== possibly lost: 0 bytes in 0 blocks
==348169== still reachable: 0 bytes in 0 blocks
==348169== suppressed: 0 bytes in 0 blocks
==348169==
==348169== For lists of detected and suppressed errors, rerun with: -s
==348169== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)Thanks!
0 -
Hi Nathan,
What C++ std are you using to compile this? I have tried reproducing this on a similar machine with no success (tried 11, 17, and 2a).
Also, what type of license are you using?Additionally, I have lost the output that you get (I accidentally deleted it while formatting your post).
Cheers,
David0 -
Hi David,
I've tried compiling with -std=c++17 and -std=c++11, same result either way. I've also tried with -O3, -O2, -O and with and without -g. All produce the same result. I've tried with the g++ compiler (version 11.4.0) and with g++-9 (which is version 9.5.0, and happens to be the version installed on the machine that does not produce the issue), both give the same. I have not tried downgrading to valgrind version 3.15, though.
I'm using named-user academic licenses on both machines.
Just so you have everything, the file I'm currently running has the following contents:
#include <iostream>
#include <thread>
#include <mutex>
#include "gurobi_c++.h"
static std::mutex mtx;
void foo(GRBModel model)
{
GRBEnv env2 = GRBEnv();
mtx.lock();
GRBModel model2 = GRBModel(model, env2);
mtx.unlock();
model2.set(GRB_IntParam_Threads, 1);
model2.optimize();
}
void bar(GRBModel model)
{
GRBEnv env2 = GRBEnv();
mtx.lock();
GRBModel model2 = GRBModel(model, env2);
mtx.unlock();
model2.set(GRB_IntParam_Threads, 1);
model2.optimize();
}
int main()
{
GRBEnv env = GRBEnv();
GRBModel model = GRBModel(env);
model.set(GRB_IntParam_Threads, 1);
model.set(GRB_IntParam_OutputFlag, 0);
GRBVar x = model.addVar(0.0, 1.0, 0.0, GRB_BINARY, "x");
model.addConstr(x == 0.0, "c1");
GRBLinExpr obj = x;
model.setObjective(obj);
std::thread first (foo, model);
std::thread second (bar, model);
first.join();
second.join();
std::cout << "foo and bar completed.\n";
return 0;
}and running
valgrind --leak-check=full ./solver
from my terminal produces:
==358636== Memcheck, a memory error detector
==358636== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==358636== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==358636== Command: ./solver
==358636==
Set parameter Username
Academic license - for non-commercial use only - expires 2025-01-10
Set parameter Threads to value 1
Set parameter Username
Set parameter Username
Academic license - for non-commercial use only - expires 2025-01-10
Set parameter Threads to value 1
Gurobi Optimizer version 11.0.0 build v11.0.0rc2 (linux64 - "Ubuntu 22.04.3 LTS")
CPU model: AMD Ryzen 7 PRO 5850U with Radeon Graphics, instruction set [SSE2|AVX|AVX2]
Thread count: 8 physical cores, 16 logical processors, using up to 1 threads
Optimize a model with 0 rows, 0 columns and 0 nonzeros
Model fingerprint: 0xf9715da1
Coefficient statistics:
Matrix range [0e+00, 0e+00]
Objective range [0e+00, 0e+00]
Bounds range [0e+00, 0e+00]
RHS range [0e+00, 0e+00]
Presolve time: 0.06s
Presolve: All rows and columns removed
Iteration Objective Primal Inf. Dual Inf. Time
0 0.0000000e+00 0.000000e+00 0.000000e+00 0s
Solved in 0 iterations and 0.11 seconds (0.00 work units)
Optimal objective 0.000000000e+00
Academic license - for non-commercial use only - expires 2025-01-10
Set parameter Threads to value 1
Gurobi Optimizer version 11.0.0 build v11.0.0rc2 (linux64 - "Ubuntu 22.04.3 LTS")
CPU model: AMD Ryzen 7 PRO 5850U with Radeon Graphics, instruction set [SSE2|AVX|AVX2]
Thread count: 8 physical cores, 16 logical processors, using up to 1 threads
Optimize a model with 0 rows, 0 columns and 0 nonzeros
Model fingerprint: 0xf9715da1
Coefficient statistics:
Matrix range [0e+00, 0e+00]
Objective range [0e+00, 0e+00]
Bounds range [0e+00, 0e+00]
RHS range [0e+00, 0e+00]
Presolve time: 0.00s
Presolve: All rows and columns removed
Iteration Objective Primal Inf. Dual Inf. Time
0 0.0000000e+00 0.000000e+00 0.000000e+00 0s
Solved in 0 iterations and 0.00 seconds (0.00 work units)
Optimal objective 0.000000000e+00
foo and bar completed.
==358636==
==358636== HEAP SUMMARY:
==358636== in use at exit: 1,785 bytes in 55 blocks
==358636== total heap usage: 985 allocs, 930 frees, 2,194,508 bytes allocated
==358636==
==358636== 1,785 (40 direct, 1,745 indirect) bytes in 1 blocks are definitely lost in loss record 10 of 10
==358636== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==358636== by 0x861CA35: ???
==358636== by 0x8616C1B: ???
==358636== by 0x8604FBA: ???
==358636== by 0x860C068: ???
==358636== by 0x79B6D9E: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:273)
==358636== by 0x5270108: GRBgetusername (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==358636== by 0x526B885: PRIVATE00000000009da1bf (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==358636== by 0x526A6A0: PRIVATE00000000009d60cc (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==358636== by 0x52695E8: PRIVATE00000000009d75e8 (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==358636== by 0x522A51F: GRBloadenvadv (in /home/adelgren/Downloads/gurobi11.0.0_linux64/gurobi1100/linux64/lib/libgurobi.so.11.0.0)
==358636== by 0x11C6D1: GRBEnv::GRBEnv(bool) (in /home/adelgren/Insync/Google_Drive/USNA/Scholarship/Lakmali/QMOSCP/Cpp_implementation/Thread_Example/solver)
==358636==
==358636== LEAK SUMMARY:
==358636== definitely lost: 40 bytes in 1 blocks
==358636== indirectly lost: 1,745 bytes in 54 blocks
==358636== possibly lost: 0 bytes in 0 blocks
==358636== still reachable: 0 bytes in 0 blocks
==358636== suppressed: 0 bytes in 0 blocks
==358636==
==358636== For lists of detected and suppressed errors, rerun with: -s
==358636== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)Interestingly, on my other machine, the number of allocs and frees is also quite different, but the total number of bytes used is very similar -- off by about 2000.
I can post the contents of my makefile, if it helps.
Thanks!
-Nate
0 -
Unfortunately I've also not been able to reproduce on a VM with
Ubuntu 22.04.3 LTS
kernel 6.2.0-39-generic
valgrind 3.18.1
g++ 11.4and different combinations of std and optimizations.
==958== HEAP SUMMARY:
==958== in use at exit: 0 bytes in 0 blocks
==958== total heap usage: 3,613 allocs, 3,613 frees, 2,318,444 bytes allocated0 -
Thanks Riley,
It's very strange! I guess I don't have anything additional to add at this point. If I'm able to reproduce the issue on any other machine, or find something that seems to resolve it, I'll post an additional comment.
I appreciate your help!
0 -
No worries Nate. If you have admin rights over that machine it could be an interesting experiment to install multipass and spin up VMs on that machine with different OS versions and kernels and see if the problem persists.
- Riley
0 -
I created a ticket to track this internally.
Cheers,
David0 -
Thanks Riley and David. I'm happy to help in whatever way I can moving forward.
-Nate
0 -
We investigated this and found out that it was due to several things and not related to multi-threading.
The leak occurs when checking the license (this is done when starting an environment) using a named-user license.
On Linux, we use a function from glibc to check the user name which is known to be leaky. Particularly, as also seen in the valgrind output in this post, this is glibc's \(\texttt{getpwuid_r}\). This has been discussed e.g.:- ubuntu - Memory-leaks when using libssh - Stack Overflow (particularly, see answer https://stackoverflow.com/a/15279341).
- valgrind reports getpwuid() leaks in c++ with Ubuntu
Seeing this leak depends on the glibc version of the machine, the OS, Valgrind options (see below), and the Gurobi version (v11.0.0 updated to a minimum glibc version of 2.17, see Detailed Release Notes).
It is safe to ignore this leak (people on Stackoverflow also say this).
Valgrind even has the option:
--run-libc-freeres=no|yes free up glibc memory at exit on Linux? [yes]
Additionally, many other license types (e.g. WLS) do not use this leaky function.
Cheers,
David0
Please sign in to leave a comment.
Comments
14 comments