Skip to main content

I have sufficient memory but get memory error running the code on HPC.

Answered
Araz Nasirian
Gurobi-versary
Conversationalist
First Question

I have a decomposition code written in julia and run it code on HPC and I am getting this error about memory which I don’t think it is related to the memory because it works in my labtop very well. I changed the memory from 32 to 128, still have the same error.  In the last line of error it says the problem is in line 299. This is the code starting from line 299.

for H in 5:5
    global df_result, df
  for T in [10, 20, 40]
    J = H
    for Ω in [20, 40, 80, 160, 200]
        df = initiate(H,J,T,Ω)
      @show typeof(df)
      df_result = vcat(df_result, df)
      print(df_result)
      CSV.write("cut_LBBD_8_9.csv", df_result)
    end
end
  ###################################
ERROR: LoadError: Gurobi Error 10001:
Stacktrace:
[1] _check_ret
 @ ~/.julia/packages/Gurobi/xvBAY/src/MOI_wrapper/MOI_wrapper.jl:393 [inlined]
 [2] Gurobi.Env(; output_flag::Int64, memory_limit::Nothing, started::Bool)
 @ Gurobi ~/.julia/packages/Gurobi/xvBAY/src/MOI_wrapper/MOI_wrapper.jl:103
 [3] Env
 @ ~/.julia/packages/Gurobi/xvBAY/src/MOI_wrapper/MOI_wrapper.jl:100 [inlined]
 [4] Gurobi.Optimizer(env::Nothing; enable_interrupts::Bool)
 @ Gurobi ~/.julia/packages/Gurobi/xvBAY/src/MOI_wrapper/MOI_wrapper.jl:324
 [5] Optimizer (repeats 2 times)
 @ ~/.julia/packages/Gurobi/xvBAY/src/MOI_wrapper/MOI_wrapper.jl:322 [inlined]
 [6] _instantiate_and_check(optimizer_constructor::Type{Gurobi.Optimizer})
 @ MathOptInterface ~/.julia/packages/MathOptInterface/3gomH/src/instantiate.jl:94
 [7] instantiate(optimizer_constructor::Type; with_bridge_type::Type{Float64})
 @ MathOptInterface ~/.julia/packages/MathOptInterface/3gomH/src/instantiate.jl:149
 [8] set_optimizer(model::Model, optimizer_constructor::Type; add_bridges::Bool)
 @ JuMP ~/.julia/packages/JuMP/Z1pVn/src/optimizer_interface.jl:110
 [9] Model(optimizer_factory::Type; add_bridges::Bool)
 @ JuMP ~/.julia/packages/JuMP/Z1pVn/src/JuMP.jl:221
 [10] Model
@ ~/.julia/packages/JuMP/Z1pVn/src/JuMP.jl:220 [inlined]
 [11] #33
@ ./none:0 [inlined]
 [12] iterate
@ ./generator.jl:47 [inlined]
 [13] collect_to!
@ ./array.jl:782 [inlined]
 [14] collect_to_with_first!
@ ./array.jl:760 [inlined]
 [15] collect(itr::Base.Generator{Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}, var"#33#39"})
@ Base ./array.jl:734
 [16] solve_sub_prob(χʳ::Matrix{Float64}, H::Int64, J::Int64, T::Int64, Ω::Int64)
@ Main ~/EJOR_review.jl:115
 [17] initiate(H::Int64, J::Int64, T::Int64, Ω::Int64)
@ Main ~/EJOR_review.jl:172
 [18] top-level scope
@ ~/EJOR_review.jl:304
in expression starting at /home/595/an4627/EJOR_review.jl:299

Related to

0

Comments

4 comments

Date Votes
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    Hi Araz,

    The code part you are showing does not contain any Gurobi elements, Gurobi seems to be called in one of the functions that are called from here (initiate).

    The Gurobi error 10001 is raised when a component in the solver tries to allocate additional memory for internal use and does not get it from the operating system. So, there seems to be an issue with memory on your HPC machine. You could try to get more details about the memory consumption of the run with Julia tools or via the operating system.

    Best regards,
    Mario

    0
  • Araz Nasirian
    Gurobi-versary
    Conversationalist
    First Question

    I assume the problem is related to this part of the code that I copied and pasted below.

    Running this code on HPC I request 32GB of memory.

    • When I set the parameters as H=J=7 and T=\Omega=10 it solves the problem and consumes 1.2 GB of memory.
    • When I set the parameters as H=J=7 and T=40, \Omega=320 it says it consumes 1.43 GB of memory. It also gave me Gurobi Error 10001

    I have two questions:

    1.  Is there anyway of writing this code more efficiently?
    2.  Why I don't get Guroby error when I run this code in my Surface labtop with 8GB of RAM?

     

    function solve_sub_prob(χʳ,H,J,T,Ω)
        sp =[ Model(Gurobi.Optimizer) for t in 1:T, ω in 1:Ω]
        θ_1 = fill(0.0,T,Ω)
        γ_1 = fill(0.0,J,T,Ω)
        st_1 = fill(0.0,T,Ω)
        for v in 1:T
            for u in 1:Ω
                @variable(sp[v,u], 0 ≤ γ[1:J , [v] , [u] ], Int )
                @variable(sp[v,u], α[1:H , 1:J, [v] , [u] ], Bin )
                @objective(sp[v,u], Min, sum(zᶜ[v,j] * γ[j,v,u] for j in 1:J) )
                @constraint(sp[v,u], con_23b[h in 1:H , j in 1:J], α[h,j,v,u] ≤ χʳ[h,j] )
                @constraint(sp[v,u], con_23c[h in 1:H], sum( α[h,j,v,u] for j in 1:J) ≤ 1 )
                @constraint(sp[v,u], con_23d[j in 1:J], sum( α[h,j,v,u] for h in 1:H ) + γ[j,v,u] ≥ D[j,v,u] )
                optimize!(sp[v,u])
                θ_1[v,u] = objective_value(sp[v,u])
                γ_1[:,v,u] = round.(value.(γ[:,v,u]) , digits=1)
                st_1[v,u] = solve_time( sp[v,u] )
            end
        end
        return θ_1, γ_1 , sum(st_1)
    end

     

    0
  • Mario Ruthmair
    Gurobi Staff Gurobi Staff

    How do you measure the memory consumption? I am surprised that the memory usage for 12,800 (T,\Omega) pairs only slightly increases compared to 100 pairs.

    To your questions:

    1. Since the model for each (T,\Omega) pair seems independent from the others, you could create the Model(Gurobi.Optimizer) in the inner loop and free it again after retrieving the results.
    2. This is certainly interesting, so obviously, there is a difference in the setup on your laptop and the one on the cluster. You request 32 GB on the cluster, could this probably be shared memory, and you actually have much less available? Are there other differences in your setup, e.g., with respect to software versions, build workflow, etc.?
    0
  • Araz Nasirian
    Gurobi-versary
    Conversationalist
    First Question

    Thank you for your hint

    I redefine the function as below and it works well now.

    Now, the solution time is much better. :)

    function solve_sub_prob(χʳ,H,J,T,Ω)
        θ_1 = fill(0.0,T,Ω)
        γ_1 = fill(0.0,J,T,Ω)
        st_1 = fill(0.0,T,Ω)
        for v in 1:T
            for u in 1:Ω
                sp = Model(Gurobi.Optimizer)
                @variable(sp, 0 ≤ γ[1:J, [v], [u]], Int)
                @variable(sp, α[1:H, 1:J, [v], [u] ], Bin )
                @objective(sp, Min, sum(zᶜ[v,j] * γ[j,v,u] for j in 1:J) )
                @constraint(sp, con_23b[h in 1:H , j in 1:J], α[h,j,v,u] ≤ χʳ[h,j] )
                @constraint(sp, con_23c[h in 1:H], sum( α[h,j,v,u] for j in 1:J) ≤ 1 )
                @constraint(sp, con_23d[j in 1:J], sum( α[h,j,v,u] for h in 1:H ) + γ[j,v,u] ≥ D[j,v,u] )
                optimize!(sp)
                θ_1[v,u] = objective_value(sp)
                γ_1[:,v,u] = round.(value.(γ[:,v,u]) , digits=1)
                st_1[v,u] = solve_time( sp )
            end
        end
        return θ_1, γ_1, sum(st_1)
    end
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post