In previous articles we looked at Databricks concepts and architecture, as well as the various architecture and licensing options for using Gurobi on Databricks. In this article we will focus on getting Gurobi up and running on a Databricks cluster. The main steps are to install the Gurobi library and configure Gurobi to use your license. There are multiple ways of doing this; below we describe the most common ones.

Installing Gurobi

Instead of running pip install within individual notebooks, use the following steps:

• Go to your cluster definition
• Click the “Libaries” tab page
• Click “Install new”
• Choose “pypi” and type “gurobipy”
• Click “Install”

You should now be able to run import gurobipy as gp in any notebook.

Setting up your license

There are multiple options for storing your licensing information on Databricks and using it to initialize Gurobi. Below we show the three most common options.

Using a license file

Using a license file is often the easiest way to get started and works with any third-party modeling framework as well. By storing your license on Databricks-accessible storage and informing Gurobi where to find the license file, Gurobi will automatically read the license parameters on initialization. Note that this approach requires DBFS (Databricks File System), which is not available with the Community Edition.

The steps are as follows:

• Retrieve your license and store the result as gurobi.lic on your local machine
  
  • WLS license: Download your WLS license from https://license.gurobi.com/
  • Compute Server: Create your client license file pointing to the Compute Server
  • Instant Cloud: Download your cloud license from https://cloud.gurobi.com/
• Upload the license to Databricks
  • In your notebook, click “File”, “Upload data to DBFS…”
  • Select the “/FileStore/tables” folder
  • Then upload the gurobi.lic file
• Point Gurobi to this license file
  • Go to your cluster definition again
  • Open the “Configuration” tab page andclick “Advanced options” at the bottom
  • Under the environment variables, add GRB_LICENSE_FILE=/dbfs/FileStore/tables/gurobi.lic
  • When you save, the cluster should restart

Here is a quick test to see if everything is set up correctly:

import os
path = os.getenv('GRB_LICENSE_FILE')
if path is None:
    print('Gurobi license path not set')
else:
    print(f'Expected license in {path}')
    print('Found' if os.path.isfile(path) else 'Not found')

Using environment variables

Instead of letting Gurobi retrieve license information from a file, you can also pass the license information to Gurobi explicitly in code. Note that this approach works fine with the official gurobipy API and some modeling frameworks, but not all. Key here is the ability to explicitly construct a Gurobi environment in which you then build your models, while some modeling frameworks take care of environment construction and hide the ability to set license information upon construction.

Defining an environment variable

Create an environment variable for each license parameter (e.g. LICENSEID, WLSACCESSID, WLSSECRETfor a WLS license) as follows:

• Go to your compute cluster definition
• On the first tab page, scroll down to Advanced options
• At the bottom you should see a field called Environment variables
• Here you can add variables with NAME=VALUE syntax, one variable per line

Retrieving and using environment variables

Assuming you have used the names above, you can now construct your Gurobi environment and model as follows.

import os
with gp.Env(params={
  'LICENSEID': os.getenv('LICENSEID'), 
  'WLSACCESSID': os.getenv('WLSACCESSID'), 
  'WLSSECRET': os.getenv('WLSSECRET')
}) as env:
  with gp.Model(env=env) as model:
    # Construct and solve your model

Using secrets

Databricks also offers a mechanism for storing your license details as secrets. You would store each license parameter (e.g. LICENSEID, WLSACCESSID, WLSSECRETfor a WLS license) in a separate Databricks secret. The procedure for doing that depends on the cloud platform you're using.

Then you can retrieve each secret using dbutils.secrets.get; find more information here. You can pass the secrets directly to the Gurobi environment, similar to the procedure above for environment variables.

The benefit of this approach is that your license information is stored securely and can be managed centrally. Make sure not to store your license data in temporary variables or (worse) write them to outputs.

Using custom containers

In order to use (custom) containers for your cluster nodes, you should first enable this option. Depending on your cloud platform and policies, you may need help from your administrator to see the options mentioned below.

• Go to your workspace
• Click your name on the top-right and choose Admin settings
• Go to the Workspace settings tabpage
• Look for the Cluster group of options
• Here you should see Container services as a checkbox; toggle that checkbox
• Click Confirm

Now you should be able to customize your compute cluster:

• Go to your compute settings
• On the first tabpage, look for the Advanced options at the bottom
• Here you should see a Docker tabpage where you can choose a container image for your nodes