Azure Configuration

MC² Client provides an interface to directly launch Azure resources – you can launch VMs, blob storage accounts, or storage containers. To do so, you’ll need to configure some parameters that will tell MC² Client how to set up your Azure resources. You should specify the path to this configuration in the ["launch"]["azure_config"] section of your Global Configuration.

Below is an example of the Azure configuration with comments about each field.

# An unique identifier for the head node and workers of this cluster.
cluster_name: default

# The total number of workers nodes to launch in addition to the head
# node. This number should be >= 0.
num_workers: 0

# Cloud-provider specific configuration.
provider:
   type: azure
   # https://docs.microsoft.com/en-us/azure/confidential-computing/virtual-machine-solutions

   # Location of resources
   # For a full list of regions that have VMs with enclave support,
   # go to the following link and look for DCs-v2 series VMs
   # https://azure.microsoft.com/en-us/global-infrastructure/services/?products=virtual-machines&regions=all
   location: eastus

   # The name of the resource group that will contain all resources
   resource_group: mc2-client-dev

   # The name of the storage account to create
   storage_name: mc2storage

   # The name of the storage container to create
   container_name: blob-container-1

   # If left blank or commented out, the default subscription ID
   # from the Azure CLI will be used
   # subscription_id:

# How MC2 will authenticate with newly launched nodes.
auth:
   # The username to use to SSH to launched nodes
   ssh_user: mc2

   # You must specify paths to matching private and public key pair files
   # Use `ssh-keygen -t rsa -b 4096` to generate a new SSH key pair
   ssh_private_key: ~/.ssh/id_rsa
   ssh_public_key: ~/.ssh/id_rsa.pub

# Provider-specific config for the head node, e.g. instance type.
head_node:
   azure_arm_parameters:
      # https://docs.microsoft.com/en-us/azure/confidential-computing/virtual-machine-solutions
      # The DCs_v2 VMs support Intel SGX
      vmSize: Standard_DC2s_v2

      # If launching a minimal Ubuntu machine
      # (and manually installing using setup commands)
      imagePublisher: Canonical
      imageOffer: UbuntuServer
      imageSku: 18_04-lts-gen2
      imageVersion: latest

# Provider-specific config for worker nodes, e.g. instance type.
worker_nodes:
   azure_arm_parameters:
      # https://docs.microsoft.com/en-us/azure/confidential-computing/virtual-machine-solutions
      # The DCs_v2 VMs support Intel SGX
      vmSize: Standard_DC2s_v2

      # If launching a minimal Ubuntu machine
      # (and manually installing using setup commands)
      imagePublisher: Canonical
      imageOffer: UbuntuServer
      imageSku: 18_04-lts-gen2
      imageVersion: latest

##############################################################################
#       Everything below this can be ignored - you likely won't have to      #
#       modify it.                                                           #
##############################################################################

# Files or directories to copy to the head and worker nodes. The format is a
# dictionary from REMOTE_PATH: LOCAL_PATH, e.g.
file_mounts: {
   # This script installs Open Enclave
   "~/install_oe.sh" : "scripts/install_oe.sh",

   # This script builds Spark 3.1.1 from source
   "~/build_spark.sh" : "scripts/build_spark.sh",

   # This script downloads a pre-built Spark 3.1.1 binary
   "~/install_spark.sh" : "scripts/install_spark.sh",

   # This script builds Opaque SQL from source
   "~/build_opaque.sh" : "scripts/build_opaque.sh",

   # This script installs Secure XGBoost from source
   "~/install_secure_xgboost.sh" : "scripts/install_secure_xgboost.sh"
}

# List of commands that will be run before `setup_commands`. If docker is
# enabled, these commands will run outside the container and before docker
# is setup.
initialization_commands:
   # Get rid of annoying Ubuntu message
   - touch ~/.sudo_as_admin_successful

# List of shell commands to run to set up nodes.
# Note: Use empty list if using image
setup_commands:
   # This script installs Open Enclave on the node
   - chmod +x ~/install_oe.sh
   - source ~/install_oe.sh
   # This script installs Apache Spark on the node
   - chmod +x ~/install_spark.sh
   - source ~/install_spark.sh
   # This script installs Opaque SQL on the node
   - chmod +x ~/build_opaque.sh
   - source ~/build_opaque.sh
   # This script installs Secure XGBoost on the node
   - chmod +x ~/install_secure_xgboost.sh
   - source ~/install_secure_xgboost.sh

# Custom commands that will be run on the head node after common setup.
# Set to empty list if using image
head_setup_commands: []

# Custom commands that will be run on worker nodes after common setup.
# Set to empty list if using image
worker_setup_commands: []

# Command to start MC2 on the head node.
# Set to empty list if using image
head_start_mc2_commands:
- cd $SPARK_HOME; ./sbin/start-master.sh

# Command to start MC2 on worker nodes.
# Set to empty list if using image
worker_start_mc2_commands:
- cd $SPARK_HOME; ./sbin/start-slave.sh $MC2_HEAD_IP:7077