Infoworks Tutorials

Deploy Infoworks for Databricks on Azure

In this fast, two part video tutorial, you will be introduced to deploying Infoworks for Databricks from the Azure Marketplace, configuring networking between Infoworks and Databricks, obtaining and deploying a Trial or Enterprise license, and creating the Environment Configuration needed for your new Infoworks node to control a specific Azure Databricks workspace.

Part A – Deploying Infoworks for Databricks from the Azure Marketplace, including network configuration

Part A – Follow the general steps below, for guidance on completing this tutorial in your own environment.

  1. Create new Vnet
    1. Home > Virtual Networks > Create virtual network
    2. Basics
      1. Subscription: (user subscription)
      2. Resource Group: tutorial-rg (create new)
      3. Instance Name: databricks-vnet
      4. Region: West US 2
      5. IP Addresses: (accept defaults)
      6. Security: (accept defaults)
    3. Review and Create (note address space octet)
  2. Create new Databricks instance
    1. Home > Azure Databricks > Create new Azure Databricks service
    2. Basics
      1. Subscription: (user subscription)
      2. Resource Group: tutorial-rg (select existing)
      3. Workspace Name: databricks-ws
      4. Location: West US 2
      5. Pricing Tier: Standard
    3. Networking
      1. Deploy with No Public IP: Yes
      2. Deploy in own Vnet: Yes
      3. Virtual Network: databricks-vnet
      4. Public Subnet: public-subnet / 10.0.1.0/24 (or match Vnet octet)
      5. Private Subnet: private-subnet / 10.0.3.0/24 (or match Vnet octet)
    4. Review and Create
  3. Review Databricks Configuration
    1. Verify MS Azure Databricks (DBX) instance settings
      1. Vnet Name: databricks-vnet
      2. Region: US West 2 (note for IWX configuration)
      3. URL: (copy DBX endpoint URL to notes)
    2. Launch DBX workspace, generate User Token from User Settings
      1. Comment: tutorial-[date]
      2. Lifetime: 90 days (default)
      3. User Token: (copy DBX user token to notes)
  4. Create new Infoworks instance
    1. Home > Marketplace > “Infoworks for Databricks” (select)
    2. (review Overview and Plans, emphasize Free Trial) > Create
    3. Basics
      1. Subscription: (select valid subscription)
      2. Resource group: tutorial-rg (select existing)
      3. Region: West US 2
      4. Virtual network: databricks-vnet (select existing)
      5. Subnet > Manage Subnet Configuration > Subnet
        1. Name: iwx-subnet
        2. IP Range: 10.0.2.0/29
        3. (Save)
      6. (step back to wizard)
      7. Subnet: iwx-subnet
    4. Configure Infoworks
      1. VM name: infoworks (accept default)
      2. VM user name: infoworks
      3. Authentication type: Password (default, note SSH Public Key)
      4. Password: (specify, and note password requirements)
      5. Confirm password: (same)
      6. Generate Public IP: (accept default)
      7. Infoworks Hostname: (accept default)
      8. Infoworks VM type: (change size, review, accept default)
    5. Deployment Notes
      1. (review need to see “outputs” after install)
      2. (note self-signed SSL and need to click through)
    6. Review and Create (review and note validation)
      1. (click Create, review deployment details, wait for completion)
      2. Outputs (copy both URL and SSH values to clipboard)

Part B – Licensing and configuring Infoworks to control a specific Azure Databricks workspace

Part B – Follow the general steps below, for guidance on completing this tutorial in your own environment.

  1. Navigate to new Infoworks instance
    1. Browse to new instance URL (copy from notes)
      1. Advanced (review self-signed SSL issue)
      2. Proceed to URL (copy Installation ID to notes)
      3. Click to License Key Request (complete form, Free Trial, defaults)
    2. Tab to email client
      1. (review License Key and Login Instructions)
      2. (note initial credentials and copy license key to notes)
      3. New Password: (specify, and note password requirements)
      4. (install license) (configure custom license)
      5. (review Environment Configuration)
  2. Configure Environment
    1. Environment
      1. (select) Define: Azure Databricks
      2. (Environment) Name: demo_environment_config1
      3. Metastore Type: Databricks-Internal
      4. Workspace URL: (from AZR portal)
      5. Databricks Token: (from Databricks workspace)
      6. Region: (same at AZR Databricks workspace)
    2. Storage
      1. Name: (default)
      2. Storage Type: DBFS (ok for demo/simple, in prod externalize)
      3. (review cloning, note sources are each assigned their own source)
      4. (save)
    3. Compute Details – Job Cluster Compute Templates (used for jobs)
      1. Name: (default)
      2. (accept defaults, region/workspace/token disabled b/c internal metastore, if external would need to point to distinct workspaces, which is why you externalize a metastore)
      3. Use Instance Pool: (review, unchecked)
      4. Worker Type: Standard_D8s_v3 (or other type with quote)
      5. Driver Type: Standard_D8s_v3 (or other type with quote)
      6. Max Allowed Workers: 4 (can override per job up to this limit)
      7. Enable Autoscale: (review, checked)
      8. Default Min: 1
      9. Default Max: 4
      10. Support for ML: (review, unchecked, recommend distinct template)
      11. (save, note copy/clone and ability to add big-job small-job templates, etc.)
    4. Compute Details – Interactive Cluster C.T. (used internally by IWX, sample data, etc.)
      1. (accept defaults)
      2. Worker Type: Standard_D8s_v3 (or other type with quota)
      3. Driver Type: Standard_D8s_v3 (or other type with quote)
      4. Max Allowed Worker Nodes: 2 (b/c small internal jobs, sample data, etc.)
      5. Number of Workers: 1
      6. Enable Autoscale: (review, checked)
      7. Support for ML: (review, unchecked)
      8. Terminate After (x) …: (default, checked)
      9. (save, return to Environments, view details)
    5. (Finish, review Environments list, view details, note cloning, no delete once used, etc..)
    6. (Go to Dashboard, begin Interactive Guide)