In this collaborative effort Icertis worked with Microsoft to define a new release process, and a target solution architecture using Azure Resource Manager (ARM) and to optimize the release pipeline.

Customer profile

Icertis is the leading provider of contract lifecycle management in the cloud. Icertis Contract Management (ICM) is an innovative, easy-to-use platform that is highly configurable and continually adapts to complex business needs. ICM is used to manage 2.5+ million contracts, by 750,000+ users, in 90+ countries and 40+ languages.

The hackfest team:

  • Subodh Patil - Architect Products at CTO Office Icertis
  • Sachin Chavan - Senior Architect Products at CTO Office Icertis
  • Sanjay Pawaskar - Associate Director - Products at Icertis
  • Shadja Chaudhari - DevOps Engineer
  • Sujith Nair - DevOps Engineer
  • Maninderjit Bindra – Senior Technical Evangelist, Microsoft
  • Ritesh Modi – Senior Technical Evangelist, Microsoft

The team during the value stream mapping and hackfest - image 1

The team during the value stream mapping and hackfest - image 2

ICM application context

During a discussion with the customer in context of the ICM application, the customer indicated that they wanted to reduce the release timelines for the ICM application, and the creation/refresh times of environments in general. To understand the bottle necks we conducted a value stream mapping exercise with Icertis, in which we reviewed the ICM solution architecture, processes, and tools used for developing and releasing new features to production.

Value Stream Mapping

Following are some of the key points from this exercise:

  • Each customer has their own deployment of the ICM application, in a separate Azure Subscription. Many of the customers have multiple ICM environments (QA, staging etc.)
  • ICM solution can be customized by building customer specific extensions to common core code base
  • Major releases for a customer happen about every 6 months and enhancements/bug fixes happen monthly
  • ICM can be provisioned using several different deployment models as agreed with the customer. These models include VM only deployments. The most common deployment model uses Cloud Services to deploy the main components. The diagram below gives a high level view of the standard deployment:

current standard deployment model overview

The key issues identified during the VSM activity are detailed in the following section

Problem Statement

  • The nightly build against the common core code base with functional and integration tests is automated however there are no automated tests executed against the customer specific extensions. This means in few cases issues are found after customer specific extensions are manually merged.
  • If new environment for a customer is needed or existing environment needs to be updated then a largely manual process is followed which includes steps where email handoffs take place and steps where the DevOps team performs some manual tasks/executes scripts. This process has a dependency on availability of DevOps team person and the whole process can take up to 3 days for a single environment. Icertis wants to increase the frequency for major and minor updates to the ICM deployments
  • Current deployments use the classic deployment model. Icertis wants to move to a new standard Azure Resource Manager (ARM) based deployment model for ICM.

Icertis wants to move standard ICM deployments to ARM model. They want to enhance the release process to include automated deployments to customer environments with automated testing

Solution

Scope of hackfest

We agreed that the scope of the hackfest would be first to define the ARM based ICM deployment Architecture and then to create the build and release pipeline for one of the new ICM customers. Single Environment was to be considered in the release pipeline with automated deployment.

ARM based ICM deployment Architecture

Couple of options were discussed to migrate the classic cloud service based components to ARM. First was App services and the second was VM scale sets (VMSS) based deployment. Currently the ICM solution has certain requirements like need to store certificate files on specific locations on virtual machine. It was decided that for scope of this hackfest we would migrate the cloud service based components to VMSS. The icertis team would also explore how much effort it would take to move to App Services after the hackfest. target standard deployment model overview

DevOps practices in scope

  • Release management: It was decided that the entire release process will be optimized. Right from requesting creation/refresh of an environment to automated merging of customer specific extensions to automated deployment to environments.
  • Infrastructure as code and Configuration Management: All steps involved in setting up and updating the infrastructure, including steps to bake the virtual machine images which would be needed by the VMSS tier, would be in configuration-managed files. This would include Azure Resource Manager templates, power shell scripts, Packer files.

Build and Release pipeline overview

Overview diagram

Build and Release pipeline overview

Description

Process to raise request for creation/Refresh of a customer environment was modified. This would now be done by creating a TFS work item. Once created the request would be reviewed and approved by a release approver. After the work item is approved the customer specific build and release pipeline is triggered. Attributes or the TFS work item will serve as configuration parameters for the build and release. When a new customer is onboarded, the TFS release based on the standard release template will need to be configured. The customer specific build would first build the customer specific extensions, then merge customer specific extension with the core common code and create the deployment artifacts. The first step of the release creates DB/Cache/Queueing/Search Tier components using the first ARM Template. This template would output configurations like DB connection string, etc. In the next step packer is used to bake the VM images for the Web/API tiers VMSS. Configurations like DB connections string are also baked in. The second ARM template then uses custom VM Images to create VMSS based Web/API tiers.

Release Implementation details

  • Step 1. Create DB, cache, queueing and search tier components :
    • The DB , Cache, Queueing and Search tier components are created based on an ARM template. The pricing tiers, sizes of the components are passed as environment variables for the release. The template below is not the actual template used for creating the ICM resources, it shows a single SQL Server and DB being created to demonstrate the release pipeline.
{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "sqlAdminUser": {
      "type": "string",
      "metadata": {
        "description": ""
      }
    },
    "sqlAdminPassword": {
      "type": "string",
      "metadata": {
        "description": ""
      }
    }
  },
  "variables": {},
  "resources": [
    {
      "name": "icertsql",
      "type": "Microsoft.Sql/servers",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "SqlServer"
      },
      "apiVersion": "2014-04-01-preview",
      "dependsOn": [],
      "properties": {
        "administratorLogin": "[parameters('sqlAdminUser')]",
        "administratorLoginPassword": "[parameters('sqlAdminPassword')]",
        "version": "12.0"
      },
      "resources": [
        {
          "name": "dbnamesrr",
          "type": "databases",
          "location": "[resourceGroup().location]",
          "tags": {
            "displayName": "Database"
          },
          "apiVersion": "2014-04-01-preview",
          "dependsOn": [
            "icertsql"
          ],
          "properties": {
            "edition": "Standard",
            "collation": "SQL_Latin1_General_CP1_CI_AS",
            "maxSizeBytes": "1073741824",
            "requestedServiceObjectiveName": "S0"
          }
        }
      ]
    }
  ],
  "outputs": {
    "sqlFQDN": {
      "type": "string",
      "value": "[reference('icertsql').fullyQualifiedDomainName]"
    }
  }
}
  • The New-AzureRmResourceGroupDeployment is used to deploy this ARM template. Parameters like resource group name, SQL server username and password are supplied using values of release environment variables, and the unique TFS release id is used as the unique deployment name. Command to deploy this template is shown below.
  New-AzureRmResourceGroupDeployment -Name $uniquedeploymentid -ResourceGroupName $resgrp -TemplateFile azuredeploy.json -sqlAdminUser $sqladminuser -sqlAdminPassword $sqladminpass
  • Following the deployment of this ARM template, output parameters like db fqdn can be retrieved by using the Get-AzureRmResouceGroupDeployment command with the unique deployment name which was used in the New-AzureRmResourceGroupDeployment. Next step shows how this can be done.

  • Step 2. The configuration values like db FQDN values are fetched using the following script

$sqlServerFQDN = (Get-AzureRmResourceGroupDeployment -ResourceGroupName $resgrp -Name $uniquedeploymentid).Outputs.sqlFQDN.Value
  • Step 3. Bake VM images which will be used by App and API tier VMSS
    • Packer setup: Packer downloads are available at [https://www.packer.io/downloads.html] : Packer version 0.12.0 was downloaded on the build server as a one time activity
    • Powershell script which triggers the packer build is shown below:

      param(
      $ARM_TENANT_ID = 'release environment variable',
      $ARM_SUBSCRIPTION_ID = 'release environment variable',
      $ARM_OBJECT_ID = 'release environment variable',
      $ARM_APPLICATION_ID = 'release environment variable',
      $ARM_CLIENT_SECRET = 'release environment variable',
      $ARM_RESOURCE_GROUP = 'release environment variable',
      $ARM_STORAGE_ACCOUNT = 'release environment variable',
      $packerExeParentPath = 'path to packer executable on build server',
      $location = '',
      $path1 = '',
      $directory = '',
      $file_name = "Web.config",
      $sqlServerFQDN = 'received from previous step'
      
      )
      
      #This script build VHDs which will be used by VMSS based tiers
      [Environment]::SetEnvironmentVariable("ARM_TENANT_ID","$ARM_TENANT_ID" )
      [Environment]::SetEnvironmentVariable("ARM_SUBSCRIPTION_ID","$ARM_SUBSCRIPTION_ID" )
      [Environment]::SetEnvironmentVariable("ARM_OBJECT_ID","$ARM_OBJECT_ID" )
      [Environment]::SetEnvironmentVariable("ARM_APPLICATION_ID","$ARM_APPLICATION_ID" )
      [Environment]::SetEnvironmentVariable("ARM_CLIENT_SECRET","$ARM_CLIENT_SECRET" )
      [Environment]::SetEnvironmentVariable("ARM_RESOURCE_GROUP","$ARM_RESOURCE_GROUP" )
      [Environment]::SetEnvironmentVariable("ARM_STORAGE_ACCOUNT","$ARM_STORAGE_ACCOUNT" )
      [Environment]::SetEnvironmentVariable("LOCATION","$location" )
      [Environment]::SetEnvironmentVariable("DIRECTORY","$directory" )
      [Environment]::SetEnvironmentVariable("FILE_NAME","$file_name" )
      [Environment]::SetEnvironmentVariable("PATH1","$path1" )
      [Environment]::SetEnvironmentVariable("SQL_SERVER_FQDN","$sqlServerFQDN" )
      
      $output = Invoke-Expression -Command "$packerExeParentPath\packer.exe build bake-API-tier-VM-Image.json"
      

      As shown below parsing the $output for “OSDiskUri” gives the image uri of the baked VM image. This image uri is used in subsequent steps.

      $vmImageUri = ( $output.Where({$_ -like 'OSDiskUri:*'}) | Out-String ).Replace("OSDiskUri: ", "")
      

      The parameters are passed to the the powershell script by TFS. ARM_CLIENT_ID, ARM_RESOURCE_GROUP, ARM_STORAGE_ACCOUNT, ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID are used by Packer to save the baked virtual machine image VHD into the configured Azure storage account. The values of parameters like SQL server FQDN are replaced in the web.config file during the VM image baking process. Let us look at the contents of the packer template file bake-API-tier-VM-Image.json

      {
      "variables": {
          "azure_ad_tenant_id": "",
          "azure_subscription_id": "",
          "object_id": "",
          "app_id": "",
          "client_secret": "",
          "resource_group": "",
          "storage_account": "",
          "location": "",
          "apiCname": "",
          "packageStorageKey": "",
          "packageLocation": "",
          "current_path": "",
          "sql_server_fqdn": "",
      },
      "builders": [
          {
          "type": "azure-arm",
          "subscription_id": "",
          "tenant_id": "",
          "object_id": "",
          "client_id": "",
          "client_secret": "",
          "resource_group_name": "",
          "location": "",
          "vm_size": "Standard_D3_v2",
          "storage_account": "",
          "capture_container_name": "images",
          "capture_name_prefix": "packer",
          "os_type": "Windows",
          "image_publisher": "MicrosoftWindowsServer",
          "image_offer": "WindowsServer",
          "image_sku": "2012-R2-Datacenter",
          "image_version": "latest",
          "communicator": "winrm",
          "winrm_use_ssl": "true",
          "winrm_insecure": "true",
          "winrm_timeout": "3m",
          "winrm_username": "packer"
          }
      ],
      "provisioners": [
          {
          "type": "powershell",
          "inline": [
              "Import-Module ServerManager",
              "Install-WindowsFeature -Name NET-Framework-Features"
          ]
          },
          {
          "type": "powershell",
          "scripts": [
              "\\IIS_Config.ps1",
              "\\Replace_Tokens_In_Config_Files.ps1",
              "\\Script3.ps1",
              "\\Script4.ps1"
          ],
          "environment_vars": [
              "API_CNAME=",
              "PACKAGE_LOCATION=",
              "PACKAGE_STORAGE_KEY=",
              "SQL_SERVER_FQDN="
          ]
          },
          {
          "type": "windows-shell",
          "script": "\\serverhardening.bat"
          },
          {
          "type": "powershell",
          "script": "\\serverhardening.ps1"
          },
          {
          "type": "windows-shell",
          "pause_before": "10s",
          "scripts": [
              "\\sysprep.bat"
          ]
          }
      ]
      }
      

      In the initial section of this file packer variables are set. Most of them are set using environment variables. The Azure arm builder is used to specify details like the base Azure VM image to use. The next packer section is the provisioners section. The first power shell provisioner is used to install windows feature. The next power shell provisioner is used execute an array of powershell scripts to provision the VM (install / configure ICM including replacing tokens in configuration files). Environment variables are passed to this array of powershell scripts using the “environment_vars” . Next windows-shell provisioner and powershell provisioners harden the server. The final provisioner is the windows-shell provisioner used to execute sysprep to generalize the VM image.

      contents of sysprep.bat are as follows :

      %windir%\system32\sysprep\sysprep.exe /generalize /oobe /shutdown /quiet
      

      It must be noted that packer supports Powershell DSC provisioning which is ideal. In this case the icertis team wanted to re-use powershell scripts which they use for VM only deployments for some of the clients, hence Powershell DSC provisioning was not considered as this stage.

  • Step 4. Create VMSS based on custom VM images created in previous step
    • ARM template is used to create/update the UI/API tiers of ICM application which deployed to VMSS. This is done by updating the VM image associated with the VMSS by the VM image we get in the previous step. The template below is not the actual template used for creating the ICM resources, it demonstrates creation of VM scaleset using the custom vm image ($vmImageUri).
{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vmSSName": {
            "type": "string",
            "metadata": {
                "description": "The Name of the VM Scale Set"
            }
        },
        "instanceCount": {
            "type": "int",
            "metadata": {
                "description": "Number of VM instances to create in the scale set"
            }
        },
        "vmSize": {
            "type": "string",
            "allowedValues": [
                "Standard_D1",
                "Standard_DS1",
                "Standard_D2",
                "Standard_DS2",
                "Standard_D3",
                "Standard_DS3",
                "Standard_D4",
                "Standard_DS4",
                "Standard_D11",
                "Standard_DS11",
                "Standard_D12",
                "Standard_DS12",
                "Standard_D13",
                "Standard_DS13",
                "Standard_D14",
                "Standard_DS14"
            ],
            "metadata": {
                "description": "The size of the VM instances Created"
            }
        },
        "dnsNamePrefix": {
            "type": "string",
            "metadata": {
                "description": "The Prefix for the DNS name of the new IP Address created"
            }
        },
        "adminUsername": {
            "type": "string",
            "metadata": {
                "description": "The Username of the administrative user for each VM instance created"
            }
        },
        "adminPassword": {
            "type": "securestring",
            "metadata": {
                "description": "The Password of the administrative user for each VM instance created"
            }
        },
        "sourceImageVhdUri": {
            "type": "string",
            "metadata": {
                "description": "The source of the blob containing the custom image"
            }
        },
        "frontEndLBPort": {
            "type": "int",
            "metadata": {
                "description": "The front end port to load balance"
            },
            "defaultValue": 80
        },
        "backEndLBPort": {
            "type": "int",
            "metadata": {
                "description": "The front end port to load balance"
            },
            "defaultValue": 80
        },
        "probeIntervalInSeconds": {
            "type": "int",
            "metadata": {
                "description": "The interval between load balancer health probes"
            },
            "defaultValue": 15
        },
        "numberOfProbes": {
            "type": "int",
            "metadata": {
                "description": "The number of probes that need to fail before a VM instance is deemed unhealthy"
            },
            "defaultValue": 5
        },
        "probeRequestPath": {
            "type": "string",
            "metadata": {
                "description": "The path used for the load balancer health probe"
            },
            "defaultValue": "/iisstart.htm"
        }
    },
    "variables": {
        "addressPrefix": "10.0.0.0/16",
        "subnetName": "Subnet",
        "subnetPrefix": "10.0.0.0/24",
        "virtualNetworkName": "vmssvnet",
        "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]",
        "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]",
        "publicIPAddressName": "publicip1",
        "publicIPAddressID": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]",
        "nicName": "networkInterface1",
        "nicId": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]",
        "lbName": "loadBalancer1",
        "lbID": "[resourceId('Microsoft.Network/loadBalancers',variables('lbName'))]",
        "lbFEName": "loadBalancerFrontEnd",
        "lbWebProbeName": "loadBalancerWebProbe",
        "lbBEAddressPool": "loadBalancerBEAddressPool",
        "lbFEIPConfigID": "[concat(variables('lbID'),'/frontendIPConfigurations/',variables('lbFEName'))]",
        "lbBEAddressPoolID": "[concat(variables('lbID'),'/backendAddressPools/',variables('lbBEAddressPool'))]",
        "lbWebProbeID": "[concat(variables('lbID'),'/probes/',variables('lbWebProbeName'))]",
        "networkApi": "2016-03-30",
        "computeApi": "2016-03-30"
    },
    "resources": [
        {
            "apiVersion": "[variables('networkApi')]",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[variables('virtualNetworkName')]",
            "location": "[resourceGroup().location]",
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[variables('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[variables('subnetName')]",
                        "properties": {
                            "addressPrefix": "[variables('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "apiVersion": "[variables('networkApi')]",
            "type": "Microsoft.Network/publicIPAddresses",
            "name": "[variables('publicIPAddressName')]",
            "location": "[resourceGroup().location]",
            "properties": {
                "publicIPAllocationMethod": "Dynamic",
                "dnsSettings": {
                    "domainNameLabel": "[parameters('dnsNamePrefix')]"
                }
            }
        },
        {
            "apiVersion": "[variables('networkApi')]",
            "name": "[variables('lbName')]",
            "type": "Microsoft.Network/loadBalancers",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[concat('Microsoft.Network/publicIPAddresses/',variables('publicIPAddressName'))]"
            ],
            "properties": {
                "frontendIPConfigurations": [
                    {
                        "name": "[variables('lbFEName')]",
                        "properties": {
                            "publicIPAddress": {
                                "id": "[variables('publicIPAddressID')]"
                            }
                        }
                    }
                ],
                "backendAddressPools": [
                    {
                        "name": "[variables('lbBEAddressPool')]"
                    }
                ],
                "loadBalancingRules": [
                    {
                        "name": "weblb",
                        "properties": {
                            "frontendIPConfiguration": {
                                "id": "[variables('lbFEIPConfigID')]"
                            },
                            "backendAddressPool": {
                                "id": "[variables('lbBEAddressPoolID')]"
                            },
                            "probe": {
                                "id": "[variables('lbWebProbeID')]"
                            },
                            "protocol": "tcp",
                            "frontendPort": "[parameters('frontEndLBPort')]",
                            "backendPort": "[parameters('backEndLBPort')]",
                            "enableFloatingIP": false
                        }
                    }
                ],
                "probes": [
                    {
                        "name": "[variables('lbWebProbeName')]",
                        "properties": {
                            "protocol": "http",
                            "port": "[parameters('backEndLBPort')]",
                            "intervalInSeconds": "[parameters('probeIntervalInSeconds')]",
                            "numberOfProbes": "[parameters('numberOfProbes')]",
                            "requestPath": "[parameters('probeRequestPath')]"
                        }
                    }
                ]
            }
        },
        {
            "type": "Microsoft.Compute/virtualMachineScaleSets",
            "apiVersion": "[variables('computeApi')]",
            "name": "[parameters('vmSSName')]",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[concat('Microsoft.Network/loadBalancers/',variables('lbName'))]",
                "[concat('Microsoft.Network/virtualNetworks/',variables('virtualNetworkName'))]"
            ],
            "sku": {
                "name": "[parameters('vmSize')]",
                "tier": "Standard",
                "capacity": "[parameters('instanceCount')]"
            },
            "properties": {
                "overprovision": "true",
                "upgradePolicy": {
                    "mode": "Manual"
                },
                "virtualMachineProfile": {
                    "storageProfile": {
                        "osDisk": {
                            "name": "vmssosdisk",
                            "caching": "ReadOnly",
                            "createOption": "FromImage",
                            "osType": "Windows",
                            "image": {
                                "uri": "[parameters('sourceImageVhdUri')]"
                            }
                        }
                    },
                    "osProfile": {
                        "computerNamePrefix": "[parameters('vmSSName')]",
                        "adminUsername": "[parameters('adminUsername')]",
                        "adminPassword": "[parameters('adminPassword')]"
                    },
                    "networkProfile": {
                        "networkInterfaceConfigurations": [
                            {
                                "name": "nic1",
                                "properties": {
                                    "primary": "true",
                                    "ipConfigurations": [
                                        {
                                            "name": "ip1",
                                            "properties": {
                                                "subnet": {
                                                    "id": "[variables('subnetRef')]"
                                                },
                                                "loadBalancerBackendAddressPools": [
                                                    {
                                                        "id": "[variables('lbBEAddressPoolID')]"
                                                    }
                                                ]
                                            }
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                }
            }
        }
    ],
    "outputs": {
        "fqdn": {
            "value": "[reference(variables('publicIPAddressID'),providers('Microsoft.Network','publicIPAddresses').apiVersions[0]).dnsSettings.fqdn]",
            "type": "string"
        }
    }
}
  • The Command used to deploy vmss with custom vm image using sample template is given below. Parameter values are passed using release environment variables.
New-AzureRmResourceGroupDeployment -Name $uniquedeployment2id -ResourceGroupName $resgrp -TemplateFile vmss-custom-image.json -vmSSName $vmssname -sourceImageVhdUri $vmImageUri -instanceCount $instanceCount -vmSize $vmsize -adminUsername $vmadminusername -adminPassword $vmadminpassword -dnsNamePrefix $dnsNamePrefix

Conclusion

The collaborative value stream mapping activity resulted in the entire team agreeing that they needed quicker and reliable deployments to different environments.

During the first day of the hackfest we finalized

  • The new process to request for release/refresh of customer environments
  • The target ICM architecture
  • And the release process

These were all validated with the key icertis stakeholders including the CTO.

The benefits of the enhanced process are:

  • The process to request for new release / refresh of an environment is more streamlined. It is possible to track where the request is pending and for how long it has been pending.
  • Since the release process is automated, including steps to provision VMs (through Packer / IaC) there is no dependency on DevOps team resource being available for deployments and as a result we will have much quicker releases than previously possible.
    • Releases to new environments for customers where new release process has been implemented can now happen in a couple of hours as compared to 2-3 days earlier.
    • This has enabled also enabled icertis to bring down the major release cycle to every 2 months for customers where the new process has been implemented.

Since the hackfest the icertis team is optimizing the release pipeline further by adding automated functional and smoke tests for the provisioned environments. This new approach will be piloted for a new customer deployment, and will then be replicated for existing customers.

Icertis are also exploring the feasibility of moving the UI and App tiers for standard deployments to Azure App services.

A quote from the Icertis team:

“This hackefest happened at crucial juncture when icertis was planning to move to new Azure ARM base platform. It helped us not only to visualize the automation workflow, bring clarity but also to implement parts of DevOps pipeline. Icertis is really happy with this engagement and looking forward for more such engagements!”