🔥Let’s Do DevOps: Building an Azure DevOps-based ARM CI/CD for Azure Cloud

Mar 09, 2021

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

Over the past few years, I’ve worked to build up a highly scalable and secure terraform CI/CD platform based on Azure DevOps for many internal teams. We now maintain 150+ terraform pipelines across both AWS and Azure (I keep hearing teams might require GCP, but none have demanded it yet!), with ~50 runs per day. That amounts to some serious computer, and a great deal of investment into our processes and technologies to keep the whole stack from falling over. I’ve written extensively about how and what, if you’re interested in reading it — just check out my profile.

And now that it all works, we’re looking to the future. What other languages do we want to support? Terraform works well, but there are some weaknesses it has which we’re able to overcome with other languages.

The one we’re most interested in supporting is ARM. Terraform fully supports Azure, but there are enough rough edges remaining in the AzureRM Terraform provider (including a state-breaking bug that remained for almost 6 months and was only fixed after much advocacy) that we expect some teams to soon require an alternative.

ARM has been in the news recently due to an evolution in the language used to configure it. ARM has historically relied on JSON, which is a finicky language. If you’ve missed a double quote or didn’t close one of the many brackets, your code doesn’t work. Too bad for you.

However, the Azure team has been working on a new product, Bicep, which is a Terraform-like syntax that is translated by Azure’s native PowerShell of az cli tooling to JSON on the back-end, hidden from the user. In this way, you get all the good parts — easy language syntax, more intuitive and declarative resource construction, and none of the icky JSON “which bracket did I fail to close” game.

resource KylerBicepVnet 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: BicepVnetName
  location: 'eastus'
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.3.0.0/16'
      ]
    }
    virtualNetworkPeerings: []
    enableDdosProtection: false
  }
}

Let’s focus on the ARM (with pure json) side for now. Our Terraform pipelines use a strong convention where they call a main.tf file, and that file calls children files using module calls. This pattern has worked well for us, and permits scaling out (more child calls), rather than up (bigger main.tf, root files). It has the added OCD-friendly benefit of permitting us to put terraform files into named folders to organize them.

We had a desire to do the same thing for ARM. Our users are used to this pattern, and it helps to organize the chaos of many resources across many verticals. ARM by default permits you to do this:

New-AzResourceGroupDeployment \
 -TemplateSpecId main.json \
 -ResourceGroupName ResourceGroupName

That permits you to specify a single main.json template spec, as well as a parameters file if it exists. While the ARM language recently introduced the ability to call “Linked Templates”, which operate as blocks of resources that are deployed together. This exactly satisfies what we’re looking for. Here’s the syntax.

{
  "type": "Microsoft.Resources/deployments",
  "apiVersion": "2020-06-01",
  "name": "vNetChild",
  "properties": {
    "mode": "Incremental",
    "templateLink": {
      "relativePath": "./modules/vnet/vnet.json"
    }
  }
}

However, there’s an unintuitive requirement here. Either the templates needs to be available via http and on a public repo (heck no are we publishing our proprietary modules and security conventions to a public repo), OR you need to use an alternate “package and then execute” method. This method works perfectly, but it’s by no means intuitive to find the information on $MSFT’s support site. The process works like this.

First, use the above syntax to call one or many child modules using the relative path to your main.json template file.

Second, package up the main.json, the parameters file, as well as any references linked template into a Template Spec. This name unfortunately doesn’t mean anything, but in practice it’s a bundle of code that permits the server-side operations ARM relies on to operate properly without access to your computer or other CI/CD storage resources. Building the Template Spec looks like this:

New-AzTemplateSpec \
 -Name UniqueTemplateSpecName \
 -Version "0.9" \
 -ResourceGroupName ResourceGroupName \
 -Location eastus \
 -TemplateFile main.json \
 -Force

The files are packaged up, tagged with a version number, and sent to Azure. For our internal convention, we selected 0.9 version as our testing version, and 1.0 as a deploy version. That keeps any streams from getting crossed in test vs deploy processes since all our operations are now on a remote code artifact, rather than the local code-base.

Third, we grab the GUID of the Template Space that’s now created and store it. We’ll use this right away to create a deployment.

$template_spec_id = (Get-AzTemplateSpec \
 -ResourceGroupName ResourceGroupName \
 -Name UniqueTemplateSpecName \
 -Version "0.9").Versions.Id

Fourth, we create a “WhatIf” deployment. This correlates pretty well to Terraform’s terraform plan function. It’s not nearly as polished yet, but $MSFT is clearly working here to improve it.

New-AzResourceGroupDeployment \
 -WhatIf \
 -TemplateSpecId $template_spec_id \
 -ResourceGroupName ResourceGroupName \
 -TemplateParameterFile parameters.json

It’ll look like this when run:

Credit: SaiGunaranjan Andhra, permission granted

If all goes well, and we’re ready to apply the changes, we just remove the -WhatIf flag:

New-AzResourceGroupDeployment \
 -TemplateSpecId $template_spec_id \
 -ResourceGroupName ResourceGroupName \
 -TemplateParameterFile parameters.json

The code to test this out in your own Azure DevOps environment is here. This will all run on cloud builders and utilize pre-existing Resource Groups and Service Connections to connect to Azure and execute your ARM testing and deployments.

The code for all of this is published here:

GitHub - KyMidd/AzureDevOps_ArmPipeline: Linked to blog…
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com

Good luck out there!
kyler

Let's Do DevOps

Discussion about this post