🔥Let’s Do DevOps: Using the Tofu/Terraform AzApi Provider to Find All the Subnets Everywhere!🚀
🔥Let’s Do DevOps: Using the Tofu/Terraform AzApi Provider to Find All the Subnets Everywhere!🚀
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can…
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
A note as we start — I’ve always been an open source kid, and I’ll continue to be so. To reflect that, I’ll be using OpenTofu/Tofu primarily, rather than Terraform, which is now not an open source tool due to a relicensing by Hashicorp. That said, at this point all the code I’ll share will work on both platform exactly the same. Let me know if you want to hear me expand on why I’ve made this decision or other topics here.
Also! If you prefer video to reading, I gave this article as a talk here: https://www.youtube.com/watch?v=THbB8tLJRPY
Hey all!
I’ve had a series of projects recently that I’ve historically told folks that “sorry, tofu can’t do that,” and as my skills have expanded I’ve found other ways to do so. One example is to find all the subnets within an entire Azure subscription and get all their GUIDs. Another is to find all the VMs matching a name pattern and then grab their primary private IP to put into a list.
Tofu is capable of doing all this when it’s in the same terraform state, or when it has exact information about what it’s looking up, like how many servers to find, and their exact names, and that the resource group they live in exists. If any of that information is incomplete (like, of course it will be in real production environments with variable counts of resources), Tofu falls on its face.
However, I’ve been digging into the Azure API (aka, AzApi) Provider, an alternate Azure provider to the very common AzureRm provider, and it’s capable of ✨amazing✨ things!
I’ll dig into two major examples, and share how I’ve been able to solve them, as well as every step of the way how I’ve resolved the problems and gotten them working. If you’d rather skip right to the code, scroll to the bottom for the github repo link.
Thanks all, let’s do this!
Find the Names and IDs of ALL the Subnets in other Subscriptions
We have our tofu configuration spread out over several layers, and across several different subscriptions. In this way we can have a dev subscription, stage, prod, and others as required. All of those are deployed using different terraform pseudo-workspaces.
Which is to say, we have a lot of different terraform states that are deploying subnets!
And we need to permit-list access for these subnets to a single storage account that is configured in another terraform layer altogether. How do we identify all the subnets among other subscriptions that we need to add to the permit-list when terraform runs? How do we keep it up to date? What if another subnet is added in any of those layers, do we have to manually update the storage account’s permit-list?
The AzureRM provider has an answer for this, sort of — the “azurerm_subnet” data source. However, this data source requires an exact name for both the resource group and subnet in order to find its information. I don’t want to have a list of all the subnet names and their resource groups to iterate through to call.
| data "azurerm_subnet" "example" { | |
| name = "backend" | |
| virtual_network_name = "production" | |
| resource_group_name = "networking" | |
| } |
Surely there’s a better way? Well, yes, but not in the AzureRM provider.
AzAPI Provider to the Rescue!
The Azure API provider is an entirely separate provider from the (very very common) AzureRM provider. It’s basically an endpoint for json responses from the Azure APIs. It’s often quite a bit less intuitive to use than the standard AzureRM resouces, but it offers a ton of flexibility, that we’ll explore here.
The primary data source we’ll be using here is called the azapi_resource_list, and it looks like the following. You can see we’re asking for a partial list of all resources, filtered for two things:
A specific resource type, here,
resourceGroupsA specific parent node ID, which is a subscription ID
| data "azapi_resource_list" "listBySubscription" { | |
| type = "Microsoft.Resources/resourceGroups@2022-09-01" | |
| parent_id = "/subscriptions/00000000-0000-0000-0000-000000000000" | |
| response_export_values = ["*"] | |
| } |
That data call is a huge json package of all the resources and their attributes. That json package isn’t exactly usable at first glance. I switched to the console terraform console and printed out the results of the data call with data.azapi_resource_list.dev_rgs to see what the response looks like. We get an ID attribute in a clean map key value pair, and the output is a huge escaped json string, which isn’t very useful.
| > data.azapi_resource_list.dev_rgs | |
| { | |
| "id" = "/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups" | |
| "output" = "{\"value\":[{\"id\":\"/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/Ue2RG\",\"location\":\"eastus2\",\"name\":\"Ue2RG\",\"properties\":{\"provisioningState\":\"Succeeded\"},\"tags\":{\"Environment\":\"Dev\",\"Team\":\"Hosting Team\",\"Terraform\":\"true\"},\"type\":\"Microsoft.Resources/resourceGroups\"},{\"id\":\"/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/NetworkWatcherRG\",\"location\":\"eastus2\",\"name\":\"NetworkWatcherRG\",\"properties\":{\"provisioningState\":\"Succeeded\"},\"type\":\"Microsoft.Resources/resourceGroups\"},{\"id\":\"/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/use2-rg\",\"location\":\"eastus2\",\"name\":\"use2-rg\",\"properties\":{\"provisioningState\" |
However, we can use the jsondecode() function to read the packaged json string and make it actual json we can filter, like this:
| > jsondecode(data.azapi_resource_list.dev_rgs.output) | |
| { | |
| "value" = [ | |
| { | |
| "id" = "/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/Ue2RG" | |
| "location" = "eastus2" | |
| "name" = "Ue2RG" | |
| "properties" = { | |
| "provisioningState" = "Succeeded" | |
| } | |
| "tags" = { | |
| "Environment" = "Dev" | |
| "Team" = "Hosting Team" | |
| "Terraform" = "true" | |
| } |
Filtering is written in a rather unintuitive syntax, but you can get it to work. We look at the value string of the json, and iterate over every entry with the splat operator ([*]), and then filter for the attribute “name”. On that same level is the attribute “id” which is just as easy to get.
Now we have the information we require, which is every resource group in a subscription based on name.
| > jsondecode(data.azapi_resource_list.dev_pod_rgs.output).value[*].name | |
| [ | |
| "Ue2RG1", | |
| "Ue2RG2", | |
| "Ue2RG3", | |
| "Ue2RG4", |
That methodology of searching the entire subscription for resource types is something we’ll rely on heavily in this blog and with these tools.
But About Those Subnets…
Okay, now that we’re on the same page about how we’ll find the information, let’s circle back to our problem-set — we need to find all the subnets in several other subscriptions. First of all, let’s register providers for the other subscriptions.
Tofu language doesn’t yet let us use iteratives on providers, so we need to individually register each provider for each other subscription. Let’s focus just on one foreign subscription, which we’ll call “dev”.
On lines 1–4 we establish some locals with the subscription IDs of the subscriptions we need to query. This is a great idea because we’ll use these IDs a lot.
On line 7 we establish a provider for the “dev” subscription, and use alias “dev”. Aliases with providers let us specify that certain resources or data calls should use that specific provivider (and really, subscription) to call data.
We also register the AzureRM provider for the same subscription. There’s no conflict here — these are entirely separate providers, so that they are pointed at the same subscription doesn’t matter at all.
| locals { | |
| account_id_dev = "797446e8-xxxx-xxxx-xxxx-11111111111" | |
| account_id_stg = "cfd60351-xxxx-xxxx-xxxx-22222222222" | |
| account_id_prd = "a8592e40-xxxx-xxxx-xxxx-33333333333" | |
| } | |
| provider "azapi" { | |
| alias = "dev" | |
| subscription_id = local.account_id_dev | |
| skip_provider_registration = true | |
| } | |
| provider "azurerm" { | |
| alias = "dev" | |
| subscription_id = local.account_id_dev | |
| skip_provider_registration = true | |
| features {} | |
| } |
Let’s start with a data call to find all the resource groups in the Dev account (line 1–6).
Then we’ll use a locals block to run some logic. First we build our list of resource group names, and then we do a for/each on this list with a conditional — if we can meet a regex string of “p\d\d\d-net”. That’ll look pretty foreign unless you’re good at regex — it means we’re looking for the character “p” followed by exactly 3 digits/numbers and then exactly the string “-net”. If we meet that regex string, the resource group name remains in the list, if not, it’s removed.
| data "azapi_resource_list" "dev_rgs" { | |
| provider = azapi.dev | |
| type = "Microsoft.Resources/resourceGroups@2022-09-01" | |
| parent_id = "/subscriptions/${local.account_id_dev}" | |
| response_export_values = ["*"] | |
| } | |
| # Identify all RG names, filter by regex string | |
| locals { | |
| dev_pod_rg_names = [for rg in jsondecode(data.azapi_resource_list.dev_rgs.output).value[*].name : rg if can(regex("p\\d\\d\\d-net", rg))] | |
| } |
Next we’ll use the same resource_list data call again, but this time looking for subnets. Note the parent_id that’s constructed based on the subscription ID of dev, as well as the each.value. Each.value is part of the iterative set on each loop by the for_each logic on line 2 — we’re iterating over every resource group, and finding all the subnets in each.
Then on line 11, we’re doing a bunch of cool stuff. First, we’re reading in all the data pulled from every iteration of line 1 (the looped data call), and then we’re iterating over it and putting it into a new map of maps.
| data "azapi_resource_list" "dev_subnets" { | |
| for_each = toset(local.dev_pod_rg_names) | |
| provider = azapi.dev | |
| type = "Microsoft.Network/virtualNetworks/subnets@2022-09-01" | |
| parent_id = "/subscriptions/${local.account_id_dev}/resourceGroups/${each.value}/providers/Microsoft.Network/virtualNetworks/dev-rg" | |
| response_export_values = ["*"] | |
| } | |
| # Preprocess | |
| locals { | |
| dev_subnet_preprocess = { for rg_name, rg_values in data.azapi_resource_list.dev_subnets[*] : rg_name => rg_values } | |
| } |
That’s a pretty complex data structure to work with, so let’s flatten it out. Here’s the whole post-processing locals block. We have individual lookups for dev, stg, and prod (line 2–4) with pretty complex structures. We need to combine and flatten those into easily iterable lists, so let’s do stuff starting on line 5.
Let’s work inside to out, so let’s start on line 8 — we’re iterating over the complex map generated on line 2, and we’re jsondecode()-ing the values, and then filtering for names. That’ll be a flat map. We’re doing the same on line 11 and 14. So we have a list of lists.
We use the concat() function on line 6 to combine them all into a single list, and then we use flatten to turn our “list of lists” into just… a “list”.
We do the same on line 18 but for the “id” attribute instead of the “name” attribute.
| locals { | |
| dev_subnet_preprocess = { for rg_name, rg_values in data.azapi_resource_list.dev_subnets[*] : rg_name => rg_values } | |
| stg_subnet_preprocess = { for rg_name, rg_values in data.azapi_resource_list.stg_subnets[*] : rg_name => rg_values } | |
| prd_subnet_preprocess = { for rg_name, rg_values in data.azapi_resource_list.prd_subnets[*] : rg_name => rg_values } | |
| subnet_names = flatten( | |
| concat( | |
| [ | |
| for rg in local.dev_subnet_preprocess["0"] : jsondecode(rg.output).value[*].name | |
| ], | |
| [ | |
| for rg in local.stg_subnet_preprocess["0"] : jsondecode(rg.output).value[*].name | |
| ], | |
| [ | |
| for rg in local.prd_subnet_preprocess["0"] : jsondecode(rg.output).value[*].name | |
| ] | |
| ) | |
| ) | |
| subnet_ids = flatten( | |
| concat( | |
| [ | |
| for rg in local.dev_subnet_preprocess["0"] : jsondecode(rg.output).value[*].id | |
| ], | |
| [ | |
| for rg in local.stg_subnet_preprocess["0"] : jsondecode(rg.output).value[*].id | |
| ], | |
| [ | |
| for rg in local.prd_subnet_preprocess["0"] : jsondecode(rg.output).value[*].id | |
| ] | |
| ) | |
| ) | |
| } |
And there you have it, a simple flat list of all the subnets, and a separate list of all the subnet IDs, of all the subnets in 3 other subscriptions, all called at runtime of terraform, and easily ingestible by other policies, like storage account permit-lists.
Summary
I also have an example where we go to find all the primary IPs of VMs based on a name pattern, but this has gotten long. I’ll create a follow-up article when I have time to go over that as well.
In this article we talked about how the common AzureRM tofu provider isn’t capable of finding the information we require, at least at the scale we want to work at. We talked about how the AzApi provider works, and how to troubleshoot and work with the huge json packages it returns, and we walked through how to use that code and methodology to find dozens of hundreds of subnets (more?) across other subscriptions, and how to combine the data in a simple, useful way.
For now, here’s a link to the aggregated config that we deployed. We talked about the “FindingSubnets_OtherSubscriptions”, and I’ll write up “FindingVMs_RGMaybeExist” in a future article.
Thanks everyone! Good luck out there.
kyler