🔥Let's Do DevOps: Dynamic Host Inventories in Azure on Ansible AWX/Tower

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

This article will be focused on AWX dynamic inventories from Azure. If you need to work with AWS, check out this article:

Let’s Do DevOps: Dynamic Host Inventories in AWS on Ansible AWX/Tower

In my last post we discussed what Ansible AWX/Tower is and how powerful it can be. We also went over how to build and deploy AWX to a local instance with Vagrant in less than 10 minutes.

So assuming you followed along, you now have a version of AWX running. Woot!

However, AWX/Tower isn’t a very intuitive software in my opinion, and it can help to get some help getting started.

The first thing I want to focus on building is dynamic inventories.

Dynamic Inventories: What Have You Done For Me Lately?

Static inventories are lists of hosts with IPs. Ansible churns through these happily, but modern networks churn constantly — building, redeploying, growing. Static lists just aren’t the thing anymore.

Rather, wouldn’t it be great if Ansible could keep track of hosts itself so we didn’t have to? And furthermore, wouldn’t it be cool if Ansible did this automatically all the time so we don’t have to worry about Ansible ever getting out of sync with our actual real deployed hosts?

And it is! With Dynamic Inventories.

Getting Started in Azure — Build an App Registration

Before we tell AWX how to talk to Azure, we need to configure an App Registration (basically a Service Account in legacy Active Directory) that we can auth to, and can have permission to read Azure Virtual Machine attributes.

Start in Azure in the “Azure Active Directory” blade. In the left panel, click on the App Registrations. Then in the top left click on “+ New registration”.

Assign a name to your AWX Reader App Registration. You can change this name later, but consider setting it to a permanent name now. If you want to limit access per subscription in AWX, you’ll need separate registrations.

Note that being able to read Virtual Machine metadata doesn’t grant any ability for AWX/Ansible to actually talk to the machine and tell it to do stuff, so you can always block it on the network, or via SSH key or windows login, too, but best practice is hosts should only be able to see the machines they can manage. That’ll make life simpler for your users.

The rest can be left unchanged. Click Register in the bottom left.

The App Registration will be created. It doesn’t have any rights yet — we’ll do that later, but we can start gathering the information we need. Here’s what the AWX credential page looks like, so let’s see what we need:

The ones we’ll fill out are:

• Subscription ID: This is the ID of the Azure subscription that the resources are in. We’ll assign this later.

• Client ID: This one we can get, it’s called the “Application (client) ID on this page. Copy it down somewhere.

• Client Secret: We’ll set this in a minute. It’s a secret value you shouldn’t share.

• Tenant ID: This one is also on this page. This is the Tenant ID — the Tenant that all your subscriptions are created within.

Okay, we have the client and tenant IDs, now let’s move on to the client secret. In the left sidebar of this page click on the “Certificates & Secrets” button.

You’ll see this page which of course shows no client secrets. We haven’t made any! Let’s change that. Click on the “+ New client secret” button to get started.

We don’t set the actual client secret, but we do give it a name. I’ll call it AwxMetadataReader here, but it’s a good idea to use something more specific here — to refer to the hostname of your server, or your group’s name, or something. There’s a limit on how many secrets you can create for an App Registration (usually, only two!), so if you need to revoke one to build a new one, it’s a great idea to have this be specific for what impact revoking the secret will have.

You also set how long the secret will expire. It’s a great security best practice to have your secrets expire, so I’ll set it to the lowest available value — 1 year. Then click on “Add”.

The client secret will be built. This value will only be shown now, so copy the “Value” string down somewhere secure. Remember, anyone on the internet can use this value to assume this user and do stuff, so don’t share it. Even a read-only role is dangerous to share.

If you navigate away from this page and the secret won’t show again, don’t worry — just delete the secret (there is a trash can icon next to it) and create a new one. No harm is done.

Azure Part 2: Assign Subscription Reader Rights

Now that our Azure App Registration is created, we’re done in Azure, right? Well, almost. The App Registration exists, and we have a secret, but it can’t DO anything yet — it has no permissions.

And we want it to have some rights, so let’s do it.

In the Azure portal navigate to the Subscriptions blade. Find the subscription you want to manage and click on it. While we’re here let’s copy down the subscription ID value — we’ll use that in a minute in AWX.

I have it redacted in this picture, but it’s the “Subscription ID” value. Copy that down.

Now let’s assign some permissions. Click on the “Access control (IAM)” button in the left bar.

Then click on “+ Add” in the top left of that window.

We’ll want to give our new App Registration access to this subscription with the “Reader” role so there’s no chance it can change anything. Set the role to “Reader” and then find your App Registration and click it.

Then hit save in the bottom right.

Boom, your new App Registration now has the right to access all resources and read their attributes, more than enough rights that we’ll need for AWX to populate a dynamic Azure inventory.

Note: This permission is admittedly pretty broad, even as a read-only user. If there’s more specific Virtual Machine metadata permissions we can assign let me know as a comment in this blog and I’ll update it!

AWX — Add App Registration Credentials

Now that the Azure credentials are ready, let’s switch back to our AWX server and tell it how to use them. AWX stores this information as a “Credential” object.

In the left panel, click on “Credentials” under the Resources banner. Then in the top right click on the “+” sign to start building a new credential.

Your page won’t look like this immediately, but it will fill out as we select options. Give your credential a name, an organization (default is fine for testing), and hit the hourglass next to the “Credential Type”.

A list of credential types will appear, and among them will be “Microsoft Azure Resource Manager”. Click the radio button next to that option and hit okay.

The App Registration info boxes along the bottom will appear — fill in the values you copied down here and then hit save.

Now AWX knows how to authenticate to Azure, but we need to tell it how to use this authentication.

Build a Git Repo

The next step involves telling AWX which plugins to install as well as providing it with a text file that shows it how to speak to Azure to import hosts. In reliably unintuitive style, you can’t just upload these files to AWX — rather, you need to put these files into a git repo, sync AWX to this git project, then run a sync command.

Thankfully, I’ve done some of this work for you to save some time.

KyMidd/AnsibleAwxAzureDynamicInventory
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build…github.com

GitHub makes it super easy to copy an existing repo, so let’s walk through what that looks like so you can build your own.

First, on your GitHub page in the top right, click on the “+” to add a new repo and click “Import repository”.

On this import page fill out the repository you want to copy (https://github.com/KyMidd/AnsibleAwxAzureDynamicInventory), and assign a new name for your repo, as well as if it’s public or private. Then click Begin Import in the bottom right. After a minute or two (or less), you’ll get an email that the repo has been copied, and you can go to the repo in your own org and see the files.

The code within this repo contains a file called collections/requirements.yml. This file tells AWX what collections it should install. You’ll see this pattern all over Ansible playbooks, so get used to it. AWX will see this file on sync and automatically download and install this collection server-side.

The file looks like the following. Note the name azure.azcollection is a collection that supports actions in Azure Cloud including synchronizing virtual machine metadata.

# Tower automatically installs these collections when project syncs
collections:
  - name: azure.azcollection

You’ll also see the script which describes how our Azure host sync should work. The script is pretty-human readable. We’re using the azure_rm plugin, and using env(ironmental variables) for authentication — we’ll go over that in a minute.

We’re using plain hostnames — this script normally appends 4 hex characters at the end of each name to support many hosts sharing the same hostname. If that’s something you have in your env, set this to false.

We’re looking for only running hosts, which is a great best practice — why target hosts that are off with Ansible jobs?

We also are creating Ansible AWX groups with the keyed_groups attribute, and setting the ansible_host attribute (that AWX uses as an IP or FQDN to connect to the host) as the fire private_ipv4_address of the host. If you need public IPs or FQDNs here, go for it.

The last line also helps us build an AWX Ansible group of all hosts in this inventory, which is useful for large-scale targeting.

I won’t go deep into this, but you can read about the available values here.

plugin: azure_rm
auth_source: env
plain_host_names: yes
exclude_host_filters:
- powerstate != 'running'
keyed_groups:
- prefix: tag
  key: tags
- prefix: azure_location
  key: location
hostvar_expressions:
  ansible_host: private_ipv4_addresses | first
conditional_groups:
  all_hosts_group: true

Note: If you build this repo as private (as you should for anything resembling a real enterprise project), you’ll need your AWX to authenticate with an SSH key. I’ll leave that as an exercise to the reader. In the following demo, I created the repo as public so no authentication at all is required to access the code.

Synchronize Git + AWX

Now that your new git repo is built, we need to tell AWX to read it. Head into AWX and find the “Projects” button on the left bar. Click into it, and then click the “+” in the top right to build a new project.

Start filling in the values — again, this page will fill out boxes as you make selections. The SCM (Source Code Management) type should be set to Git. The URL will be your URL — the URL of your own git repo that you built as a copy of mine.

You can skip the SCM Credential box since this is a public repo. If you set up yours as a private repo select the AWX credentials for the SSH key that can access this private repo.

Once saved, click back on the “Projects” button on the left bar, and find your new project. Mine’s the one on the bottom. Click the circular arrows to trigger a sync. It’ll take about a minute or so, and you’ll either get a green dot on the left (yay!) or a red dot (failed!). Regardless, click on the dot to see the output from Ansible that includes some diagnostic information for you to slog through.

If you got a green dot, congratulations! AWX will update the local copy from this git repo anytime something from this “project” is referenced, including the Azure script.

Finally, an Inventory

Now that we’ve laid the (so so very much) groundwork, we can actually go build our Inventory that utilizes the resources we’ve created to sync hosts.

Over on the left bar click on Inventory, then in the top right click on “+” to build a new inventory. We want a regular “Inventory”, and not a “smart inventory” which is something different and ironically not very smart.

Fill out the name and select an Organization for this inventory. Once done, hit save, then click on the “Sources” tab on the top, then hit on the green “+” to add a new “source” to this inventory.

As you can imagine, a “source” is a method of pulling hosts into this inventory. One of those sources is the script we are reading from git. Name this data source, and change the source type to “Sourced from a Project” — the git repo we synced AWX to earlier.

Click the magnifying glass under “Credential” and we’ll find the Azure Metadata Reader credentials we created earlier. Hit the radio button next to it and then Select.

Hit the magnifying glass under Project and find the git project we configured earlier. Then hit the button next to it and hit Select.

For whatever reason, AWX doesn’t recognize this file as a valid inventory file — I think it’s because it’s looking for a static inventory, rather than a query type file.

Regardless, we need to tell it where to look for the file. Click into the editable area and put the path to the Azure search file in our git repo into it. This is the path to enter. Then click it or hit enter and it’ll select that path.

inventories/azure/all_running_hosts.azure_rm.yml

Once done, it’ll look like this:

And now we only have a few more items. One, we need to check the “Update on Launch” check box. This means anytime this inventory is updated, it’ll sync to the real Azure environment and read the hosts. This will create a transparent action anytime a job is run which makes our inventories always accurate.

We also need to populate the Environmental Variables box at the bottom. The way credentials work in AWX (And in Ansible) is the credentials assign values to variables, but we need to map the credentials variable names to what the data source is expecting. To do that, set the environmental variables field to this:

AZURE_SUBSCRIPTION_ID: "{{AZURE_SUBSCRIPTION_ID}}"
AZURE_CLIENT_ID: "{{AZURE_CLIENT_ID}}"
AZURE_SECRET: "{{AZURE_SECRET}}"
AZURE_TENANT: "{{AZURE_TENANT}}"

When you’re done it’ll look like this:

Hit save and then at the top select SOURCES to jump back to our inventory data sources list.

Hit the circular arrows to begin an inventory sync. AWX will sync the git project, read the data query, authenticate to Azure and read the hosts, creating host entries with metadata and adding those hosts to groups within this inventory! Right? It’s a lot.

If you see a green cloud after a minute or two on the left, it means this all succeeded. Click on the Hosts button at the top to see all the imported hosts. and their groups.

You can also see all the groups that are created and can now be targeted:

Summary and Closing

AWX is an idiosyncratic but powerful platform. Things that should be easy, like uploading a single query file, turn into 5-step processes that require a git repo.

However, our users will never see that complexity. From their perspective, they have an always-correct inventory of hosts with groups based on Virtual Machine tags. Anytime they point a job template at this inventory (in totality, at specific groups, or at specific hosts), AWX will update it in the background before launching the job.

And don’t stop here — add multiple inventory sources to pull hosts from multiple accounts, or from other clouds or providers like VMware ESXi.

You can do it. Let’s do DevOps. Good luck out there!
kyler