🔥Let’s Do DevOps: Reduce Noise in Terraform Plans

Hey all!

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

I’ve written in the past about my trials and tribulations designing and running a terraform and general CI/CD platform that’s used by 10+ teams to run ~130 terraform pipelines to deploy to 50+ environments across Azure and AWS.

When designing the system, I knew that’d it’d need to scale up and out to a great degree — any pattern and solution we chose would be stressed both by upward growth (running terraform and other deploys hundreds of times per day) and outward (scaling out to hundreds or maybe thousands of pipelines and workflows). Because of that, I’m extremely sensitive to:

• Good design patterns that make sense and can scale out ad infinitum

• Automating all that can be automated — lower the bar of knowledge that lets a dev or operations engineer take advantage of the system

I’ve also had to train all the users on this system and good patterns, and for every single person, I’ve stressed that they need to read the pull request validation before approving a PR. That pull request validation runs tflint as well as terraform init + validate + plan.

And for every single person I’ve trained, I’ve had to include the caveat that on the “terraform plan” stage there will be dozens and probably HUNDREDS of trash lines that they need to ignore and scroll past to get to the real information — what terraform plan says it’s going to do based on the PR.

In this (heavily redacted) photo you can see the “Refreshing state…” lines. One will be generated for each individual resource managed by terraform, and it can add up to thousands of cruft lines in your output.

I was on a HashiCorp call this past week and floated this problem past them. Would they add a flag to terraform to remove these refresh lines?

They suggested two things, one of which is brilliant. One idea was to filter all the terraform plan output through sed to remove the unwanted lines. That’d work, but I’d have to be careful with regex to make sure I don’t remove anything we’d want later — after all, there’s a large variability in terraform output from n number of pipeline runs as we scale to hundreds of them.

The second idea I loved and immediately implemented — for both the terraform “plan” and “apply” steps, put a step immediately preceding them to do the refresh, then use the existing terraform CLI flag that disables refresh on those steps.

This is brilliant because it changes exactly nothing about how terraform runs, outputs to a file, or logs anywhere (all data is still captured) but changes how the data is presented to the user, and significantly improves the UI. It’s a win all around.

If this was run from your local computer, the previous setup looks like this:

Terraform plan:

terraform plan

The new method which separates the output is:

terraform refresh
terraform plan -refresh=false

And the output, rather than having several hundred lines of “terraform refreshing (path)” you get the information you’re looking for right at the top. In this below photo with no changes, the entire output is only 21 lines, even if you’re managing hundreds or thousands of resources.

Here’s an example code to build the YAML pipeline in Azure DevOps, as well as some AWS VPC build code so you can set this up in your own environment. For more detailed instructions on how to deploy this and where you can see the improved shortened. Here’s the code:

KyMidd/AzureDevOps_TerraformPipeline_YML-SeparateRefresh
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com

Hope it helps. Good luck out there folks! 
kyler