This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
All code for you to build this new pattern is provided as a GitHub link at the bottom of this article ;)
Hey all!
Building S3 policies is hard. Thereâs lots of stanzas needed for even simple use cases, and the potential for doing something wrong and exposing data to the wrong partner, or even to the internet, is ever-present.
Thatâs pretty scary for folks in any industry, but for regulated industries like finance and healthcare, itâs can be a Lawsuit Generating Event! đŹ
Which means most companies have two choicesââârequire that only senior engineers touch these policies due to risk, or take the risk that a junior engineer wonât accidentally assign the wrong policy.
However, I have a new way to shareâââusing Terraform in an iterative constructor pattern to build the policies for you, by passing Terraform only the tiniest amount of information.
Hereâs what youâd pass your iterator to build out per-partner access on an S3 bucket with a home folder and automatic data and versioning clean-up. Thatâs pretty cool and easy, isnât it? And the names and ARNs for the partner user could even be strings to make life even simpler!
Letâs first talk about how youâd do this by hand, writing all the IAM policies yourself, before we go over how to automate it.
Doing it By Hand
S3 is an AWS service thatâs used for storing and sharing files. Access to the data plane (files) and the control plane (to make changes) are both controlled via IAM policies, which are long JSON documents.
Our goal with this policy is to:
Permit a specific AWS IAM User ARN to access a bucket
Permit the user to access only the target folder, no other usersâ folders
To do that, we have to grant a few permissions:
s3:ListBucket on the root directory of the bucketâââThis permits the partner to connect to the S3 bucket directly and view the folder weâll permit them to access, rather than remembering their folder name to connect directly to it
s3:GetObject and s3:DeleteObject recursively on the partnerâs folder path, usually the partnerâs name (but not always)âââThis permits the partner to access and delete objects in their bucket folder path recursively
s3:PutObject on the partnerâs folder path with a conditional to assign bucket owner full controlâââThis permits the partner to put files into the bucket, and the conditional tells the S3 service to change ownership to the bucket owner (us!)
Deny s3:ListBucket access to the bucket and folder pathsâââThis blocks seeing any other folder paths in the bucket, so our partners arenât even aware there are other clients (or what their names are!)
Letâs look at what each policy looks like. As we iterate through these, keep in mind what level of engineer youâd be comfortable with adding an entirely new partnerâââentry level, medium level, senior.
s3:ListBucket on the root directory of the bucket
First, letâs let our partners access the bucket and list their folders only. If we are copying and pasting a working stanza from another client, we have to make sure to update:
The AWS principal, which is the partnerâs IAM User ARN
The folder path the client is permitted to access
If you miss either of these, you risk a partner being able to read the file names of a different partner, which might leak data.
Hereâs what the stanza looks like:
s3:GetObject and s3:DeleteObject recursively on the partnerâs folder path
Next, we need to permit the partner to GetObject (download) and DeleteObject (delete) files and folders recursively in their folder path only.
Just like the above access, if we are copying and pasting a working stanza from another client, we have to make sure to update:
The AWS principal, which is the partnerâs IAM User ARN
The folder path the client is permitted to access
If you miss either of these, you risk a partner being able to read and delete someone elseâs leaking data <âââScary!
Hereâs what that stanza looks like:
s3:PutObject on the partnerâs folder path with a conditional
Next, we want to permit our partner to upload files to only the folder path assigned to them, recursively. Note the conditional here. Usually, conditionals control access, but in this case, the S3 service is mis-using it to instruct the S3 service to update the ownership permissions of any files uploaded to change the ownership to the S3 owner.
How interesting is it that S3 team is mis-using IAM in this way? Super cool
Just like the above access, if we are copying and pasting a working stanza from another client, we have to make sure to update:
The AWS principal, which is the partnerâs IAM User ARN
The folder path the client is permitted to access
If you miss either of these, you risk a partner being able to data to someone elseâs folder path, which could be confusing or lead to a data leak incident where partners share data with one another by accident.
Hereâs what that stanza looks like:
Deny s3:ListBucket access to the bucket and folder paths
Next, we want to block any partner from being able to read the names of other partnersâ folders. This likely isnât neededâââweâre only granting access above to their own folder, but this is a nice security back-stop in case access is accidentally granted.
IAM denies take precedence, so if a deny matches, the access will be blocked
Hereâs what that stanza looks like. Note the 2 conditionals, and the folder and ARN names that need to be updated.
WOW
So. Thatâs a lot of stanzas that arenât terribly obvious, right? Especially the conditions, and thereâs even one condition that doesnât work as a condition.
Unless you have experience with S3, you wonât be able to write this from scratch, and unless youâre very careful youâll miss a partner ARN and folder path pairing when youâre copying and pasting for a new client.
Clearly, this is a risk for your business. It would be so very much better if we could construct these policies, rather than requiring our engineers to write them by hand.
Enter: Terraform!
Terraform is an Infra as Code tool that lets us use declarative configurations to build resources, including IAM resource policies. This is amazing since no one writes JSON on purpose.
Terraformâs AWS provider has an aws_iam_policy_document resource that permits us to build a stanza of IAM policy. We can specify the specific permissions, and ingest the user ARN as a variable.
We can pair that with the for_each argument to build out a few stanzas, say, one for each partner.
The following terraform looks scary at first compared to the IAM, but keep in mind that youâd write this exactly one time. Every future update to this configuration would be to send another variable pair (Partner User ARN + home directory name) only. Thereâs no more writing IAM once this is implemented.
Each of the blocks looks very similar to the above, so letâs just look at one. Again, all the code is linked at the bottom of the article, if youâd like to copy and paste this module in its entiretyâââheck yeah.
Note the try() for the s3:prefix condition. Thatâs a lookup in case the partnerâs folder path doesnât match the partnerâs name. This permits us to write tweaked policies if someone needs to access a folder path with a slightly incorrect name. Think PartnerA needs to access folder path partner_a or similar. We can do that easily, woot.
Note that this is a âdataâ resource which means it wonât do anything in your environment. Itâs only a method of organizing data within Terraform itself. Weâll need to apply this configuration somehow in order to apply it.
Super-Policy!
So weâve built out n stanzas above for each clientâââit could be A LOT. We now have a few dozen stanzas we need to combine. Thankfully, Terraform and the AWS provider have some very cool tricks to help us.
First, we use the same aws_iam_policy_document resource and provide it a source_policy_documents argument. That is built to take lots of JSON IAM policies and combine them into a single valid policy.
However, each of our policies is built a bunch of timesâââwe have lots of clients, so we build out the s3_allow_list policy a lot of times. So we use the for loop construction in terraform. What that means is we wrap our for loop in square brackets, to structure it as an list (or array), and we tell it to iterate over each instance of the data block and print out the .json attribute of each. We run that for every single IAM stanza data meta-resource.
We also have one other trick that might make this module more resilient, try(). Try says to attempt to do what we ask, and if it fails, pass a null instead. So if we decide to turn off any of these access permissions, we wouldnât need to update the combined meta-resource. It would try to do that, fail, and instead pass a null, which the flatten() would entirely get rid of.
Writing resilient terraform modules is a great business best practice, and also guarantees you can post it and then never look at it again. Look at you, you superstar.
Use the Module!
So now we have an amazing module thatâs prepared to iterate over all our partners, each partnerâs IAM User ARN, and potentially the updated folder path.
I chose a map of maps as a data storage methodâââwe have each partnerâs name as the map key, Partner1. We assume this is the folder name also, but permit over-writing it with the folder_name argument (not shown in this snapshot).
Then we pass in the principals list argument. This is a list of ARNs in case a partner has multiple IAM roles that need to access the folder. After all, some partners want to have different roles do different things. The module fully supports a list of partners and will expand our IAM policy accordingly.
Summary
And thatâs it! We only send the map of maps to the module, and we donât have to write ANY IAM POLICY at all, which is incredible. You as an engineer or PR reviewer have way more time, get better sleep, and your business is protected from leaking data with a mis-type on these long, confusing policies.
And thatâs pretty amazing.
All Terraform code used to construct these IAM bucket policies and bucket lifecycle configurations can be found in GitHub here:
GitHub - KyMidd/CloudSecNext2022-TfIamConstruction: Don't build IAM by hand, use TF for thatâŠ
This repo is created to pair with CloudSecNext 2022 presentation titled "Zero Trust: Building IAM with Terraform" AWSâŠgithub.com
And keep an eye out for this talk at CloudSecNext2022, where Iâll be presenting to the SANS audience for CPE credits :)
Good luck out there!Â
kyler









