🔥Let’s Do DevOps: Set GitHub Repo Permissions on Hundreds of Repos using GitHub’s Rest API using a GitHub Action
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
I’m helping migrate a few teams from different places — TFS, Azure DevOps, Stash/Bitbucket, into GitHub. And one feature that all those services offer is setting the default permissions for all repos created in different organizations or projects. Stuff like, set the number of reviewers whom need to approve a PR before it can be merged, or set permissions for different teams to new repos.
GitHub interestingly doesn’t do that(!), and requests to our GitHub Enterprise team suggested we go and build it ourself. I’m surprised this is something I need to build myself — shouldn’t Enterprises and Orgs have trickle-down permissions and settings that can be automatically enforced?
But build it myself is exactly what I did. Using GitHub’s admittedly quite robust REST API, I built a GitHub Action that reads a CSV of repos and settings flags (more on this below), to send authenticated curls to GitHub’s REST APIs to set permissions. Then we read the output of the curl to make sure it worked properly.
This lets us standardize permissions for dozens or hundreds of repos in a few minutes, and to keep those permissions in check.
Which is pretty cool, huh? Let’s walk through the steps I took. And all code is linked in a public github repo at the end of this article if you want to skip ahead to the code and go build it yourself!
REST API Basics + cURL
REST (Representational State Transfer) APIs are standardized inputs for requests, usually using HTTP format, to send a message. Since we’re using HTTP, we’ll use GET http requests to gather data, POST to send data, and PATCH to modify a setting. You’ll also see DELETE in some instances.
Curl (which you’ll also see as cURL, the proper capitalization), is a linux-based tool that can send http requests. It’s available for linux, mac, and has even been released for Windows! The way curl specifies the type of request is with the -X flag. So if you want to send a PATCH request, you’d do the following. Note the -s argument also, which tells curl to silently send the request, and not report bytes send and downloaded (which we don’t care about, since request and response is likely quite small).
| curl -s -X PATCH \ #<-- Here we specify the type of http request to send | |
| (etc.) |
We can also set headers with the -H flag. There are two headers GitHub requires on most REST API requests — the Accept header and the Authorization header. The Accept header tells the server what type of response our client expects and can understand, and the Authorization header is how we’re choosing to authenticate to GitHub — in this example case, with a “Bearer” token, what GitHub’s platform calls a PAT (Personal Access Token).
This PAT is generated at your github developer settings page. Make sure it has the right permissions to set whatever things you want below. Remember if your enterprise requires MFA, you’ll need to “authorize” it for that Org once the PAT is created.
In curl lingo, sending the headers looks like this:
| curl -s -X PATCH \ | |
| -H "Accept: application/vnd.github+json" \ #<-- GitHub's REST API requires this for some requests | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ #<-- Auth to GitHub | |
| (etc.) |
We also need to tell curl where to send the request. The API endpoint for github is https://api.github.com/ and there are more URI paths to append depending on what type of information we’re setting. GitHub’s REST API docs are here, and show which path to specify. In this case we’re setting a repo-wide setting, so we send the request to api.github.com/repos/(org)/(repo).
| curl -s -X PATCH \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO \ | |
| (etc.) |
The final item to add is the data! REST APIs usually understand json since it can be minimized quite well. In the below example, we send a data field (-d for curl) to turn on the delete_branch_on_merge setting, as well as changing the type of repo to private . We can set a whole mess of settings if we’d like.
Keep an eye on the type of request we’re sending. POST implicitly over-writes other settings, while PATCH generally modifies one setting at a time.
| curl -s -X PATCH \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO \ | |
| -d '{"delete_branch_on_merge":true,"private":true}' #The data payload of settings to deliver |
In case you’re totally new to bash, the $NAME indicates a variable, so we’ll define a variable at the very start of a script, and then we can use it later on.
Writing the REST API CURLs, Automating Validation
So our first step is to write all the REST API calls we want to send for a specific repo. We don’t care about any looping or GitHub Actions or anything like that yet. Right now, we want to hard-code an Org, a Repo, a bearer token, and go to town to make sure our syntax is right. The curls we’ll send will look like this:
| curl -s -X PATCH \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer ghp_asdfasdfasdfasdfasdf" \ | |
| https://api.github.com/repos/my_github_org/repo_name1 \ | |
| -d '{"delete_branch_on_merge":true,"private":true}' |
When we send them, we will usually get a response. (Note that some requests don’t send back a response, like a -X DELETE, because if you don’t get an error you should assume it works.) But -X POST and -X PATCH usually give us a read-out of the settings for the API group we’re updating once the changes have been made, which is super helpful.
For instance, this request receives 151 lines back, showing all settings and their values. Let’s filter it for just allow_ for a few settings that are turned on or off.
| ~ curl -s -X PATCH \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO \ | |
| -d '{"delete_branch_on_merge":true,"private":true}' | jq | grep -E 'allow_' | |
| "allow_forking": false, | |
| "allow_squash_merge": true, | |
| "allow_merge_commit": true, | |
| "allow_rebase_merge": true, | |
| "allow_auto_merge": false, | |
| "allow_update_branch": true, |
Looks like for this repo, forking is not allowed, but squash-merging is. Neat!
We can also set permissions on a repo using the slug (the all-lowercase github ID) of a team that should get access to a repo. In our use case, that helps our CI (automation) user always get the rights they need. Setting “admin” permissions for the AutomationUsers team looks like this:
| curl -s \ | |
| -X PUT \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/orgs/$GH_ORG/teams/automationusers/repos/$GH_ORG/$GH_REPO \ | |
| -d '{"permission":"admin"}' |
Interestingly, sending this request doesn’t get any result back unless it fails. Keep track of what a “happy” result looks like, vs a “something went wrong”. As we are iterating through these REST calls, we can also write some github logic to trap the curl response and grep (search) against it. If we can teach our script how to tell if it worked or not, we can limit what is printed out on screen and better handle errors.
For example, setting the CI user access to “admin” above can trap the response in a variable like this:
| CURL=$(curl -s \ | |
| -X PUT \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/orgs/$GH_ORG/teams/ci/repos/$GH_ORG/$GH_REPO \ | |
| -d '{"permission":"admin"}' 2>&1) |
That means the $CURL variable now contains what GitHub sent back to us. In our case, that means nothing, right? A happy response means no response. If something failed, we’ll get an error message back. So let’s write a little bash if statement to read the result and count the lines. If we see any problem statements, like “Problems” or “Not Found” (errors I saw when sending incorrect team names), we report an error and print the $CURL response for future-us to dig into and diagnose.
However, if there’s no error message, we report that we successfully set the permission, yay! 💥
| if [[ $(echo "$CURL" | grep -E "Problems|Not Found") ]]; then | |
| echo "☠️ Something bad happened granting CI user access to the repo, please investigate response:" | |
| echo "$CURL" | |
| else | |
| echo "💥 Successfully granted CI user write access to repo $GH_REPO" | |
| fi |
Loop All The Repos
Okay, let’s assume you have now written all the REST calls for all the permissions you want to set. That’ll take a while, so do a good job!
We’ll also assume for $Org and $Repo, you used variables. If you didn’t, go back and update to it. Our next step is to loop over several (maybe a lot!) of repos, and set permissions for them too. Once done, let’s continue.
Bash has a few tools we can use to loop over values. A “for” loop can do this, and also a “while” loop. “While” has a few tricks that we can use though, like reading a CSV (Comma Separated Value) list, where it can also read some other attributes, like whether to turn on specific stuff. That is perfect for this, so let’s use a while loop.
But first we need to know what our data that we’ll read will look like. So let’s build our CSV first. Here’s an example:
On line 1, CSVs don’t normally support comments, but we’ll add logic for this in our while loop.
On line 2, bash has a hard time excluding data labels from a while loop also (come on bash!), but we’ll write some logic to exclude those also.
On line 3+, those are github repos, as well as flags for stuff we want to set (or not). Note that the last field can be left blank (indicated by the ,, on line 3, where line 4 and 5 set a value.
| # The COLLECTION_LEAD_TEAM_GITHUB_SLUG can be left blank, and permissions won't be assigned | |
| GH_ORG,GH_REPO,REQUIRED_APPROVERS_COUNT,COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| org_name,repo_name1,1,, | |
| org_name,repo_name2,1,1,teamname2 | |
| org_name,repo_name3,2,1,teamname3 |
So that’s our data. Let’s save that as “repo_info.csv”.
Let’s read it with bash with a while loop. Let’s start from the beginning — first we set the IFS, or Internal Field Separator, to a comma. That means to interpret a comma as a break between values. (Note that if you need to have a comma within a value this wouldn’t be a good solution!)
Then we set the -r (to ignore some file slashes and stuff) and then start listing what our data fields in the CSV should be mapped to variables in bash. So basically, in order, read the first value (in our CSV, this is GH_ORG, and set it to bash variable GH_ORG. And so on for the other fields. Keeping the same field names help keep the humans reading this later sane! :)
Within the context of the while loop, it’ll set those values from the CSV. Woot!
| while IFS="," read -r GH_ORG GH_REPO REQUIRED_APPROVERS_COUNT COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| do | |
| (loop stuff!) | |
| done < repo_info.csv |
There are a couple of control things we need to do first, though. Remember how CSVs don’t normally have comments? I love comments. I comment all the code. So we need some clever way that when our while loop stumbles onto our comment, to be able to understand what is going on and not crash, and not try to read the comment info as a repo and org and do stuff.
So let’s add an if statement, where we check the value of $GH_ORG, the first value we’re reading from the CSV. If it start with #, indicated by the simple regex ^\#, then we continue. Continue has a special meaning in a while loop. It means to skip the rest of the while loop and continue on to the next iteration. So basically, skip!
Awesome. Now our while look can safely skip over comments in our CSV.
| while IFS="," read -r GH_ORG GH_REPO REQUIRED_APPROVERS_COUNT COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| do | |
| # Ignore comment lines in the CSV | |
| if [[ $GH_ORG =~ ^\# ]]; then | |
| continue | |
| fi | |
| done < repo_info.csv |
And remember CSVs also can’t skip over headers in CSVs very easily. That’s annoying. So let’s do pretty much the same thing — if the first value we read from the CSV is the literal string GH_ORG, indicating we’re reading line 2, the one with the column headers, also continue.
Sweet, now we ignore our headers (and let the humans understand the CSV a bit better), win/win!
| while IFS="," read -r GH_ORG GH_REPO REQUIRED_APPROVERS_COUNT COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| do | |
| (etc.) | |
| # Ignore the headers line of the CSV | |
| if [[ $GH_ORG == "GH_ORG" ]]; then | |
| continue | |
| fi | |
| done < repo_info.csv |
We can also set some default values if they’re skipped in the CSV. For instance, if someone doesn’t set the REQUIRED_APPROVERS_COUNT, we obviously need some number to use for our REST calls, right? So let’s set it.
The -z in bash means “test if this variable is blank/null/not set”. If it’s not set (blank), let’s set it to 1 and continue.
| while IFS="," read -r GH_ORG GH_REPO REQUIRED_APPROVERS_COUNT COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| do | |
| (etc.) | |
| # If approvers count isn't set, default to 1 | |
| if [ -z "$REQUIRED_APPROVERS_COUNT" ]; then | |
| REQUIRED_APPROVERS_COUNT=1 | |
| fi | |
| done < repo_info.csv |
And since we’ll be looping over lots of repos, we should probably print which Repo we’re targeting now. That’s easy with variables:
| # Print out info | |
| echo "⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️ ⚙️" | |
| echo "Updating settings for $GH_ORG/$GH_REPO" | |
| echo "Setting required approvers count to: $REQUIRED_APPROVERS_COUNT" |
Let’s also do some data validation — is the leads team (if specified), a valid one? First we check if the COLLECTION_LEAD_TEAM_GITHUB_SLUG variable is blank/not set. If it is, we skip all this logic (can’t validate if blank is valid, right?) and continue.
If it is valid, we send a GET request (when curl doesn’t see an -X to specify a type, it sends a GET) to github to get that team’s information. If we see a response with Problems|Not Found in the response, we print an error and unset this variable so our downstream REST calls don’t fail. If it doesn’t fail, that team slug must be right, yay! With that validation in hand, we continue to setting real permissions.
| # If COLLECTION_LEAD_TEAM_GITHUB_SLUG blank, skip it | |
| if [ -z "$COLLECTION_LEAD_TEAM_GITHUB_SLUG" ]; then | |
| # No collection lead team specified, will skip this step | |
| echo "COLLECTION_LEAD_TEAM_GITHUB_SLUG is blank, not setting that permission" | |
| else | |
| # Collection lead team specified, check if exists | |
| unset CURL | |
| CURL=$(curl \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/orgs/$GH_ORG/teams/$COLLECTION_LEAD_TEAM_GITHUB_SLUG 2>&1) | |
| if [[ $(echo "$CURL" | grep -E "Problems|Not Found") ]]; then | |
| # Team doesn't exist or slug is wrong | |
| echo "☠️ The collection lead slug isn't correct, that team doesn't exist. Continuing, but won't set this permission. Error message:" | |
| echo "$CURL" | |
| # Unset this value so we don't attempt to set this permission below, as it will fail | |
| unset COLLECTION_LEAD_TEAM_GITHUB_SLUG | |
| else | |
| # Leads team specified and does exist | |
| echo "Setting the collection lead team permissions for team: $COLLECTION_LEAD_TEAM_GITHUB_SLUG" | |
| fi | |
| fi |
Next we start sending our REST calls. First we unset the CURL variable — we’ll use that variable each time, and don’t want to risk it being set to what the last call sent us. Then we send our REST call to GitHub. In this case, we’re enabling GitHub Actions in the repo, and setting the “allowed_actions” to the “selected” ones.
We read the response with our if statement, and if the line count (wc -l ) is less than or equal to 1, it worked, yay! If it’s greater than 1, it’s printing some error message, and we note it and continue.
| # Enable github actions on repo | |
| unset CURL | |
| CURL=$(curl -s \ | |
| -X PUT \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO/actions/permissions \ | |
| -d '{"enabled":true,"allowed_actions":"selected"}') | |
| if [[ $(echo "$CURL" | wc -l) -le 1 ]]; then | |
| echo "💥 Successfully set Actions enabled" | |
| else | |
| echo "☠️ Something bad happened setting actions enable, please investigate response:" | |
| echo "$CURL" | |
| fi |
Note that we don’t bail if there’s an error message. There’s no exit commands anywhere. We’re potentially looping over hundreds of repos, so we don’t want to exit our script on the first error. Rather, we note it in the log and continue.
Do that for all your REST calls to set all the permissions. That can be a LOT! But if you get this all right, think of all the time you’ll save clicking the permissions buttons in GitHub. Instead, you’ll have your feet up with a margarita. That’s worth it, I promise.
The branch protections are some of the most useful ones to set — protect your deployment branches to protect your environment. That’s huge. But we can’t set protections on a branch that doesn’t exist, right? So we need to confirm if the branch exists. For instance, if you want to check if the “master” branch exists for this repo, we can do this:
| # Get branches | |
| curl -s \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO/branches 2>&1 > repo_branches | |
| if [[ $(cat repo_branches | grep -E "\"name\"\: \"master\"") ]]; then | |
| MASTER_EXISTS=true | |
| fi |
And if the branch exists, we can set the branch protections for it. This JSON payload is massive, and we directly encode the number of approvers into it. Note that in bash, the ' characters mean to not interpret variables. We DO want to interpret variables, so we have to wrap the whole data payload with " instead, and escape all the " with \" which is very annoying, but required by bash to know where the data payload ends for real.
| if [[ "$MASTER_EXISTS" = true ]]; then | |
| unset BRANCH | |
| BRANCH=master | |
| unset CURL | |
| CURL=$(curl -s \ | |
| -X PUT \ | |
| -H "Accept: application/vnd.github+json" \ | |
| -H "Authorization: Bearer $GITHUB_TOKEN" \ | |
| https://api.github.com/repos/$GH_ORG/$GH_REPO/branches/$BRANCH/protection \ | |
| -d "{\"required_status_checks\":{\"strict\":true,\"checks\":[{\"context\":\"name_of_status_check\"}]},\"restrictions\":{\"users\":[],\"teams\":[],\"apps\":[]},\"required_signatures\":false,\"required_pull_request_reviews\":{\"dismiss_stale_reviews\":true,\"require_code_owner_reviews\":true,\"required_approving_review_count\":$REQUIRED_APPROVERS_COUNT,\"require_last_push_approval\":true,\"bypass_pull_request_allowances\":{\"users\":[],\"teams\":[],\"apps\":[]}},\"enforce_admins\":true,\"required_linear_history\":false,\"allow_force_pushes\":false,\"allow_deletions\":false,\"block_creations\":false,\"required_conversation_resolution\":true,\"lock_branch\":false,\"allow_fork_syncing\":false}" 2>&1) | |
| if [[ $(echo "$CURL" | grep -E "Problems|Not Found") ]]; then | |
| echo "☠️ Something bad happened setting branch protections on $BRANCH, please investigate response:" | |
| echo "$CURL" | |
| else | |
| echo "💥 Successfully set branch protections on $BRANCH" | |
| fi | |
| fi |
GitHub Action-ify It!
Putting this whole mess into a GitHub Action is actually super easy. We create Actions file set_repo_permissions.yml at .github/workflows path in our github repo.
Note the ${{ secrets.GITHUB_TOKEN }} near the bottom — that’s a repo secret that stores the github PAT that has permissions to make these changes. Just put the same value you’re using for your testing.
| name: Set Repo Permissions | |
| on: | |
| # Run automatically when master updated | |
| push: | |
| branches: | |
| - master | |
| # Permit manual trigger | |
| workflow_dispatch: | |
| jobs: | |
| set_repo_permissions: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - name: Set Repo Permissions | |
| env: | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| run: | | |
| ./repoUpdateScript.sh |
Note that it’ll run the ./repoUpdateScript.sh script at the root of your repo, and that script needs the repo_update.csv file at the root of your repo, so make sure you commit both of those to your repo so it can find them!
The Action can be run either by a commit to master branch, or can be run manually from the Actions page. Running the Action manually looks like this:
Regardless of how it’s run, the Action output looks like this:
Summary
In this blog we went over how to use the linux tool curl to send authenticated REST API messages to GitHub in order to set permissions for GitHub Repos. We converted those calls to use variables so we can iterate over lots of repos, then built a CSV file to contain our repo names as well as some salient settings we might want to change.
Then we wrote a bash while loop to read the CSV file, and send all the right authenticated requests to github. Then we put it all in a GitHub Action so others can easily do PRs against the CSV or even run the Action manually in order to set the permissions they need in our Organization.
All code can be found here:
GitHub - KyMidd/GHAction_REST_SetRepoPermissionsIterator
This repo and automation in it are designed to set all GitHub repo permissions required for a new repo. To use it…github.com
Go forth and build cool stuff! Good luck out there ❤ :)
kyler