🔥Let’s Do DevOps: Using GitHub Actions with Bash to Copy Files to Every Repo in an Org
🔥Let’s Do DevOps: Using GitHub Actions with Bash to Copy Files to Every Repo in an Org
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can…
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
I wrote a previous post about a requirement for the business I’m at to validate all our git commit messages look like Jira ticket numbers. We managed to implement that in a single repo using a GitHub Action that runs a bash script and uses regex matching. Which is so awesome! But it only integrates this check into a single repo.
Some CI tools let you set a global check for every repo, but GitHub isn’t one of them. Our enterprise support says this is in the works, but not implemented yet. Based on how GitHub has implemented every repo to operate individually, and even enterprise policies aren’t very mature yet, I imagine this is going to take a while. And we want to implement this now!
A solution advised by our GitHub enterprise team is to build it ourselves, and a link to a powershell-driven GitHub Action published by jhutchings1. If you’ve been reading this blog for a while, you know I’m not a huge fan of powershell — rather I love bash, and use it wherever I can, so I re-implemented the tool in bash, as well as added a lot of error catching and correction.
I just got it working, so it’s in a beta/mostly-stable state, however I want to share with you right away since you may have a similar need. Let’s go over the bash commands, and then embed the script in an Actions file, and talk about how we idempotently manage PR creation in GitHub.
GitHub Actions — Clone Metadata Fetch Depth
GitHub Actions operates in an efficient but also a bit opaquely. You’d imagine that an automation step called “checkout” on a repo tracked with git would do a “git checkout”. You’d be wrong.
| - uses: actions/checkout@v2 | |
| with: | |
| fetch-depth: 0 |
actions/checkout fetches the most recent version of files, but leaves the branch in a detached head state, and also doesn’t clone the git history. That’s a lot missing we’ll require that we want to read (to infer PR titles, and commits, and all sorts of cool stuff below).
So the first thing we do is update the checkout action to use a fetch-depth of 0, which means, “grab all the git history”.
We’re still not attached to a branch, but we can fix that below in our bash script.
Bash Scripting with GitHub CLI, Git
Set it Up
We start with setting the IFS, or in bash lingo, how to tell how to separate two strings as individual entries. This tells bash to read a newline as a separator, a good practice for reading data. It’s probably not required here, but I set it on most scripts because it’s bitten me before.
Then we do what I talked about above — we need to checkout the master branch (that’s the only place our script runs). Even though actions/checkout cloned the files and git metadata down, it doesn’t attach to a branch, but we’ll require that for our git commands below. Boom, now our git commands can work.
| # Set new line as separator for reading data in for loops | |
| IFS=$'\n' | |
| # Pull repo to get updated metadata | |
| git pull origin main | |
| git checkout main |
Then we set a couple of static vars. These are the name of our GitHub org and the “automation” repo where this code is stored and will run from.
| org=org_name_in_github | |
| automation_repo=name_of_repo_where_script_running_from |
CI/CDs are great, but they can be kind of dumb. Imagine we use a PR to update our template file, then gets approvals and merge it. This Action wouldn’t be aware of the PR or branch’s name that triggered the merge (maybe this is a GitHub-provided variable?). However, we want that info! We’re planning to open a PR in every repo in this GitHub Org, and having a descriptive branch and commit name would be pretty helpful, no?
So we use some git magic to find the last commit message used to merge into master/main branch, and then we replace any spaces with dashes to make sure it’s value. We store it in a variable for use later.
| # Pull commit message from merge commit into master | |
| commit_message_spaces=$(git log -n 1 --pretty="%b" | cut -d "/" -f 2-) | |
| commit_message="${commit_message_spaces// /-}" |
We’d also like to use the same branch name, right? So we look at our git metadata again to find the branch name that was used in our construction repo, and store it as a variable for use below.
| # Since this runs right after merge, use last commit message to construct new branch name | |
| branch=$(git log -n 1 --pretty="%B" | grep "Merge pull request" | cut -d "/" -f 2-) |
The PR body can be entirely static — that’s what I’ve done here, just filling in the Org and Construction Repo name. This will be the description of every PR we open in our entire Org.
| # PR body | |
| pr_body="Updating the PF commit checker due to a merge at this repo: https://github.com/$org/$automation_repo" |
The PR will be opened by my own use, because that’s where the security token was generated at. However, we can influence the git author info seen by our Dev teams when these PRs are opened. So we set this information with global git config commands. Remember, this is run on a temporary box, so who cares if we set these configs globally?
| # Program git defaults | |
| git config --global user.email "commit-checker-bot@yourCompany.com" | |
| git config --global user.name "TfGithubAutomation YACC Deploy Bot" |
Find All The Repos
Next we need to find all the repos in our org to iterate over. The GH CLI tooling is perfect for this, and can list out all the repos, then with a few cut commands, we have a list.
I left in my testing command — you can instead set that static variable for the iterative loop to iterate over only 1 repo for testing. That’s what I did (many, many times!) to make sure it all worked properly before rolling it out live. I recommend the same for you! :)
| # List all repos, cut for just repo name in org | |
| org_repos=$(gh repo list $org -L 1000 | cut -d "/" -f 2 | cut -f 1) | |
| #TEMP: testing with individual repo | |
| #org_repos=KylerTestRepo1 |
Iterate!
Now that we’ve set the information that our process will need, we start iterating over our repos. We print out some happy info messages and start up our machine.
| # Iterate over each repo | |
| for repo in $org_repos; do | |
| echo "********" | |
| echo "** Cloning repo" $repo "to update Action" |
We move to a temp repo — this is a must! At first I didn’t do this, and got confused when my git commands below were getting the wrong results. Which of course, is because it’s operating in the construction (host) repo, rather than the cloned repo I want it to read. Nesting git inside git is a bad, confusing proposition. Putting each in a temp repo is easier.
| # Create landing space to avoid name collission with this repo | |
| mkdir -p /tmp | |
| cd /tmp |
We’re doing a few cool things in this one-liner. First, we’re using git clone to grab the target repo (the one we’re pushing our Action to). We’re using the $GITHUB_TOKEN to read our cryptographic token, which is great for CI/CDs — no username/passwords for us.
Second, we’re storing all the stderr (standard error) to the stdout (standard out) using this bolt-on config 2>&1. Finally, we’re running this command silently and trapping all output in a variable. This is important — we’ll use several commands in a second to read it for error messages and respond if we find any.
| # Clone the repo | |
| gitCloneResults=$(git clone https://oauth2:$GITHUB_TOKEN@github.com/$org/$repo.git 2>&1) |
The first unusual situation to bite me was an empty repo. If the repo is empty, there are neither any files or any default branch. Which means our normal process of creating a new branch won’t work — we need to create an empty main branch, push to remote, then do our normal checkout method.
| # If cloned repo is empty, initialilze main branch | |
| initRepo= | |
| if echo "$gitCloneResults" | grep -q "You appear to have cloned an empty repository"; then | |
| echo "Empty repo detected, initializing main branch" | |
| initRepo=true | |
| cd $repo | |
| git init | |
| git checkout -b main | |
| git commit --allow-empty -m "SOP-22353 Initialize repo" | |
| # Attempt git push, check for archived to safely catch | |
| gitPushResults= | |
| gitPushResults=$(git push --force --set-upstream origin main 2>&1 || true) |
Then we do another push, and see if the error is resolved. It might not be — we catch the results in the same way, and if we get any issues, we address them as well as we can. Note that we’re doing an || true to catch any non-0 exit code. Interestingly, GitHub Actions exits at the first non-0 exit code (runs bash scripts with -e set by default), which is a brittle way to run a service. Therefore, you’ll see these || true scattered through my code, which means, or exit 0 if the first command didn’t, and permits me to read errors and attempt recovery, rather than exiting right away.
If the repo is archived (set to read-only), there’s nothing we can or should do, so we continue which is bash-speak means skip to the next iteration of the loop.
If the PR already exists with the exact same name, we just move on. This is unlikely to be seen outside of my testing. We could probably append a random string to the end, or date/time or something, but this isn’t well built-out because we probably don’t need it.
Lastly, if there is no caught error message, we consider this a success and move into the repo’s folder and git worktree.
| # Attempt git push, check for archived to safely catch | |
| gitPushResults= | |
| gitPushResults=$(git push --force --set-upstream origin main 2>&1 || true) | |
| # Check if failed to push due to archived read-only | |
| if echo "$gitPushResults" | grep -q "This repository was archived"; then | |
| echo "This repository is archived, and is read-only, skipping" | |
| # Cleanup | |
| cd ../ | |
| rm -rf $repo | |
| # Continue to next loop | |
| continue | |
| elif echo "$gitPushResults" | grep -q "already exists"; then | |
| echo "Pull request already exists with this name, skipping" | |
| # Cleanup | |
| cd ../ | |
| rm -rf $repo | |
| # Continue to next loop | |
| continue | |
| fi | |
| else | |
| # Repo cloned successfully | |
| cd $repo | |
| fi |
Idempotence is a hard thing to organize here — there are a LOT of moving pieces. I didn’t attempt to build a database or idempotence around PRs. Rather, I built it around whether the file had changed. The idea is that if the new template file that we checked in hasn’t changed at all, then there’s no need to go further, right? It’d be an empty PR anyway.
So we use md5sum to hash the old and new file, capture the outputs, and compare their hashes. If they match, the Actions file hasn’t changed, and we can continue to the next loop. If they don’t match, the Actions file is new to this repo (right at the beginning) or has changed, and we should attempt to open a PR!
I’m proud of this one — it solves a hard problem (idempotence) in a simple way.
| # Add idempotence circuit breaker by checking md5 of old vs new file | |
| newFileMd5=$(md5sum ../../actions-runner/_work/TfGithubAutomation/TfGithubAutomation/commit_checker/github_commit_checker.yml.template 2>/dev/null | cut -d " " -f 1) | |
| existingFileMd5=$(md5sum .github/workflows/_PfGitCommitChecker.yml 2>/dev/null | cut -d " " -f 1) | |
| # If md5 match, continue to next iteration of loop | |
| if [[ $newFileMd5 == $existingFileMd5 ]]; then | |
| echo "Actions file has not changed, continuing to next iteration of loop" | |
| continue | |
| else | |
| echo "Actions file is missing or has changed, attempting to open new PR" | |
| fi |
If we should attempt to open a PR, we first copy the template file into the cloned repo on our local host.
Then we checkout a new branch, add the file, and notch on a commit. If we’ve init’d this repo locally my magic git command has issues for a reason I don’t understand, so I took the easy way out and just set the base to master if we’re initializing it.
Else, we actually read the git tree and find out the HEAD branch. This is important, because there is a variety of default branches in our repos, so we can’t just assume they’re all called “master” or “main” or whatever.
| # Create workflows directory if not exist, copy source over existing | |
| mkdir -p .github/workflows | |
| cp ../../actions-runner/_work/$automation_repo/$automation_repo/commit_checker/github_commit_checker.yml.template .github/workflows/_GitCommitChecker.yml | |
| # Checkout new branch | |
| git checkout -b $branch | |
| # Add file, commit | |
| git add . | |
| git commit -m $commit_message | |
| # Identify base/main branch | |
| if [[ "$initRepo" == "true" ]]; then | |
| base_branch=master | |
| else | |
| base_branch=$(git symbolic-ref refs/remotes/origin/HEAD | cut -d "/" -f 4) | |
| fi |
This probably looks familiar — it’s the exact same push we did above. I probably should’ve created a function since it exactly matches, but I am a big fan of declarative, simple code, so I didn’t. Maybe one day I’ll see the light of abstracted, DRY code, but not yet! :)
| # Attempt push, open PR | |
| gitPushResults= | |
| gitPushResults=$(git push origin $branch --force 2>&1 || true) | |
| # Check if failed to push due to archived read-only | |
| if echo "$gitPushResults" | grep -q "This repository was archived"; then | |
| echo "This repository is archived, and is read-only, skipping" | |
| # Cleanup | |
| cd ../ | |
| rm -rf $repo | |
| # Continue to next loop | |
| continue | |
| elif echo "$gitPushResults" | grep -q "already exists"; then | |
| echo "Pull request already exists with this name, skipping" | |
| # Cleanup | |
| cd ../ | |
| rm -rf $repo | |
| # Continue to next loop | |
| continue |
If there are no errors pushing our branch, we assume it all worked, and we’re clear to open our PR. We reference some of those variables we set a long time ago, before our loop, and create a PR. If any failure is seen, we also || true here so our script continues and prints the error, rather than stopping and not opening the rest of the PRs.
We also clean up our cloned code. This runner is ephemeral, so who cares, right? My only concern is if we clone 100+ repos we fill the disk of our runner and our script fails, so we do the tidy thing, and sweep up after ourselves.
| else | |
| # Normal rules apply, creating PR | |
| gh pr create -b $pr_body -t $commit_message -B $base_branch || true | |
| fi | |
| # Cleanup files | |
| cd ../ | |
| rm -rf $repo |
Sleep
I bet you’re feeling about ready for a nap too if you’re following along with me, but we’re not ready for that yet. However, we do need to tell our script to sleep for a bit. It turns out GitHub doesn’t appreciate if you try and open 100 PRs in about a minute, and their API blocks your requests.
I tried with a 3 second sleep timer and they still didn’t permit every request, so I tried 5 seconds, and that appears to be the sweet spot.
We finish our iterative loop with done.
| # Sleep to avoid github API rate-limits | |
| echo "sleeping for 5 seconds" | |
| echo "********" | |
| sleep 5 | |
| done |
Actions Code
The final step is to drop all this code into a GitHub Actions file and repo. I’ve done that hard part for you. Here’s the code for you to copy and use!
GitHub - KyMidd/GitHubActionsBashDeployerEntireOrg
Contribute to KyMidd/GitHubActionsBashDeployerEntireOrg development by creating an account on GitHub.github.com
Summary
We wrote a bash-based tool that integrates heavily with the GH CLI, git, and is able to build n PRs to keep an Actions file up to date everywhere. It’s not as simple as it could be — this will mature as time goes on, but it works! Woot!
Thanks all! Good luck out there.
kyler