This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

Hey all!

This article is a continuation of what we built last time — namely, an event-driven GitHub Action, utilizing a custom GitHub App to send a webhook to an API gateway, which triggers a python3 lambda, which trigger a GitHub Action. We built all those resources last time in Part 1. If you haven’t read part 1, you should:

Now we’re going to look at the python3 code in the lambda, as well as the changes needed in the GitHub Action to take a repo name as input for a REST call.

If you just care about the code, scroll to the end of this write-up — a github repo is linked that contains all the terraform to build all the resources, as well as the python3 lambda.

First, let’s talk Lambda.

Lambda: Context

Before we look at the lambda code, we need to talk about the context — how is Lambda being launched, what does the json package in the webhook that makes its way to Lambda look like?

Let’s first go to our GitHub App and look at what the json payload looks like before we send it over. Go to your Org Settings → Third-Party Access → GitHub Apps → (Your app name) Configure → App Settings → Advanced. This very awesome page shows all the webhooks generated by the events the GitHub App is listening to and what their json payload looks like.

You’ll see one on this window that has a green check mark, which means it received a successful html code back from our lambda — we’ll build that soon. If you’re following along, yours likely all have the red exclamation mark, because your lambda isn’t in ship shape yet. We’ll get there soon!

Let’s expand one of the repository.created events and look around. First of all on the top, we see the X-Hub-Signature which is a sha-1 hash of the json payload body. That’s there because we setup a password on the webhook, and it’s critically important. That hash is seeded with the password we set, which means only GitHub should be able to create a payload which passes muster — this is an important security layer to prevent a random Bad Guy on the internet from sending us crafted json to make our lambda do stuff.

The bottom half of the screen is the json-encoded payload of the webhook. First of all, the action is a top-level key which tells us this is a repo created event. Remember, we’re now subscribed to a whole bunch of repo-related actions. Our Github Action only cares about when new repos are created, so this is something we can use to filter.

And lastly, the webhook has the name of the new repo, which is apparently KylerDynamicActionsTesting. We need to pass this repo name to our GitHub Action if we want it to configure that one repo.

There’s one wrinkle here — we’re not sending this payload directly to the lambda — we’re sending it to an AWS API gateway. And why cares, right? It’s in proxy+ mode, it’ll just pass the payload through! Well, kind of. It actually does something weird that we’ll have to compensate for — it encodes the json body of the webhook into a new arbitrary data format.

The json RFC requires double quotes for all keys, which means:

This payload isn’t even valid json anymore!

That beautiful, valid json webhook is now embedded in a non-valid-json kinda-sorta-json payload under a key called body.

So our first task — go fetch the webhook body.

Lambda — Let’s Code!

We’ll bounce around between the main function, called lambda_handler. Remember from Part 1, we told our lambda handler that when starting our lambda, to execute the lambda_handler function. It’ll call all our subordinate functions, which we’ll walk through together.

Python3 Lambda — Get our Webhook Back

The payload that started our lambda is handed off to our lambda as an object called event. That’s not configurable — that’s how lambda works. The context object is also handed off, which contains all sorts of stuff around the IP that triggered us, and other operational stuff we don’t care about (and will ignore). Let’s first decode the non-json json back to our pretty webhook json. We sent the event object to isolate_event_body function, and receive back our webhook body.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	def lambda_handler(event, context):

	print("🚀 Lambda execution starting")

	# Isolate the event body from the event package
	body = isolate_event_body(event)

view raw lambda_handler_isolate_body.py hosted with ❤ by GitHub

You’d think this would be easy — and with valid json, it would be. But since this isn’t, we’ll do some hacky stuff.

We receive the event from main, and first grab the event as a string using json.dumps function (line 4). Then we convert that string into a python dictionary (line 5). This type coercion means Python will enable us to lookup data based a key/value pair model.

Which is exactly what we do, on line 8 — lookup the event body key from the invalid json event object. Then on line 9, we load the event_body using json.loads into a valid json object, called body. Then we return it, on line 12.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Isolate the event body from the event package
	def isolate_event_body(event):
	# Dump the event to a string, then load it as a dict
	event_string = json.dumps(event, indent=2)
	event_dict = json.loads(event_string)

	# Isolate the event body from event package
	event_body = event_dict['body']
	body = json.loads(event_body)

	# Return the event
	return body

view raw isolate_event_body.py hosted with ❤ by GitHub

Ahh, that fresh taste of valid json.

Python3 Lambda — Webhook Secret from Secrets Manager

Remember how the header contains a SHA hash of the json body? And how that hash is seeded with a password we set on the GitHub side? We want to validate that hash by doing some hashing of our own on this side. And the first step to that is to go grab the password that is used for hash seeding.

Gotcha: Remember this password must be the same in Secrets Manager and in GitHub, or the hash will never match, even if everything else goes right.

Let’s call get_secret with the name of our secret, GitHubWebhookSecret and since Secrets Manager is regional, where to look for the secret, us-east-1.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Main function
	def lambda_handler(event, context):
	(removed)

	# Fetch the webhook secret from secrets manager
	GITHUB_SECRET = get_secret("GitHubWebhookSecret", "us-east-1")

view raw main_get_webhook_secret.py hosted with ❤ by GitHub

First we accept the arguments as secret_name and region_name (line 2), then we start up a boto3 session (line 5). Boto3 is an AWS SDK for python, which means it has all sorts of goodies that make our lives easier.

We establish a client session pointed at secrets manager (line 6), and then try to get the secret on line 12. Try lets us catch a failure, and rather than terminating the script, actually print out error messages, which is what we do on line 15–19. If there’s any ClientError (like missing permissions, secret doesn’t exist, etc.), print it out in the logs. Super useful stuff, particuarly as you dial in your IAM permissions.

Then on line 22, we call get_secret_value_response, which is a monster function that we happily don’t have to write — it looks up the KMS key used to encrypt the secret, then fetches it, then decrypts the secret with it, then returns the value.

We print a happy little dance (line 25), then return the decrypted secret, line 28.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Get GitHubPAT secret from AWS Secrets Manager that we'll use to start the githubcop workflow
	def get_secret(secret_name, region_name):

	# Create a Secrets Manager client
	session = boto3.session.Session()
	client = session.client(
	service_name='secretsmanager',
	region_name=region_name
	)

	try:
	get_secret_value_response = client.get_secret_value(
	SecretId=secret_name
	)
	except ClientError as e:
	# For a list of exceptions thrown, see
	# https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
	print("Had an error attempting to get secret from AWS Secrets Manager:", e)
	raise e

	# Decrypts secret using the associated KMS key.
	secret = get_secret_value_response['SecretString']

	# Print happy joy joy
	print("🚀 Successfully got secret", secret_name, "from AWS Secrets Manager")

	# Return the secret
	return secret

view raw get_secret.py hosted with ❤ by GitHub

Python3 Lambda — Validate Header SHA Hash

Now we have everything we need to do some cryptography! We’ll hash the body we received, then compare it with the hash GitHub generated on their side. If they match, no one has messed with our payload between them and us.

So let’s call the validate_signature function, and pass it our GITHUB_SECRET (the password that both GitHub and us used to hash the body), the whole event object (the not-json json), and the body, the decoded webhook payload.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Main function
	def lambda_handler(event, context):
	(removed)

	# Validate the signature
	validate_signature(GITHUB_SECRET, event, body)

view raw main_validate_signature.py hosted with ❤ by GitHub

We receive those arguments and keep the same names for consistency (line 2). First we delive into the body object — it doesn’t have our webhook json data, but it does have the http head with the SHA GitHub computed for the body.

First we identify the GitHub signature line using regex (line 4), and then grab the SHA value out of the payload and strip all the quotes out with unquote. Sweet, now we have a string called incoming_payload that’s what GitHub computed for a seeded hash.

Next we want to calculate our own hash — we call a function called calculate_signature that I’ll talk about next. For now, let’s assume it worked, and calculated the has based on the json body we received, line 6.

We then compare the GitHub computed hash incoming_signature vs the hash we computed calculated_signature. If they aren’t the same, something’s gone wrong, and we exit. If they are the same, WOOT, we validated our header, we continue. We don’t actually pass anything back to our main function — if we didn’t exit, that’s enough assurance we should continue.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Validate the signature
	def validate_signature(GITHUB_SECRET, event, body):

	incoming_signature = re.sub(r'^sha1=', '', event['headers']['X-Hub-Signature'])
	incoming_payload = unquote(re.sub(r'^payload=', '', event['body']))
	calculated_signature = calculate_signature(GITHUB_SECRET, incoming_payload.encode('utf-8'))

	if incoming_signature != calculated_signature:
	print("Unauthorized attempt")
	sys.exit()
	else:
	print("🚀 Confirmed HMAC matches, authorized access, continuing")

view raw validate_signature.py hosted with ❤ by GitHub

At line 6, we called a function called calculate_signature that computed a SHA1 hash and returned it. Let’s go through that — it’s super simple (as long as you speak cryptography). On line 3, we convert the github_signature, which is the secret used to seed the hash, to bytes using utf-8 encoding, on line 3.

Then we generate a digest hash using the hmac tool — it passes in our secret key (signature_bytes), the json webhook body (github_payload), and specifies which hash type to compute (sha1), on line 4.

Then we convert that digest into a hex value (how hashes are normally shown), and send it back to the parent function. And that’s how you hash a hash!

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Calculate the signature
	def calculate_signature(github_signature, githhub_payload):
	signature_bytes = bytes(github_signature, 'utf-8')
	digest = hmac.new(key=signature_bytes, msg=githhub_payload, digestmod=hashlib.sha1)
	signature = digest.hexdigest()
	return signature

view raw calculate_signature.py hosted with ❤ by GitHub

Python3 Lambda — Check Webhook Action

Now that we’ve validated the body is actually from github and we’re not being man-in-the-middled (yay!), let’s go get the name of the repo that was created. Let’s call check_event_action and pass it the decoded json webhook body object (line 6).

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Main function
	def lambda_handler(event, context):
	(removed)
	# Check event action. If new repo added, get repo name. Else, exit
	repo_name = check_event_action(body)

view raw lambda_handler_repo_name.py hosted with ❤ by GitHub

First of all, we look into the valid json webhook for the action top-level key. For the webhook we looked at above, it was created, which means a new repo was built. But remember, we’ll get lots of webhook from GitHub — many of them for stuff that we don’t care about. So we look at that value, and store it as action (line 5).

Notably, this fails if the json is an incompatible structure — like if it doesn’t contain a key called action. This should probably be inside a try block so we could elegantly catch a failure, instead of exiting, but I don’t care right now — if that key isn’t there, we don’t want to continue regardless, so ¯\_(ツ)_/¯

On line 8, we check if the action string is 'created'. If not, we print out what the action is, then we do something cool. We define a return map object, with a status code of 200. This return object is special — it’s returned from the lambda to the API gateway, and passed all the way back to GitHub. Remember the red exclamation mark and the green check within GitHub’s Apps UI? That’s this! We even define a json body, that’ll be printed out in the UI, which is frankly, just super awesome.

Then we exit, line 19. But hopefully, it is actually created, and we can continue. If yes, we print a happy message on line 21, then lookup the repo_name on line 24, print another happy message with the repo name, and return our repo_name string back to main.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Get the event action
	def check_event_action(event):

	# Check action of event
	action = event['action']

	# If action isn't "created", exit
	if action != 'created':
	print("🚫 Event action detected as: " + action)
	print("🚫 Event is not creating a repo, exiting")

	# Return 200 code
	return {
	'statusCode': 200,
	'body': json.dumps("Since action is ", action, ", and not 'created', we are exiting")
	}

	# Exit script
	sys.exit()
	else:
	print("🚀 Successfully detected action: " + action)

	# Get repo name
	repo_name = event['repository']['name']
	print("🚀 Successfully detected repo: " + repo_name)
	return repo_name

view raw check_event_action.py hosted with ❤ by GitHub

Python3 Lambda — Start Our GitHub Action!

Now we’re getting to the good stuff. We’ve validated the json, we’ve validated this is a new repo event. It’s time to poke our GitHub Action, and tell it to get to work!

Well, almost. We need to fetch one more secret — a GitHub PAT, a token which authenticates us to GitHub and lets us do authenticated stuff, like start Actions and such. On line 5, we do it. We already went over the get_secret function (and frankly, I’d rather get to the cool Action stuff anyway), so let’s skip it.

On line 8, we call start_repo_cop_targeted, and pass it the repo_name (the new repo), and the PAT token.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Main function
	def lambda_handler(event, context):
	(removed)
	# Get the PAT from secrets manager
	PAT = get_secret("GitHubAccessToken", "us-east-1")

	# Start the githubcop workflow
	start_repo_cop_targeted(repo_name, PAT)

view raw lambda_handler_start_repo_cop_targeted.py hosted with ❤ by GitHub

This function is a little huge, so let’s split it into two parts.

First, we need to create our payload map for sending to GitHub Actions REST endpoint. It requires two arguments — the ref (the branch name to execute against), and the inputs map, which is custom. For my GitHub Action, it accepts an input called repo-to-police, which is something I added as part of this project. It’s the valid of the input in the Actions yaml file — we’ll go over that next. We inject the repo_name argument there on line 8, and then print a happy debug message on line 11.

Then we create our POST headers map to prep for the REST call to GitHub’s API, line 13. It’s a json payload (line 14), we’re authenticating using a PAT bearer token (line 15), and we’re using a particular API version, which I copied from the REST page for this call. Then we print a happy message again, line 18.

Some of these “happy debug messages” are things I used to see how far the lambda would get before failing, but I like them now, so I left them in.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Start repo cop workflow targeting a single repo
	def start_repo_cop_targeted(repo_name, PAT):

	# Define new data to create
	payload = {
	"ref": "master",
	"inputs": {
	"repo-to-police": repo_name
	}
	}
	print("🚀 Successfully created payload")

	post_headers = {
	"Accept": "application/vnd.github+json",
	"Authorization": "Bearer " + PAT,
	"X-GitHub-Api-Version": "2022-11-28"
	}
	print("🚀 Successfully created post headers")

view raw start_repo_cop_targeted1.py hosted with ❤ by GitHub

Okay, continuing with that same function. We prep the url_post string, which is where we’ll send the REST call. We want to start an Action in a particular Org org_name, Repo repo_name, with a particular name. You can specify the Action with a UUID called a node_value or with the filename of the Action. That seemed easier than looking up a UUID, so that’s what I did.

Gotcha: Fill in the {stuff_names} with your own values, these are just placeholders. You can remove the {}.

Then we try to send the POST (line 8), by including all the stuff we’ve put together. If there’s any error, we raise it on line 14.

And on line 16, we check the status_code response of our POST. If it’s not 204 (happy http code), something’s gone wrong, and we print the error message and exit. The most likely issue here is you put the wrong org name, repo name, or workflow name. Or your PAT is wrong or doesn’t have the right permissions to start the Action.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	def start_repo_cop_targeted(repo_name, PAT):
	(part 2)
	# The API endpoint to communicate with
	url_post = "https://api.github.com/repos/{org_name}/{repo_name}/actions/workflows/{yaml_actions_file_name}.yml/dispatches"

	# A POST request to tthe API
	try:
	post_response = requests.post(
	url_post,
	headers=post_headers,
	json=payload
	)
	except ClientError as e:
	raise e

	if post_response.status_code != 204:
	print("🚨 Error: ", post_response.status_code)
	print(post_response.text)
	sys.exit()

view raw start_repo_cop_targeted2.py hosted with ❤ by GitHub

Python3 Lambda —Return a Happy Little Response Header

And we’re done! well, mostly done. Let’s print out a happy little debug message (line 5) and then return the response headers indicating success — that function contains almost no code.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	# Main function
	def lambda_handler(event, context):
	(removed)
	# Print happy joy joy
	print("🚀 Successfully started repo cop workflow targeting repo: " + repo_name)

	# Return response code
	return return_response_code()

view raw lambda_handler_return_response_code.py hosted with ❤ by GitHub

The only thing we do here is return an http header map with a status of 200 (happy http), and send a body message back that says everything went great.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	def return_response_code():
	return {
	'statusCode': 200,
	'body': json.dumps('Processed by GitHubCop Trigger Lambda!')
	}

view raw return_response_code.py hosted with ❤ by GitHub

The GitHub Apps UI even rewards us with the happy green checkmark, that it got the http 200 code back, and if you scroll down, you’ll see our message — Processed by GitHubCop Trigger Lambda!. Which is like, so cool.

GitHub Action — Accept Inputs

Okay, this is getting long, so let’s breeze over the changes I made to my GitHub Action to accept an input of a single repo.

First, the Actions file is updated to accept an optional (required: false, line 7) input called repo-to-police (remember how we set that in the json body of our Lambda REST call to start this action?).

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	on:
	(removed)
	workflow_dispatch:
	inputs:
	repo-to-police:
	description: 'The name of the repo to police. If not set, all repos will be policed'
	required: false
	type: string

view raw github_cop_input.yml hosted with ❤ by GitHub

It even looks cool from the GitHub web page UI, when starting the Action by hand — you can type in the name of the specific Repo to police, and it works just the same way as if started from an API call.

In the jobs section, I have some branching logic — if the input repo-to-police is blank (line 10), we run the script that gets ALL the repos (thousands). If it’s not blank (line 16), we echo the single repo to all_repos file — that’s the file the ./repoCop_getRepos.sh populates to pass to the next job in the Action.

Then on line 22, we upload the repos list as an artifact. We select the file using the path attribute — usually this would use wildcards, but we literally want to cache only the single file. And we use the github.run_id attribute to has the name of the artifact. That’ll make sure the artifact for this run is used only by this run’s downstream jobs.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	jobs:
	repo_cop_get_repos:
	runs-on: ubuntu-latest

	steps:
	- uses: actions/checkout@v3

	- name: Get All Repos
	# If no repo specified, get all repos
	if: github.event.inputs.repo-to-police == ''
	run: \|
	./repoCop_getRepos.sh

	- name: Stage specific repo
	# If repo specified, stage the repo in the list
	if: github.event.inputs.repo-to-police != ''
	run: \|
	echo "${{ github.event.inputs.repo-to-police }}" > all_repos

	# Action "Outputs" are unicode strings, so can't be a "list", or at least would be more complex to handle the data structure
	# Rather than deal with that, going to upload the binary list file, and will read in next step
	- name: Upload repos list
	uses: actions/upload-artifact@v3
	with:
	name: all-repos-${{ github.run_id }}
	path: all_repos

view raw repo_cop_get_repos.yaml hosted with ❤ by GitHub

The next job runs as normal — it downloads the list of repos (well, repo, singular, for this job) on line 13–17. It uses the same github.run_id hash to make sure it has the right artifacts to continue.

And then we run our repoCop.sh script that does all the work.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	repo_cop:
	runs-on: ubuntu-latest
	timeout-minutes: 720 # 12 hours - Default is 360 minutes / 6 hours
	needs: repo_cop_get_repos # Require the repos list to be generated first
	strategy:
	fail-fast: false # Keep going even if one shard fails
	matrix:
	shard: [1/2, 2/2]

	steps:
	- uses: actions/checkout@v3

	- name: Download repos list
	id: download-repos
	uses: actions/download-artifact@v3
	with:
	name: all-repos-${{ github.run_id }}

	- name: GitHub Repo Cop
	run: \|
	./repoCop.sh
	env:
	SHARD: ${{ matrix.shard }}

view raw repo_cop.yaml hosted with ❤ by GitHub

There are a few changes I made to the internal script to better handle running only a single repo, spanning two sharded workers (because that’s hard to make dynamic) (but I’m working on it!), but that’s so bespoke to my script, I’ll leave it as an exercise for the reader.

Summary

Whew, this got long, even split into two parts! As part of these two articles, we built a GitHub App that listens for Repository events (including new repo events), and sends a webhook (with a seeded SHA1 hash in the header) to our API gateway. That API gateway packages up the json webhook body, and passes it along to our lambda.

Our lambda kicks off, grabs some secrets, validates the SHA1 hash by doing its own hash and comparing, then extracts the type of action that triggered the webhook. If it’s not a new repo, we exit. If it is, we construct a REST json body, and send an authenticated API call to GitHub to trigger an Action to tell it to police (configure) this specific repo only.

The entire event-driven architecture takes less than 60 seconds to do all of this, and can scale to huge volumes. It wouldn’t even take much to read the Org from the same webhook, and handle multiple Orgs. Heck, it could be a product without too much work.

All code is here, please steal it and use it to build cool stuff!

GitHub - KyMidd/GitHubCopLambda-EventDriven: Part 1…
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com

Hope you enjoyed reading it as much as I did writing it.

Good luck out there.
kyler

Let’s Do DevOps

🔥Let’s Do DevOps: Making a GitHub Action…