🔥Let’s Do DevOps: GitHub to Jenkins Custom Integration using Actions, Bash, Curl for API Hacking
🔥Let’s Do DevOps: GitHub to Jenkins Custom Integration using Actions, Bash, Curl for API Hacking
This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
This past few weeks I’ve skipped all my meetings. I literally got the “hey are you okay” messages in slack. And I’ve been okay! But I’ve found a juicy problem, and I have been hacking on it, and I finally got it working to a sufficient quality and cleanliness that I want to share it in case you also want to!
The problem set is this — Your source code used to be internal to your network, and worked with Jenkins to build and deploy all your code. That integration is a black box, but it works well! However, your source control is moving to ✨The Cloud✨ on GitHub, which is very cool, but presents a series of problems:
How will GitHub talk to Jenkins in a secure way (Do we put Jenkins on the internet?)
How will GitHub integrate with Jenkins to run the appropriate jobs and track the outcome?
How will GitHub authenticate to Jenkins to run jobs?
In the course of solving this problem I’ve learned a ton about Jenkins and even a little about GitHub. Let’s do this.
If you want to skip all the how we got there and jump right to the working GitHub Action to track a Jenkins job based on GitHub commit message, scroll to the end of this blog for the GitHub link ❤ :)
Step 0: Grok The Problem
Working with other CI/CD tools, the pipeline always executes within the context of the updated code. There’s no need to propagate a branch name or commit or anything like that, because it’s implicitly known.
However, that’s not how Jenkins works, at least in my environment. Working with dev teams, they described the following workflow:
A Pull Request (PR) is created with a branch and at least 1 commit.
This triggers a notice to Jenkins (later on I identified this is a commitNotification API call) that says, “hey Jenkins, ≥1 branches with ≥1 commits have been created in this git repository and need to be built
Jenkins says “cool” and thus ends the synchronous portion of the exchange.
Jenkins asynchronously iterates through all projects (ALL projects!) to see if any grab information from this particular source control (and also if they have
pollingenabled, even if the polling isn’t set to ever run this decade)Jenkins clones the repo fully and finds all new branches and all new commits.
Jenkins checks the projects source control that use this repo and looks at their branch filters. If any project matches the “new branch(es)” filters, that project is executed.
The project run checks the source control and runs at least one job. I say
at least onebecause Jenkins casually and frequently spins out multiple jobs to build many commits in parallel. That makes things hard to track! I can’t watch for thefirstjob to run for the expected name, I have to find therightjob. More on that later.The job runs and publishes an exit code. Software written by people much smarter than me polls Jenkins and surfaces the exit code to the source control tool so folks know if they need to keep iterating on their software.
^^ This list is WAY MORE than I expected. Compared to other CI/CDs, that basically “run code in this commit within this branch, exit 1 or exit 0”, this is super cool and complex, and took me a few days (okay, a week or so) to fully grok.
And we need to do this. At least the external parts:
Notify Jenkins there is a new commit in a particular git repo
(Jenkins does cool Jenkins things, and builds the jobs)
Find the right job and watch it
Surface the exit code to the PR
Block the PR’s merge unless Jenkins built successfully
Awesome. Now that we’ve worked hard, let’s work hard to avoid more work. Has anyone else solved this problem?
Step 1, Copy The Smart Kid
Working as a software engineer isn’t a lot like school. In school, you’re encouraged to build your own tools. Don’t look over anyone’s shoulder please. In the post-school world, that is exactly what you ARE encouraged to do. So let’s do it.
Jenkins has a native method for triggering jobs via a WebHook, which GitHub fully supports. However, Jenkins would have to be on the internet to receive those webhooks. Our primary CI/CD tooling is absolutely critical for us to build software, so we’re quite careful with updating it — read, we’re not applying patches the day they come out. That could mean our Jenkins is vulnerable for a while after a zero day is discovered — that is a pretty scary host to put on the internet, so let’s not do that.
We could potentially obfuscate our Jenkins host behind an AWS API Gateway. That sounds like a good idea, but it would only work for the inbound request to send a commit notification. It doesn’t solve identifying the right job, tracking it, and surfacing the resulting exit code to GitHub. Not a full solution. Let’s consider it. Back burner.
There is a project from Mickey Goussetorg to trigger a specific Jenkins job and track it. This looked quite promising but didn’t actually work. Potentially because it was built for a different version of Jenkins. It sets out to trigger a specific job using user credentials (a deprecated and no longer recommended method of authenticating to Jenkins), triggering a specific job on Jenkins, and tracking it. Other than the not working part, this looks great. Also back burner.
GitHub - mickeygoussetorg/trigger-jenkins-job: I am a composite action that triggers a Jenkins job…
I am a composite action that triggers a Jenkins job using the Jenkins API, and returns the success/failure of the job…github.com
As for sending notifications back to GitHub about the status of jobs, there’s an app for that! The GitHub Checks plugin is an integration layer for Jenkins that helps other apps send notifications back to a GitHub PR for the status of builds. It looks so, so cool. But there is one small caveat:
This very awesome tool requires a significantly newer version of Jenkins than I’m working with. I even reached out to the developers and asked,
🥺 Is it possible this tool would work with our older version of Jenkins 🥺?
Tim Jacomb responded very quickly (yay!) that this tool absolutely won’t work (crap), and advised us to upgrade to a newer version of Jenkins. I don’t disagree at all, but not the problem set I’m solving right now.
Well
So where does that leave us? Not with much, honestly. There are a smattering of tools that would potentially work if we were on a much newer version of Jenkins. Unfortunately that’s not on the table now — we have more than 1k projects that would require updates. I’d love to get on board with that, but not right now. For now we kick the can and solve it ourselves.
So, let’s write some software!
Fine, I’ll Do It Myself
Laziness having lost out, I suppose I’ll write it myself. Let’s start walking through the problem set and solving each one.
Get Into the Network
First, we need to be inside our network. I don’t want to expose Jenkins to the internet. We could build an API gateway, but I have an entire pool of Runners that live in our network, so let’s use those!
Bam, problem solved, we’re in.
Authenticate to Jenkins
Almost every API request to Jenkins is authenticated. Weirdly, not the commitNotification API call, we’ll talk about that soon). We can send a PAT, or Personal Access Token. However, that method is deprecated, and I don’t want to implement something that’ll break in the near future if I can help it.
The new method is using an API Token. This token is created under a particular user and permits securely authenticating to Jenkins. Sweet, that works for me. I created an API token under my user for testing, and it works from my host! Let’s create the API token under our CI user and move forward.
| curl -I -s -X POST -u "$userName:$api_token" "${urlOfJenkinsServer}/api/stuff/goes/here" |
Send a commitNotification
As much as possible, we’re replicating what already happens. Goodness knows I don’t want to update thousands of projects or GitHub repos to do something new. So rather than calling a particular job, let’s send a commitNotification and gather the job from there!
Jenkins requires the notifyCommit API call to include the name of the repo. We construct that value with 2 pieces of info. First, the GitHub action running this was executed from a repo, and GitHub sets some default environment variables.
We send that to our custom GitHub Action and combine it with some static values to build a git link. This will match what your Jenkins projects show under their “SCM” (Source Code Management) links.
| repo_clone_url="https://github.com/${github_repository}.git" |
And then we tack it onto the API call like this. Note that we get back a bunch of Triggered jobs. Woot, we’ve told Jenkins to do some SCM scraping and spin up some jobs!
| > COMMITNOTIFICATION=$(curl -I -s -X POST -u "$userName:$api_token" "${urlOfJenkinsServer}/git/notifyCommit?url=${repo_clone_url}") && echo "$COMMITNOTIFICATION" | |
| HTTP/1.1 200 OK | |
| X-Content-Type-Options: nosniff | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_Any/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_GitHub/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_Release/ | |
| Content-Type: text/plain;charset=ISO-8859-1 | |
| Content-Length: 203 | |
| Server: xxxxxxx |
Identify Our Validation Job
My org uses a standard nomenclature for job naming — the job that is built for any commit is called JobName_Any. Super easy to follow.
Since that standard is always true, and there should always only be 1 job with that name, we filter the COMMITNOTIFICATION var we stored our commitNotification API call response in, to find just that job name. We’ll use that next to track stuff.
| jenkinsJobName=$(echo "$COMMITNOTIFICATION" | grep -E 'Triggered' | cut -d " " -f 2 | grep -E '_Any' | rev | cut -d "/" -f 2 | rev | head -n 1) && echo "$jenkinsJobName" |
Find the RIGHT Job
Remember I said that Jenkins will build all commits that it hasn’t yet. That means that we can’t just grab the first job with name $jenkinsJobName, we have to grab ALL JOBS with $jenkinsJobName and filter them for something to link it to this particular version of code. In this case, we filter for the SHA of the commit that drove this build.
Since both GitHub (where the code is stored) and Jenkins (which clones the repo to check for changes) understand the git metadata, we can send the git attributes we need from GitHub to our Action, which can filter jobs based on the git metadata at Jenkins. Perfect! Well, sort of.
Interestingly (and this is still a mystery to me), GitHub Actions inserts a 👻ghost commit👻 when the code is handed off to the Action to run. Which means that the well know environment variable in Actions for the commit that drove our build, $GITHUB_SHA, doesn’t work.
Which is lame because, 1. It’s confusing and 2. I’m lazy. However, all problems can be solved, so let’s go find that information ourselves as part of our build. We first (line 11), clone the whole repo. Note that fetch-depth: 0, which is required to pull all git metdata. If you don’t, you won’t have the whole git log to filter.
Which we do, on line 19 — grab the git log, then find all the commits and then grab the 2nd one (the one after the 👻ghost commit👻 GitHub apparently inserts) with sed -n 2p, and then we cut the commit text from the front to just have our SHA message for the commit driving this build.
Cool, now our Action knows which commit message is driving our build. Let’s poll Jenkins for jobs and grab all the required info we need.
This is where our API hacking starts to get super interesting.
First we send Jenkins an authenticated request to get all the jobs it has in recent memory (the last 30 or so jobs are returned for some reason) and their metadata. Check our the tree argument. This tells Jenkins before it even sends us information back exactly what we’re looking for. So rather than getting back 30k lines of Json, we only get back a few thousand, which is both faster and more efficient, woot!
Then (line 2), we filter for all builds, and on line 3, we select only the stanza that matches the SHA1 of our build we generated on GitHub Actions. Select is neat — it doesn’t remove data from our data-set, it just allows filtering for attributes.
We’re left with a stanza that contains 2 jobs. I have no idea why, but the key of our data is the name of our branch — so cool, let’s grab that specific branch, then the build number.
However, we’re not guaranteed to have only a single entry here — imagine you have an external dependency that’s broken, so you build your job a few times without pushing any code changes! So we sort the list for the newest Jenkins build sort -r (reverse sort) and grab only the first entry with head -n 1 .
| curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&pretty=true&tree=allBuilds\[actions\[buildsByBranchName\[*\[*\[*\]\]\]\]\]" \ | |
| | jq ".allBuilds[].actions[].buildsByBranchName \ | |
| | select (.[]?.marked.SHA1==\"${commit_sha}\") \ | |
| | .\"${branch}\" \ | |
| | .buildNumber" \ | |
| | sort -r \ | |
| | head -n 1 |
Awesome, we now have the particular build number that corresponds to this particular commit in our source control. Sweet. I have my code iterate over this call a few times until it’s populated because Jenkins might delay building jobs if it’s busy.
We also grab the Job’s URL so we can print it in our Action log for users to easily click through from GitHub to Jenkins (ease of life is important!).
| curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" \ | |
| | jq -r ".builds[] \ | |
| | select(.id==\"${JOBID}\") \ | |
| | .url" \ | |
| | head -n 1 |
Poll for Job Status
Awesome, now we know our job ID and job URL. We don’t have any cool Jenkins-side tooling that’ll alert our Action when Jenkins is done, so we set up a forever while loop.
We check if the job is building (true/false, line 4) and the job status (SUCCESS or other, line 5). If there is no job status (the job is still running), we report the job is still building. If the job has a status of any kind (success or fail), we break out of our while loop.
| while [ i = i ]; do | |
| # Check job status | |
| echo "Query Build Job Status" | |
| BUILDING=$(curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" | jq -r ".builds[] | select(.id==\"$JOBID\") | .building") | |
| JOBSTATUS=$(curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" | jq -r ".builds[] | select(.id==\"$JOBID\") | .result") | |
| if [ "$JOBSTATUS" == "null" ]; then | |
| # Job is still building | |
| echo "Job is still building: ${BUILDING}" | |
| else | |
| # If JOBSTATUS populated, it has finished. Break and exit watch loop | |
| break | |
| fi | |
| # Wait pollTime seconds | |
| echo "Jenkins still working hard, sleeping for ${pollTime} seconds" | |
| echo "--------------------------------------" | |
| sleep $pollTime | |
| currentTimeSeconds=$(date +%s) | |
| if [[ "$currentTimeSeconds" > "$finishTimeInSeconds" ]]; then | |
| echo "Timeout value reached. Exiting with error due to timeout" | |
| echo "--------------------------------------" | |
| exit 1 | |
| fi | |
| done |
We sleep for $pollTime to give Jenkins some time to do its hard work, then we check if we’ve taken too long — by comparing the current time with our time-limit. If we’ve taken too long, something has probably gone wrong, so we exit too.
We loop until we get a job status, or until we pass our timeout. Either breaks the while loop.
Surface the Exit Code
We assume the script has continued (we have a job status!) instead of exiting due to timeout. Let’s evaluate if the job status is a happy one or a sad one. If $JOBSTATUS is SUCCESS (line 10), that’s easy — exit 0 to tell GitHub that this is a happy exit, it worked.
If the job exited with UNSTABLE, I decided to classify as a happy result, also exit 0.
However, any other code (FAILURE, NOT_BUILT, ABORTED) sounded pretty unsuccessful to me, so exit 1, which will tell GitHub that our Action is unhappy with the state of things.
| # Read output value from job | |
| #Potential Values: https://javadoc.jenkins-ci.org/hudson/model/Result.html | |
| #SUCCESS - Build had no errors | |
| #UNSTABLE - Build had some errors but they were not fatal | |
| #FAILURE - Build had a fatal error | |
| #NOT_BUILT - Module was not build | |
| #ABORTED - Manually aborted | |
| # Close out | |
| echo "Job status is ${JOBSTATUS}" | |
| if [[ "$JOBSTATUS" == "SUCCESS" ]]; then | |
| echo "Build completed successfully" | |
| echo "--------------------------------------" | |
| exit 0 | |
| elif [[ "$JOBSTATUS" == "UNSTABLE" ]]; then | |
| echo "Build completed with UNSTABLE result" | |
| echo "--------------------------------------" | |
| exit 0 | |
| else | |
| echo "Job status is ${JOBSTATUS}" | |
| echo "Build did not complete successfully" | |
| echo "Sorry dude" | |
| echo "--------------------------------------" | |
| exit 1 | |
| fi |
Publish as an Action
Problems solved! Well, it works. Surely we’ll iterate on this in the future. Rather than including all the logic in every repo (there are thousands of them dude), let’s publish this as a GitHub Action in our github Org and let folks call it. That way our code lives in one place for iterating and others can just use it.
We create a new repo in our GitHub Organization and publish our Action code to the root. The Action block that others will call is called, what else, action.yml.
The Action code itself is kinda neat. First we iterate over all the information we require to be sent to us. We set a description, which are required, and optionally can set a default value (line 16).
| name: Jenkins Notify, Track, Surface Exit Code | |
| description: 'I am used to tell Jenkins a commit was made' | |
| inputs: | |
| jenkins-server: | |
| description: 'URL of the Jenkins server with trailing slash' | |
| required: true | |
| jenkins-username: | |
| description: 'User name for accessing Jenkins. Store this in a secret for security' | |
| required: true | |
| jenkins-api-token: | |
| description: 'API token for accessing Jenkins. Store this in a secret for security' | |
| required: true | |
| verbose: | |
| description: 'true/false - turns on verbose logging' | |
| required: false | |
| default: 'false' |
Then we establish what the Action actually does using a runs: block. We use a composite type (as opposed to Docker or Javascript, ref).
We map our parameters into bash (line 7–15).
Then we start doing stuff!
| runs: | |
| using: "composite" | |
| steps: | |
| - id: jenkins-any-pr-validate | |
| run: | | |
| #Parameters | |
| urlOfJenkinsServer="${{ inputs.jenkins-server }}" | |
| pollTime=${{ inputs.poll-time }} | |
| timeoutValue=${{ inputs.timeout-value }} | |
| verbose=${{ inputs.verbose }} | |
| userName=${{ inputs.jenkins-username }} | |
| api_token=${{ inputs.jenkins-api-token }} | |
| github_repository="${{ inputs.github_repository }}" | |
| commit_sha="${{ inputs.commit_sha }}" | |
| branch="${{ inputs.branch }}" | |
| # More bash code here |
In the Action module repo, make sure to update from the default Not accessible to whichever option matches what you’re looking for. I only need this Action with the context of my individual Organization, not the entire Enterprise
.
Call the Action and Block the PR (Or Not!)
Then we switch over to the GitHub repo that needs to run a job and block PRs. We need to create an Action file at the .github/workflows/* path. We give it a name (line 1), and set the Action to run within a pull request for any action, which maps to commits (line 4).
We also importantly set where the Action runs — like, which compute actually runs the code. We set this to our internal runner. If your Jenkins is private, make sure to run this trigger and polling from internal to edge security so it can reach Jenkins.
| name: PR Validate | |
| # Run this job on pull request with any action, which includes opening and any commit | |
| on: [pull_request] | |
| jobs: | |
| jenkins_pr_validate_any: | |
| runs-on: [self-hosted, MyInternalRunnerPool] |
Then we start on our Action steps. We first identify the right commit_sha (line 10), then write it to our terminal with echo, then pipe to tee -a to also write it to the $GITHUB_ENV variable that makes it usable for follow-up actions, which we require since we next call our custom Action.
Then on line 15 we call our custom GitHub Action. Line 17 is the location (OrgName/RepoNameWhereActionIs) and the version (@v1) or whatever Release/tag/commit you want to call.
Next we send the information the Action requires to run, like the Jenkins secret info, the github repo info, and the commit sha and branch.
| steps: | |
| - uses: actions/checkout@v3 | |
| with: | |
| fetch-depth: 0 | |
| - name: Identify PR commit | |
| run: | | |
| # Interestingly, GITHUB Actions are served a git log with an extra commit | |
| # Genuinely no idea why, so we grab the appropriate commit from the log | |
| commit_sha=$(git log | grep commit | sed -n 2p | cut -d " " -f 2) | |
| # Write commit sha to GITHUB_ENV so downstream tasks can use it | |
| echo "commit_sha=$commit_sha" | tee -a $GITHUB_ENV | |
| - name: Jenkins Any PR Validate | |
| id: jenkins_any_pr_validate | |
| uses: your-internal-org/repo-with-action-in-it@v1 # Note, the `v1` refers to a "release" of your Action | |
| with: | |
| jenkins-username: "${{ secrets.JENKINS_USERNAME_FOR_RUNNING_JOB }}" # Store the secrets Org wide so all can access | |
| jenkins-api-token: "${{ secrets.JENKINS_API_TOKEN_FOR_RUNNING_JOBS }}" # Store this API secret Org wide so all can access | |
| github_repository: $GITHUB_REPOSITORY # This is a GitHub well known environment var | |
| commit_sha: ${{ env.commit_sha }} # This is populated by us above | |
| branch: $GITHUB_HEAD_REF # This is a GitHub well known environment var |
The Action will run and surface an exit 0 (our job ran and worked, yay!) or an exit 1 (our job failed to start, or validation failed somehow). That will cause this Action in our repo to succeed or fail to match those exit codes.
You can tell your repository to require this status check to run, and to pass, by going into the branch protection policies. You’ll require admin on this repo (or Org, or Enterprise) in order to set these policies, so if you don’t see them that’s why).
Go to Settings in the repo → Branches → (branch name) → and set the “Require status check to pass before merging” and find the status check that matches the job name that we’re running, then hit Save at the bottom.
Pull Requests in your repo will now require this Action to succeed in order to qualify for merge, all without human intervention.
What’s it Look Like?
The job run is laughably boring compared to the complexity of what’s going on behind the scenes. We deliver a commit notification, the job starts after a few dozen seconds, we publish the job info, then track it until it finishes.
Which is the goal here! Someone reading this will be convinced this was an easy project to build, “Why did it take you so long?”. 😂
Similarly, a failure looks about the same, but it’s clear from the exit 1 / red bar GitHub publishes that we failed. We don’t attempt to suss out why we failed — folks can click on the URL we published which links to the Jenkins job to go see what unit tests failed as to why their code isn’t valid.
Future Improvements
Right now we track only a single job with a constructed name — xxx_Any. It’d be neat to be able to track multiple xxx_Any jobs if they are spun out. I’m sure there will be repos in the project that are linked to multiple Jenkins jobs.
Summary
We did A LOT. This was about 2 weeks of my life for a while, skipping all meetings and building cool 💩 instead.
We built a custom GitHub Action that we published within our Org. That Action talks to Jenkins within our secure environment, using authenticated API calls via curl to send a commit notification for our source repo, then tracks n spin-off jobs using a well-known nomenclature name, and matches the commit SHA that triggered our GitHub Action to the Jenkins job to definitely link the right job to the right PR commit.
We required PRs in our repo to require a pass on this org.
Here’s the source code so you can build it yourself:
GitHub - KyMidd/GitHubActions-JenkinsCommitNotificationTrackJob
This Action is intended for PRs in GitHub that use the xxxxx_Any job on Jenkins for validations. It sends a Jenkins…github.com
Thanks all, good luck out there!
kyler