This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!
Hey all!
This past few weeks Iâve skipped all my meetings. I literally got the âhey are you okayâ messages in slack. And Iâve been okay! But Iâve found a juicy problem, and I have been hacking on it, and I finally got it working to a sufficient quality and cleanliness that I want to share it in case you also want to!
The problem set is thisâââYour source code used to be internal to your network, and worked with Jenkins to build and deploy all your code. That integration is a black box, but it works well! However, your source control is moving to âšThe Cloudâš on GitHub, which is very cool, but presents a series of problems:
How will GitHub talk to Jenkins in a secure way (Do we put Jenkins on the internet?)
How will GitHub integrate with Jenkins to run the appropriate jobs and track the outcome?
How will GitHub authenticate to Jenkins to run jobs?
In the course of solving this problem Iâve learned a ton about Jenkins and even a little about GitHub. Letâs do this.
If you want to skip all the how we got there and jump right to the working GitHub Action to track a Jenkins job based on GitHub commit message, scroll to the end of this blog for the GitHub link â€Â :)
Step 0: Grok The Problem
Working with other CI/CD tools, the pipeline always executes within the context of the updated code. Thereâs no need to propagate a branch name or commit or anything like that, because itâs implicitly known.
However, thatâs not how Jenkins works, at least in my environment. Working with dev teams, they described the following workflow:
A Pull Request (PR) is created with a branch and at least 1 commit.
This triggers a notice to Jenkins (later on I identified this is a commitNotification API call) that says, âhey Jenkins, â„1 branches with â„1 commits have been created in this git repository and need to be built
Jenkins says âcoolâ and thus ends the synchronous portion of the exchange.
Jenkins asynchronously iterates through all projects (ALL projects!) to see if any grab information from this particular source control (and also if they have
pollingenabled, even if the polling isnât set to ever run this decade)Jenkins clones the repo fully and finds all new branches and all new commits.
Jenkins checks the projects source control that use this repo and looks at their branch filters. If any project matches the ânew branch(es)â filters, that project is executed.
The project run checks the source control and runs at least one job. I say
at least onebecause Jenkins casually and frequently spins out multiple jobs to build many commits in parallel. That makes things hard to track! I canât watch for thefirstjob to run for the expected name, I have to find therightjob. More on that later.The job runs and publishes an exit code. Software written by people much smarter than me polls Jenkins and surfaces the exit code to the source control tool so folks know if they need to keep iterating on their software.
^^ This list is WAY MORE than I expected. Compared to other CI/CDs, that basically ârun code in this commit within this branch, exit 1 or exit 0â, this is super cool and complex, and took me a few days (okay, a week or so) to fully grok.
And we need to do this. At least the external parts:
Notify Jenkins there is a new commit in a particular git repo
(Jenkins does cool Jenkins things, and builds the jobs)
Find the right job and watch it
Surface the exit code to the PR
Block the PRâs merge unless Jenkins built successfully
Awesome. Now that weâve worked hard, letâs work hard to avoid more work. Has anyone else solved this problem?
Step 1, Copy The Smart Kid
Working as a software engineer isnât a lot like school. In school, youâre encouraged to build your own tools. Donât look over anyoneâs shoulder please. In the post-school world, that is exactly what you ARE encouraged to do. So letâs do it.
Jenkins has a native method for triggering jobs via a WebHook, which GitHub fully supports. However, Jenkins would have to be on the internet to receive those webhooks. Our primary CI/CD tooling is absolutely critical for us to build software, so weâre quite careful with updating itâââread, weâre not applying patches the day they come out. That could mean our Jenkins is vulnerable for a while after a zero day is discoveredâââthat is a pretty scary host to put on the internet, so letâs not do that.
We could potentially obfuscate our Jenkins host behind an AWS API Gateway. That sounds like a good idea, but it would only work for the inbound request to send a commit notification. It doesnât solve identifying the right job, tracking it, and surfacing the resulting exit code to GitHub. Not a full solution. Letâs consider it. Back burner.
There is a project from Mickey Goussetorg to trigger a specific Jenkins job and track it. This looked quite promising but didnât actually work. Potentially because it was built for a different version of Jenkins. It sets out to trigger a specific job using user credentials (a deprecated and no longer recommended method of authenticating to Jenkins), triggering a specific job on Jenkins, and tracking it. Other than the not working part, this looks great. Also back burner.
GitHub - mickeygoussetorg/trigger-jenkins-job: I am a composite action that triggers a Jenkins jobâŠ
I am a composite action that triggers a Jenkins job using the Jenkins API, and returns the success/failure of the jobâŠgithub.com
As for sending notifications back to GitHub about the status of jobs, thereâs an app for that! The GitHub Checks plugin is an integration layer for Jenkins that helps other apps send notifications back to a GitHub PR for the status of builds. It looks so, so cool. But there is one small caveat:
This very awesome tool requires a significantly newer version of Jenkins than Iâm working with. I even reached out to the developers and asked,
đ„ș Is it possible this tool would work with our older version of Jenkins đ„ș?
Tim Jacomb responded very quickly (yay!) that this tool absolutely wonât work (crap), and advised us to upgrade to a newer version of Jenkins. I donât disagree at all, but not the problem set Iâm solving right now.
Well
So where does that leave us? Not with much, honestly. There are a smattering of tools that would potentially work if we were on a much newer version of Jenkins. Unfortunately thatâs not on the table nowâââwe have more than 1k projects that would require updates. Iâd love to get on board with that, but not right now. For now we kick the can and solve it ourselves.
So, letâs write some software!
Fine, Iâll Do It Myself
Laziness having lost out, I suppose Iâll write it myself. Letâs start walking through the problem set and solving each one.
Get Into the Network
First, we need to be inside our network. I donât want to expose Jenkins to the internet. We could build an API gateway, but I have an entire pool of Runners that live in our network, so letâs use those!
Bam, problem solved, weâre in.
Authenticate to Jenkins
Almost every API request to Jenkins is authenticated. Weirdly, not the commitNotification API call, weâll talk about that soon). We can send a PAT, or Personal Access Token. However, that method is deprecated, and I donât want to implement something thatâll break in the near future if I can help it.
The new method is using an API Token. This token is created under a particular user and permits securely authenticating to Jenkins. Sweet, that works for me. I created an API token under my user for testing, and it works from my host! Letâs create the API token under our CI user and move forward.
| curl -I -s -X POST -u "$userName:$api_token" "${urlOfJenkinsServer}/api/stuff/goes/here" |
Send a commitNotification
As much as possible, weâre replicating what already happens. Goodness knows I donât want to update thousands of projects or GitHub repos to do something new. So rather than calling a particular job, letâs send a commitNotification and gather the job from there!
Jenkins requires the notifyCommit API call to include the name of the repo. We construct that value with 2 pieces of info. First, the GitHub action running this was executed from a repo, and GitHub sets some default environment variables.
We send that to our custom GitHub Action and combine it with some static values to build a git link. This will match what your Jenkins projects show under their âSCMâ (Source Code Management) links.
| repo_clone_url="https://github.com/${github_repository}.git" |
And then we tack it onto the API call like this. Note that we get back a bunch of Triggered jobs. Woot, weâve told Jenkins to do some SCM scraping and spin up some jobs!
| > COMMITNOTIFICATION=$(curl -I -s -X POST -u "$userName:$api_token" "${urlOfJenkinsServer}/git/notifyCommit?url=${repo_clone_url}") && echo "$COMMITNOTIFICATION" | |
| HTTP/1.1 200 OK | |
| X-Content-Type-Options: nosniff | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_Any/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_GitHub/ | |
| Triggered: https://jenkins.hq.practicefusion.com/job/Job1_Release/ | |
| Content-Type: text/plain;charset=ISO-8859-1 | |
| Content-Length: 203 | |
| Server: xxxxxxx |
Identify Our Validation Job
My org uses a standard nomenclature for job namingâââthe job that is built for any commit is called JobName_Any. Super easy to follow.
Since that standard is always true, and there should always only be 1 job with that name, we filter the COMMITNOTIFICATION var we stored our commitNotification API call response in, to find just that job name. Weâll use that next to track stuff.
| jenkinsJobName=$(echo "$COMMITNOTIFICATION" | grep -E 'Triggered' | cut -d " " -f 2 | grep -E '_Any' | rev | cut -d "/" -f 2 | rev | head -n 1) && echo "$jenkinsJobName" |
Find the RIGHTÂ Job
Remember I said that Jenkins will build all commits that it hasnât yet. That means that we canât just grab the first job with name $jenkinsJobName, we have to grab ALL JOBS with $jenkinsJobName and filter them for something to link it to this particular version of code. In this case, we filter for the SHA of the commit that drove this build.
Since both GitHub (where the code is stored) and Jenkins (which clones the repo to check for changes) understand the git metadata, we can send the git attributes we need from GitHub to our Action, which can filter jobs based on the git metadata at Jenkins. Perfect! Well, sort of.
Interestingly (and this is still a mystery to me), GitHub Actions inserts a đ»ghost commitđ» when the code is handed off to the Action to run. Which means that the well know environment variable in Actions for the commit that drove our build, $GITHUB_SHA, doesnât work.
Which is lame because, 1. Itâs confusing and 2. Iâm lazy. However, all problems can be solved, so letâs go find that information ourselves as part of our build. We first (line 11), clone the whole repo. Note that fetch-depth: 0, which is required to pull all git metdata. If you donât, you wonât have the whole git log to filter.
Which we do, on line 19âââgrab the git log, then find all the commits and then grab the 2nd one (the one after the đ»ghost commitđ» GitHub apparently inserts) with sed -n 2p, and then we cut the commit text from the front to just have our SHA message for the commit driving this build.
Cool, now our Action knows which commit message is driving our build. Letâs poll Jenkins for jobs and grab all the required info we need.
This is where our API hacking starts to get super interesting.
First we send Jenkins an authenticated request to get all the jobs it has in recent memory (the last 30 or so jobs are returned for some reason) and their metadata. Check our the tree argument. This tells Jenkins before it even sends us information back exactly what weâre looking for. So rather than getting back 30k lines of Json, we only get back a few thousand, which is both faster and more efficient, woot!
Then (line 2), we filter for all builds, and on line 3, we select only the stanza that matches the SHA1 of our build we generated on GitHub Actions. Select is neatâââit doesnât remove data from our data-set, it just allows filtering for attributes.
Weâre left with a stanza that contains 2 jobs. I have no idea why, but the key of our data is the name of our branchâââso cool, letâs grab that specific branch, then the build number.
However, weâre not guaranteed to have only a single entry hereâââimagine you have an external dependency thatâs broken, so you build your job a few times without pushing any code changes! So we sort the list for the newest Jenkins build sort -r (reverse sort) and grab only the first entry with head -n 1Â .
| curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&pretty=true&tree=allBuilds\[actions\[buildsByBranchName\[*\[*\[*\]\]\]\]\]" \ | |
| | jq ".allBuilds[].actions[].buildsByBranchName \ | |
| | select (.[]?.marked.SHA1==\"${commit_sha}\") \ | |
| | .\"${branch}\" \ | |
| | .buildNumber" \ | |
| | sort -r \ | |
| | head -n 1 |
Awesome, we now have the particular build number that corresponds to this particular commit in our source control. Sweet. I have my code iterate over this call a few times until itâs populated because Jenkins might delay building jobs if itâs busy.
We also grab the Jobâs URL so we can print it in our Action log for users to easily click through from GitHub to Jenkins (ease of life is important!).
| curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" \ | |
| | jq -r ".builds[] \ | |
| | select(.id==\"${JOBID}\") \ | |
| | .url" \ | |
| | head -n 1 |
Poll for Job Status
Awesome, now we know our job ID and job URL. We donât have any cool Jenkins-side tooling thatâll alert our Action when Jenkins is done, so we set up a forever while loop.
We check if the job is building (true/false, line 4) and the job status (SUCCESS or other, line 5). If there is no job status (the job is still running), we report the job is still building. If the job has a status of any kind (success or fail), we break out of our while loop.
| while [ i = i ]; do | |
| # Check job status | |
| echo "Query Build Job Status" | |
| BUILDING=$(curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" | jq -r ".builds[] | select(.id==\"$JOBID\") | .building") | |
| JOBSTATUS=$(curl -s -X GET -u "$userName:$api_token" "${urlOfJenkinsServer}/job/${jenkinsJobName}/api/json?depth=10&tree=builds\[queueId,id,building,result,url\]&pretty=true" | jq -r ".builds[] | select(.id==\"$JOBID\") | .result") | |
| if [ "$JOBSTATUS" == "null" ]; then | |
| # Job is still building | |
| echo "Job is still building: ${BUILDING}" | |
| else | |
| # If JOBSTATUS populated, it has finished. Break and exit watch loop | |
| break | |
| fi | |
| # Wait pollTime seconds | |
| echo "Jenkins still working hard, sleeping for ${pollTime} seconds" | |
| echo "--------------------------------------" | |
| sleep $pollTime | |
| currentTimeSeconds=$(date +%s) | |
| if [[ "$currentTimeSeconds" > "$finishTimeInSeconds" ]]; then | |
| echo "Timeout value reached. Exiting with error due to timeout" | |
| echo "--------------------------------------" | |
| exit 1 | |
| fi | |
| done |
We sleep for $pollTime to give Jenkins some time to do its hard work, then we check if weâve taken too longâââby comparing the current time with our time-limit. If weâve taken too long, something has probably gone wrong, so we exit too.
We loop until we get a job status, or until we pass our timeout. Either breaks the while loop.
Surface the Exit Code
We assume the script has continued (we have a job status!) instead of exiting due to timeout. Letâs evaluate if the job status is a happy one or a sad one. If $JOBSTATUS is SUCCESS (line 10), thatâs easyâââexit 0 to tell GitHub that this is a happy exit, it worked.
If the job exited with UNSTABLE, I decided to classify as a happy result, also exit 0.
However, any other code (FAILURE, NOT_BUILT, ABORTED) sounded pretty unsuccessful to me, so exit 1, which will tell GitHub that our Action is unhappy with the state of things.
| # Read output value from job | |
| #Potential Values: https://javadoc.jenkins-ci.org/hudson/model/Result.html | |
| #SUCCESS - Build had no errors | |
| #UNSTABLE - Build had some errors but they were not fatal | |
| #FAILURE - Build had a fatal error | |
| #NOT_BUILT - Module was not build | |
| #ABORTED - Manually aborted | |
| # Close out | |
| echo "Job status is ${JOBSTATUS}" | |
| if [[ "$JOBSTATUS" == "SUCCESS" ]]; then | |
| echo "Build completed successfully" | |
| echo "--------------------------------------" | |
| exit 0 | |
| elif [[ "$JOBSTATUS" == "UNSTABLE" ]]; then | |
| echo "Build completed with UNSTABLE result" | |
| echo "--------------------------------------" | |
| exit 0 | |
| else | |
| echo "Job status is ${JOBSTATUS}" | |
| echo "Build did not complete successfully" | |
| echo "Sorry dude" | |
| echo "--------------------------------------" | |
| exit 1 | |
| fi |
Publish as an Action
Problems solved! Well, it works. Surely weâll iterate on this in the future. Rather than including all the logic in every repo (there are thousands of them dude), letâs publish this as a GitHub Action in our github Org and let folks call it. That way our code lives in one place for iterating and others can just use it.
We create a new repo in our GitHub Organization and publish our Action code to the root. The Action block that others will call is called, what else, action.yml.
The Action code itself is kinda neat. First we iterate over all the information we require to be sent to us. We set a description, which are required, and optionally can set a default value (line 16).
| name: Jenkins Notify, Track, Surface Exit Code | |
| description: 'I am used to tell Jenkins a commit was made' | |
| inputs: | |
| jenkins-server: | |
| description: 'URL of the Jenkins server with trailing slash' | |
| required: true | |
| jenkins-username: | |
| description: 'User name for accessing Jenkins. Store this in a secret for security' | |
| required: true | |
| jenkins-api-token: | |
| description: 'API token for accessing Jenkins. Store this in a secret for security' | |
| required: true | |
| verbose: | |
| description: 'true/false - turns on verbose logging' | |
| required: false | |
| default: 'false' |
Then we establish what the Action actually does using a runs: block. We use a composite type (as opposed to Docker or Javascript, ref).
We map our parameters into bash (line 7â15).
Then we start doing stuff!
| runs: | |
| using: "composite" | |
| steps: | |
| - id: jenkins-any-pr-validate | |
| run: | | |
| #Parameters | |
| urlOfJenkinsServer="${{ inputs.jenkins-server }}" | |
| pollTime=${{ inputs.poll-time }} | |
| timeoutValue=${{ inputs.timeout-value }} | |
| verbose=${{ inputs.verbose }} | |
| userName=${{ inputs.jenkins-username }} | |
| api_token=${{ inputs.jenkins-api-token }} | |
| github_repository="${{ inputs.github_repository }}" | |
| commit_sha="${{ inputs.commit_sha }}" | |
| branch="${{ inputs.branch }}" | |
| # More bash code here |
In the Action module repo, make sure to update from the default Not accessible to whichever option matches what youâre looking for. I only need this Action with the context of my individual Organization, not the entire Enterprise
.
Call the Action and Block the PR (Or Not!)
Then we switch over to the GitHub repo that needs to run a job and block PRs. We need to create an Action file at the .github/workflows/* path. We give it a name (line 1), and set the Action to run within a pull request for any action, which maps to commits (line 4).
We also importantly set where the Action runsâââlike, which compute actually runs the code. We set this to our internal runner. If your Jenkins is private, make sure to run this trigger and polling from internal to edge security so it can reach Jenkins.
| name: PR Validate | |
| # Run this job on pull request with any action, which includes opening and any commit | |
| on: [pull_request] | |
| jobs: | |
| jenkins_pr_validate_any: | |
| runs-on: [self-hosted, MyInternalRunnerPool] |
Then we start on our Action steps. We first identify the right commit_sha (line 10), then write it to our terminal with echo, then pipe to tee -a to also write it to the $GITHUB_ENV variable that makes it usable for follow-up actions, which we require since we next call our custom Action.
Then on line 15 we call our custom GitHub Action. Line 17 is the location (OrgName/RepoNameWhereActionIs) and the version (@v1) or whatever Release/tag/commit you want to call.
Next we send the information the Action requires to run, like the Jenkins secret info, the github repo info, and the commit sha and branch.
| steps: | |
| - uses: actions/checkout@v3 | |
| with: | |
| fetch-depth: 0 | |
| - name: Identify PR commit | |
| run: | | |
| # Interestingly, GITHUB Actions are served a git log with an extra commit | |
| # Genuinely no idea why, so we grab the appropriate commit from the log | |
| commit_sha=$(git log | grep commit | sed -n 2p | cut -d " " -f 2) | |
| # Write commit sha to GITHUB_ENV so downstream tasks can use it | |
| echo "commit_sha=$commit_sha" | tee -a $GITHUB_ENV | |
| - name: Jenkins Any PR Validate | |
| id: jenkins_any_pr_validate | |
| uses: your-internal-org/repo-with-action-in-it@v1 # Note, the `v1` refers to a "release" of your Action | |
| with: | |
| jenkins-username: "${{ secrets.JENKINS_USERNAME_FOR_RUNNING_JOB }}" # Store the secrets Org wide so all can access | |
| jenkins-api-token: "${{ secrets.JENKINS_API_TOKEN_FOR_RUNNING_JOBS }}" # Store this API secret Org wide so all can access | |
| github_repository: $GITHUB_REPOSITORY # This is a GitHub well known environment var | |
| commit_sha: ${{ env.commit_sha }} # This is populated by us above | |
| branch: $GITHUB_HEAD_REF # This is a GitHub well known environment var |
The Action will run and surface an exit 0 (our job ran and worked, yay!) or an exit 1 (our job failed to start, or validation failed somehow). That will cause this Action in our repo to succeed or fail to match those exit codes.
You can tell your repository to require this status check to run, and to pass, by going into the branch protection policies. Youâll require admin on this repo (or Org, or Enterprise) in order to set these policies, so if you donât see them thatâs why).
Go to Settings in the repo â Branches â (branch name) â and set the âRequire status check to pass before mergingâ and find the status check that matches the job name that weâre running, then hit Save at the bottom.
Pull Requests in your repo will now require this Action to succeed in order to qualify for merge, all without human intervention.
Whatâs it Look Like?
The job run is laughably boring compared to the complexity of whatâs going on behind the scenes. We deliver a commit notification, the job starts after a few dozen seconds, we publish the job info, then track it until it finishes.
Which is the goal here! Someone reading this will be convinced this was an easy project to build, âWhy did it take you so long?â. đ
Similarly, a failure looks about the same, but itâs clear from the exit 1 / red bar GitHub publishes that we failed. We donât attempt to suss out why we failedâââfolks can click on the URL we published which links to the Jenkins job to go see what unit tests failed as to why their code isnât valid.
Future Improvements
Right now we track only a single job with a constructed nameâââxxx_Any. Itâd be neat to be able to track multiple xxx_Any jobs if they are spun out. Iâm sure there will be repos in the project that are linked to multiple Jenkins jobs.
Summary
We did A LOT. This was about 2 weeks of my life for a while, skipping all meetings and building cool đ© instead.
We built a custom GitHub Action that we published within our Org. That Action talks to Jenkins within our secure environment, using authenticated API calls via curl to send a commit notification for our source repo, then tracks n spin-off jobs using a well-known nomenclature name, and matches the commit SHA that triggered our GitHub Action to the Jenkins job to definitely link the right job to the right PR commit.
We required PRs in our repo to require a pass on this org.
Hereâs the source code so you can build it yourself:
GitHub - KyMidd/GitHubActions-JenkinsCommitNotificationTrackJob
This Action is intended for PRs in GitHub that use the xxxxx_Any job on Jenkins for validations. It sends a JenkinsâŠgithub.com
Thanks all, good luck out there!Â
kyler