Let's Do DevOps

Let's Do DevOps

Share this post

Let's Do DevOps
Let's Do DevOps
šŸ”„Building Better GenAI: Pre-Contextualizing Knowledge Base ResultsšŸ”„
Copy link
Facebook
Email
Notes
More

šŸ”„Building Better GenAI: Pre-Contextualizing Knowledge Base ResultsšŸ”„

aka, the knowledge base would really like to know this

Kyler Middleton's avatar
Kyler Middleton
Apr 22, 2025
āˆ™ Paid

Share this post

Let's Do DevOps
Let's Do DevOps
šŸ”„Building Better GenAI: Pre-Contextualizing Knowledge Base ResultsšŸ”„
Copy link
Facebook
Email
Notes
More
Share

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

I’m calling my ā€œBuilding a Slack Bot with AI Capabilitiesā€ series done, but I’m not done improving and building GenAI tools. This article will continue building the maturity of GenAI by establishing patterns that make these tools more powerful.

šŸ”„Building a Slack Bot with AI Capabilities - From Scratch! Part 1: Slack App and EventsšŸ”„

šŸ”„Building a Slack Bot with AI Capabilities - From Scratch! Part 1: Slack App and EventsšŸ”„

Kyler Middleton
Ā·
December 3, 2024
Read full story

When you ask Vera, the bot we’ve built over the past 9 entries into the Building… series, a question, a LOT of stuff happens in the background. It’s mostly invisible to you, but it:

  1. Walks the Slack thread to construct a bedrock-compatible conversation

  2. Flattens the conversation to a string-ified convo, then queries the Bedrock Knowledge Base (which is really OpenSearch in a trenchcoat) for 25-50 relevant vectorized results

  3. Use the reranker() model to match results to your question to improve the fidelity of results

  4. Query the actual model for an answer, including all the compiled context

And it generally does a very good job, particularly for ā€œchat with the knowledge baseā€ style queries. However, that’s not always what we want. Sometimes we have highly structured data and we want to standardize or shim in other contextual data derived from what the user has entered before we query the knowledge base.

I’ve been playing around with adding an additional step, highlighted in the picture below, that derives structured data to assist the knowledge base in obtaining better results, from the unstructured information the user provides. It changes the pattern to look like this, bolded step is the new one:

  1. Walks the Slack thread to construct a bedrock-compatible conversation

  2. Query a model with instructions on how to standardize the data and what keywords would be needed to return high quality info. Ephemerally store that answer and provide to the knowledge base dip.

  3. Flattens the conversation to a string-ified convo, then queries the Bedrock Knowledge Base (which is really OpenSearch in a trenchcoat) for 25-50 relevant vectorized results

  4. Use the reranker() model to match results to your question to improve the fidelity of results

  5. Query the actual model for an answer, including all the compiled context

This sounds really abstract, so lets add an example. I’m building ā€œNetBotā€, a tool to help answer the ā€œis it the network?ā€ type questions engineers ask all the time. This tool is both trained on our large general data store (Confluence) but also specifically on highly structured data (the iOS and ASA configs that filter traffic across our network).

When folks ask ā€œdoes my VPN permit me to 1.2.3.4 hostā€, the knowledge base fails terribly - some/most network device rules don’t permit to a host, they permit to a CIDR, or a whole VPC/vNet. Knowledge base fetching using RAG generally isn’t smart enough to figure that out, so we have a model with system prompt instructions to:

Assistant should find any specific (/32) IPs in the thread, and specify the class B and Class C CIDRs for the network.

Assistant should provide no other information.

If there are no IPs, assistant must not respond at all.

With that context, the NetBot knowledge base results are markedly more related to the question, since they are now able to find rules that would permit or deny the flow, but which don’t have an exact keyword match.

Notably, NetBot isn’t finished, and I may transition the whole thing to agentic skills (via MCP?) in order to get it working to a level I want it to. However, this pattern has proven useful for all sorts of other use cases, so I wanted to share how it might improve your AI architectures.

Lets walk through how this is implemented - I wrote it all out so you can easily steal borrow it!

Implementation - Constants

Keep reading with a 7-day free trial

Subscribe to Let's Do DevOps to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
Ā© 2025 Kyler Middleton
Privacy āˆ™ Terms āˆ™ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More