This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. If you find these useful, please consider subscribing - it’s what keeps these articles coming. <3

Hey all! Alex here again. In Part 1 we covered the polling architecture: EventBridge, delta queries, idempotency, and handing off to AgentCore. That got us to the point where the AgentCore worker receives an email payload and needs to do something useful with it.

This article covers what happens next. The worker needs to understand the full conversation, not just the latest message. It needs to run the agent. And it needs to send a reply that actually looks good in Outlook, Gmail, and Apple Mail. Each of those is its own rabbit hole.

This is Part 2 of a 2-part series:

• Part 1: Full architecture and Microsoft 365 integration

• Part 2 (this article): Building the email thread, sending formatted replies, and operating in production

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support writing more of these articles <3 :)

Hey all! I’m Alex, a new contributor to Let’s Do DevOps. At work we’ve been building out a fleet of agentic AI bots backed by AWS Bedrock AgentCore: bots that live in Slack, Teams, and now email. Kyler has covered the Slack bot in depth across several articles. I want to show you the email version: same AgentCore foundation, completely different integration challenge.

The goal was to lower the friction for people to interact with our internal agentic bot. Slack and Teams work great if you’re in them all day, but not everyone is. The C-suite lives in email, and there are great email-native use cases too. Forward a vendor offer and ask the bot if accepting it would violate the company gift policy. Get CC’d on a question you don’t know the answer to? CC the bot and let it respond. The inbox is already where a lot of conversation happens, so we brought the bot there.

This is Part 1 of a 2-part series:

• Part 1 (this article): Full architecture and Microsoft 365 integration

• Part 2: Building the email thread, sending formatted replies, and operating in production

The Problem: Email Is Not Slack

If you’ve built a Slack bot before, you know how it works. Slack sends you an event when someone messages the bot. You process it, respond, done. It’s push-based so Slack does the work of notifying you.

Email is more complicated.

While Microsoft Graph supports push notifications for mail via a subscription, those subscriptions expire after 7 days and have to be renewed, you have to handle a validation handshake on creation, and if your endpoint is slow or unavailable, Graph will start dropping notifications permanently with no recovery. It’s more moving parts than it looks.

We chose polling instead. Simpler to operate, no subscription lifecycle to manage, and the delta query pattern makes it efficient.

That choice shapes the whole architecture. Instead of an event-driven Lambda that wakes up when Graph pings it, you need a polling layer that runs on a schedule, checks for new messages, and decides what to do with them.

But once you have polling, you immediately have two new problems: efficiency (you don’t want to re-fetch every email every minute) and idempotency (you don’t want to process the same email twice if something goes wrong).

We solved both. Let me show you how.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In this 4th and last article in the AgentCore series, we’ll talk about real life deployment, operations, and lessons learned while building this project. Here are links to the other articles in the series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 (this article) - AgentCore Deployment, Operations, and Lessons Learned

Deploying an AgentCore runtime isn’t like deploying a Lambda function. There’s a container to build, a two-Lambda architecture to wire up, module boundaries to get right in Terraform, and a regional split that will trip you up if you’re not expecting it. Once it’s running, the operational model is different too. Your bot is a long-lived application now, not a function that spins up and dies. That changes how you think about health checks, debugging, cost, and failure modes.

This article walks through the Terraform patterns that hold everything together, what container lifecycle actually looks like in practice, how to debug when things go wrong, and an honest look at what I’d do differently if I started over. If you’ve been following along and thinking about building your own AgentCore agent, this is the practical stuff you’ll want before you start.

Let’s get into it.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In the last article, we covered how Vera connects to tool providers through the MCP Gateway and direct connections. Now let’s talk about something that makes AgentCore feel genuinely different from Lambda: memory.

This article is part of a series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 (this article) - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

This is persistent memory that survives across conversations, across days, across sessions. Not only can users teach the bot to call them a nickname, users can teach Vera multi-step business processes: “When I ask for standup notes, check Jira for my tickets, pull my recent PRs from GitHub, and summarize my Confluence updates from the past week.” Once learned, Vera executes that workflow every time without the user repeating the instructions. In the first month of deployment, Vera memorized roughly 2,000 facts across our user base, including work patterns, team structures, preferred workflows, and procedural preferences.

This is one of those features where the “learning AI” model really pays off. In Lambda, every invocation starts with a blank slate. You can bolt on memory by writing to DynamoDB or S3, but you’re building and maintaining that whole system yourself. AgentCore gives you a managed memory service with strategies, namespaces, and semantic search out of the box.

AgentCore memory has two layers that work together. Short-term memory events capture each conversation as it happens. A backend process then analyzes those events and extracts durable facts and preferences into long-term memory records. This article covers both layers, how we use the user preference strategy to give Vera per-user memory, the identity model that ties it all together, and the tools we built so users can see and control what’s been remembered about them.

Let’s get into it.

If you’d prefer to skip the write-up and just read the code, the entire codebase is public and MIT open sourced here.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In the last article, we talked about why I migrated from Lambda to AWS Bedrock AgentCore, the architecture with two Lambdas feeding into an AgentCore runtime, and the features you get out of the box.

This article is part of a series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 (this article) - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

Now let’s talk about tools.

Vera (our Slack bot) currently has access to tools from PagerDuty, GitHub, Atlassian (Jira + Confluence), Azure, AWS CLI, and Splunk. That’s a lot of tool providers, and each one has its own authentication mechanism, transport protocol, and quirks. Managing all of that in application code can cause a lot of sprawl.

AgentCore’s MCP Gateway is designed to solve this - a single, managed endpoint for all your tool providers. But today, the gateway doesn’t support every auth pattern, so I’m running a hybrid: some tools go through the gateway, others connect directly. This article covers both patterns, how they coexist, and where this is all heading.

If you want a deeper look at MCP tool integration with the Strands SDK specifically, I covered that in a previous article. We’ll be lighter on that here and focus more on the gateway pattern.

Let’s get into it.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

It’s been a while! I’ve been busy migrating Vera, my agentic AI helper, over to AWS AgentCore.

This article is part of a series:

• Part 1 (this article) - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

AgentCore has a few very cool features that made it attractive for migration, including:

• Longer running jobs: Can run for 8 hours! Much longer than Lambda’s 15 minute hard limit

• Avoiding cold starts: Lambda starts from scratch each time (with some exceptions for warm more expert-tier stuff), so having the bot ready to go is helpful

I’ve been building AI bots for enterprise use for a while now, and if you’ve been following along, you’ve seen the evolution. We started with a genAI Slack bot (https://www.letsdodevops.com/p/solving-aws-bedrocks-enterprise-logging) that could answer questions using RAG and knowledge bases. Then we leveled it up to an agentic Slack bot (https://www.letsdodevops.com/p/building-agentic-slack-bot) that could actually do things - query GitHub, check PagerDuty, search Jira.

Here’s the thing: that Lambda-based agentic bot? It works great. Seriously. Lambda with the Strands SDK running Claude can handle complex multi-tool workflows without breaking a sweat. If you’re building an agentic bot today and Lambda fits your needs, go for it.

But.

Lambda has a hard 15-minute timeout. For most agentic workflows, that’s plenty. But what happens when someone asks your bot to analyze a quarter’s worth of incidents, cross-reference them with deployment logs, and generate a report? Or when the agent needs to iterate through dozens of GitHub repos looking for a specific pattern? Those workflows can push past 15 minutes, and when Lambda hits that wall, it hits it hard.

Beyond the timeout, I kept hearing about AWS Bedrock AgentCore and its growing feature set. Memory that persists across conversations? A managed MCP gateway for tool access? Multi-agent orchestration? These aren’t things you can’t build on Lambda, but they’re things AgentCore gives you out of the box.

So I decided to migrate. Not because Lambda failed me, but because AgentCore offered a different model - one where my bot isn’t a function that spins up and dies, but an application that stays warm, maintains state, and processes requests like a real service.

This is the first article in a series documenting that migration. We’ll cover:

1. This article: What AgentCore is and why the “living application” model matters

2. MCP Gateway - one gateway to rule all your tools

3. AgentCore Memory - teaching your bot to remember

4. Guardrails and Knowledge Bases - safety and smarts

5. Deployment, operations, and lessons learned

Let’s do it.

If you want to skip all this architecture and just see the code, it’s all published here under an MIT license:

GitHub.com/KyMidd/AgentCore_AgenticSlackBot

The Problem with Lambda for AI Agents

Don’t get me wrong - Lambda is fantastic for AI workloads. I’ve been running agentic bots on Lambda for months, and the model works well:

• Pay-per-invocation: You only pay when the bot is actually working

• Auto-scaling: Slack blows up with requests? Lambda handles it

• Simple deployment: Push code, done

But there are friction points when you’re running long-lived agentic workflows:

The 15-minute ceiling. Most agent tasks complete in 2-5 minutes. But “most” isn’t “all.” When your agent needs to iterate through large datasets, make dozens of tool calls, or wait for slow external APIs, you can hit that timeout. And when you do, everything stops. No graceful degradation, no “let me finish this thought” - just death.

Cold starts during conversations. Lambda containers can be reused, but there’s no guarantee. If your user sends a follow-up message and it hits a cold container, you’re paying the initialization cost again - reloading models, re-establishing MCP connections, re-fetching secrets. It’s not catastrophic, but it’s not elegant either.

No persistent state. Every Lambda invocation starts fresh. Want to remember what you talked about 5 minutes ago? You need to build that yourself - fetch conversation history, reconstruct context, hope you got it right. Lambda doesn’t help you here.

The invoke-and-die pattern. This is the big mental model issue. Lambda wants you to do a thing and exit. Agents want to exist - to maintain connections, hold context, be ready for the next question. These models are fundamentally different.

None of these are dealbreakers. I worked around all of them. But when AgentCore offered a different approach, I was curious.

Enter AgentCore: An Application, Not a Function

AWS Bedrock AgentCore flips the model. Instead of invoking a function that dies after each request, you’re running a container that stays alive.

Think of it like this:

| Aspect | Lambda | AgentCore |
|--------|--------|-----------|
| Lifecycle | Invoke → Execute → Die | Start → Stay warm → Process many requests → Eventually die |
| State | None (rebuild each time) | Maintained in memory |
| Timeout | 15 minutes hard limit | Configurable (hours) |
| Connections | Re-establish each invocation | Keep alive |
| Mental model | Function | Application |

Your AgentCore runtime is a Docker container running on AWS-managed infrastructure. You define how long it can sit idle before AWS kills it (idle_timeout), and how long it can live total (max_lifetime). Between requests, it just... waits. Connections stay open. Memory stays loaded. When the next request comes in, you’re already warm.

This is a fundamental shift in how you think about your bot. You’re not writing a function anymore. You’re writing an application - one that handles HTTP health checks, processes incoming requests on a loop, and maintains state between calls.

The lambda pattern is to handle one single event and then die. The AgentCore pattern is your application spins up at the first request, and then it hangs around to handle other requests.

It’s fully warm, tokens are cached, MCP clients are prepped, and so we now need to understand which pieces of the application are shared vs distinct, and even which pieces of data are sensitive and need to be kept separate between requests.

The runtime stays warm based on your configuration:

# Runtime lifecycle configuration
idle_timeout = 900   # 15 minutes idle before shutdown
max_lifetime = 28800 # 8 hours max before forced restart

AWS manages the health checks. If your container becomes unhealthy, AgentCore restarts it. If traffic spikes, AgentCore can scale. You focus on the application logic; AWS handles the infrastructure.

Architecture Overview

Here’s the thing about AgentCore: you can’t just point Slack directly at it. Slack requires a webhook response within 3 seconds, and your AI agent is definitely not responding in 3 seconds. So we still need Lambda - but in a different role.

The architecture looks like this:

Receiver Lambda

Catches the Slack webhook, validates the signature, and immediately returns 200. Slack is happy. Then it asynchronously invokes the Invoker Lambda. This function is tiny and fast - 128MB, sub-second execution.

Invoker Lambda

Takes the payload and calls InvokeAgentRuntime. This is a synchronous call to AgentCore, but the Invoker doesn’t wait for it to complete - it fires and forgets. The Invoker exists mainly to provide IAM isolation; it has only the permission to invoke AgentCore, nothing else.

AgentCore Runtime

The actual bot. Receives the request, loads conversation history, spins up MCP connections, runs the agent loop with Claude, and posts the response directly back to Slack. This is where the magic happens, and it can take as long as it needs.

Why the two-Lambda dance? Separation of concerns. The Receiver needs to be fast and needs access to validate Slack signatures. The Invoker needs permission to call AgentCore. By splitting them, we follow least-privilege principles and keep each component focused.

What You Get Out of the Box

AgentCore isn’t just “Lambda but longer.” It comes with a suite of features that would take significant effort to build yourself:

MCP Gateway

A managed gateway for MCP (Model Context Protocol) servers with OAuth2 authentication via Cognito. Register your tools once, authenticate once, access everything. We’ll dive deep into this in Article 2.

AgentCore Memory

Persistent memory across conversations. Not just “what did we talk about in this thread” but “what are this user’s preferences” and “what facts have we learned.” Three memory strategies out of the box: session summaries, user preferences, and semantic knowledge. Article 3 covers this in detail.

Multi-Agent Orchestration

This one’s interesting. AgentCore supports agent-to-agent communication and hierarchical agent patterns - think a “manager” agent that delegates to specialist agents. I haven’t implemented this yet, but the capability is there. A supervisor agent could coordinate between a “GitHub specialist” and a “Jira specialist,” each with their own tools and context. Something to explore in a future article.

Container Lifecycle Management

AWS handles health checks, restarts, and scaling. You define the parameters; AWS keeps things running.

The Entrypoint Pattern: From Handler to Application

We were already using Docker containers for the Lambda-based bot, so the Dockerfile doesn’t change dramatically. What changes is how the code runs inside that container.

With Lambda, you write a handler function that processes one event:

def lambda_handler(event, context):
# Process this one request
# Return response
# Container may or may not stick around

With AgentCore, you write an application that stays alive and processes many requests:

from strands.multimodal.application import Application
app = Application()
@app.entrypoint
async def handle_slack_message(payload: dict):
# Process this request
# Container keeps running after we return
pass
if __name__ == “__main__”:
    app.run()  # Start HTTP server, wait for requests

That app.run() call is the key difference. It starts an HTTP server inside the container that:

• Responds to health checks from AgentCore

• Routes incoming requests to your handler

• Keeps the process alive between requests

Your handler gets called for each incoming request, but the container - and everything you’ve initialized - stays warm. MCP connections persist. Cached tokens stay valid. The mental model shifts from “spin up, do work, die” to “stay ready, handle work as it comes.”

What’s Next

This article covered the “why” and “what” of AgentCore. We talked about Lambda’s limitations for long-running agents, how AgentCore’s application model differs, and what features come bundled in.

In the next article, we’ll go deep on the MCP Gateway - how to set up unified authentication with Cognito, register multiple tool providers, and handle the quirks of different MCP servers (spoiler: Azure MCP on ARM64 is... special).

Until then, if you’re running agentic bots on Lambda and they’re working fine, don’t let this article convince you to migrate for migration’s sake. But if you’re hitting the 15-minute wall, or you want persistent memory, or you’re tired of rebuilding MCP clients on every invocation - AgentCore is worth a look.

The code for this entire project is open source: [GitHub link]. Feel free to poke around, steal ideas, or open issues when things don’t make sense.

Happy building!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

Here’s the thing about building custom solutions: they work great until the library you depend on ships an update. Then you’re stuck choosing betIen “maintain my clever workaround” or “rewrite everything to match the new API.” Neither option sounds fun.

VeraResearch is an agentic Slack bot built with AWS Strands that connects to six different MCP (Model Context Protocol) servers—GitHub, Atlassian, PagerDuty, Azure, AWS CLI, and Splunk. Each MCP server provides tools the agent can use to ansIr questions and complete tasks. Pull incident data from PagerDuty. Search Jira tickets. List AWS resources. You get the idea.

When you load multiple MCP servers, they inevitably have overlapping tool names. GitHub has get_user. PagerDuty has get_user. The next MCP you install probably has get_user. Without prefixing, the agent crashes when it tries to load the second one.

So I built a solution: a custom PrefixedMCPTool wrapper class that manually prefixed every tool name. github_get_user, pagerduty_get_user, azure_get_user. Problem solved. It worked perfectly.

Except now I owned ~75 lines of code that reached into MCPClient internals, manually managed client lifecycles, extracted tool lists, wrapped them, and returned tuples. Every time Strands updated how MCPClient worked, I had to check if our custom wrapper still functioned. Maintenance debt.

Then Strands PR #895 landed: first-class support for tool prefixing and filtering directly in MCPClient. The library now provides prefix and tool_filters parameters. Just pass a prefix string and optional tool filter instructions, and MCPClient handles everything.

If I’m maintaining custom abstractions, I’m on my own when breaking changes happen. If Strands provides the feature natively, their updates won’t break my code because they own the abstraction.

Let’s walk through removing my custom cruft and use the code the Strands maintainers team wrote.

To skip right to the code changes, see the commit here.

The Old Way: Custom Tool Wrapping with PrefixedMCPTool

The custom solution had three parts: a wrapper class, builder functions for each MCP server, and consumption code in worker_agent.py.

Here’s the PrefixedMCPTool wrapper class:

Every MCP client builder followed the same pattern. Here’s GitHub:

Then in worker_agent.py, I consumed these tuples:

This pattern repeated six times for six MCP servers.

The problems: I’m calling __enter__() to manually start clients, but Agent expects unstarted clients and calls load_tools() itself. I’m reaching into MCPClient internals with list_tools_sync(). Every builder function returns a tuple that needs unpacking, prefixing, and extending. When Strands changes MCPClient behavior, this breaks.

The New Way: What Strands Added and What I Deleted

Strands PR #895 added three things to MCPClient: a prefix parameter, a tool_filters parameter, and ToolProvider interface support. Each addition let me delete chunks of custom code.

Prefix Parameter

Strands now adds a prefix to every tool name automatically. Pass prefix=”github” and the client handles the rest.

Before:

After:

I deleted the entire PrefixedMCPTool class (25 lines) and the add_prefix_to_mcp_tools() function.

tool_filters Parameter

Strands now filters tools using lambda functions you provide. Want only read-only tools? Pass a filter that checks tool names.

Before:

After:

I deleted the manual filtering loops from six MCP client files.

ToolProvider Interface

MCPClient now implements ToolProvider, which means Agent starts the client automatically when it needs tools. I don’t call __enter__() or list_tools_sync() anymore.

Before:

After:

Then in worker_agent.py, instead of unpacking tuples and extending tool lists, I just append the client:

Before:

After:

Agent calls load_tools() on the client when it needs to, which starts the connection and extracts tools automatically.

Gotchas

• Prefix separator: Strands adds an underscore between prefix and tool name automatically. I initially set TOOLS_PREFIX = “pagerduty_” (with trailing underscore) and got tool names like pagerduty__list_incidents (double underscore). Remove the trailing underscore from your prefix constants.

• Filter timing: tool_filters run after the prefix is applied. Check for github_get_user, not get_user.

Summary

In this blog we walked through how to move from my custom-built MCP management interface to the MCP tool management pattern that the Strands maintainers built. It’s both less code for us to manage, as well as quite a bit faster to get all those MCPs built in parallel instead of in sequence.

Remember, every custom abstraction is maintenance debt when the upstream library changes. When the library ships the feature natively, future updates are forward compatible by default.

Always let someone else do the hard work.

Full code here: https://github.com/KyMidd/SlackStrandsAgenticBot

Good luck out there!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

We’ve built three AI bots across our organization: VeraSlack, VeraTeams, and VeraResearch. All three run on AWS Bedrock using Claude Sonnet. They answer questions, search knowledge bases, and help employees find information.

When we first deployed these bots, we relied on AWS Bedrock’s built-in logging. Every API call generates a log entry with timestamps and request IDs. Our Lambda functions logged to CloudWatch. We figured we had full visibility. If security needed to audit a conversation, they’d just look at the logs.

The first time our security team asked “What did Bob from Finance ask the bot last Tuesday?” we couldn’t answer it. We had thousands of Bedrock API log entries, but no way to connect them to Bob’s question.

An employee asks: “What’s our process for requesting SSL certificates?” That single question triggers the bot to query the knowledge base, call a reranking service, search Confluence, check PagerDuty, maybe query Jira, synthesize the results with Bedrock, and format a response. That’s fifteen separate API calls across multiple services.

Each API call logs separately. Each log entry is atomic—just a request ID, timestamp, and parameters. No user name. No original question. No conversation context.

To reconstruct what Bob asked, you’d need to find his username in Slack’s event logs, correlate that to a Lambda execution timestamp, connect it to fifteen different Bedrock API calls scattered across log groups, and piece together the conversation.

Pro tip: This is awful, I don’t want to spend all my time reading logs

The problem gets worse with conversational bots. VeraResearch uses Bedrock Agents and can have multi-turn conversations. A five-minute troubleshooting session generates hundreds of log entries. Bedrock logs API requests, but has no concept of “user,” “conversation,” or “session.”

We needed to answer basic questions: What is Bob asking? What information are we providing about sensitive topics? Which bot is being used most? Bedrock’s logging couldn’t answer these.

So we built our own audit logging system. One that captures user identity, the original question, conversation context, the bot’s response, and ties it together with a single session ID.

In this article, I’ll walk you through why platform logging fails for agentic bots, how we designed an audit system across Slack and Teams, and the platform-specific challenges we solved.

It’s Harder Than It Looks

Good audit logging needs specific fields: who asked (user_id), what they asked (user_query), the conversation context, what we told them (response), when it happened (timestamp), and which bot answered (bot_name). We also need the AWS account ID for environment tracking and a session ID that ties audit logs directly to Lambda execution logs.

We’re writing these logs to CloudWatch Logs using a shared log group called /aws/ai-bots/audit-logs. Each bot gets its own log stream pattern: VeraSlack/{session_id}, VeraTeams/{session_id}, VeraResearch/{session_id}. The session ID is the Lambda log stream name, which means we can correlate audit logs with execution logs instantly. No more hunting through multiple log groups trying to piece together what happened.

Here’s where it gets interesting: Slack and Teams are completely different platforms with different user identity models.

Slack is straightforward

When a user messages the bot, we get a Slack event that includes their user ID. We already call the Slack API’s users.info endpoint to fetch the user’s profile for conversation building. That API returns a username like kyler.middleton. We just reuse that data for audit logging. No extra API calls, no complexity.

Teams is … different

Honestly, I would have been surprised if Teams made this easy. Read through the whole Teams Vera series to learn all the awful patterns I had to build to make it work. This one isn’t as terrible as all that, but still not great.

When a user messages the bot through Microsoft Teams, we get a Bot Framework event with an Azure AD Object ID—basically a GUID like d4bxxxx-xxxx-xxxx-xxxx-xxxxxxxxcbb. We also get a display name like “Kyler Middleton,” but that’s not useful for audit correlation. Display names aren’t unique and they change.

To get a consistent username from Teams, we need to call the Microsoft Graph API with that Azure AD Object ID. The Graph API returns the User Principal Name, which is typically an email address: Kyler.Middleton@veradigm.me. That’s better, but still not consistent with Slack’s format.

So we normalize it: split on the @ symbol, take everything before it, convert to lowercase. Kyler.Middleton@veradigm.me becomes kyler.middleton. Now it matches Slack’s username format, and we can search audit logs across platforms using consistent identifiers.

The normalization matters more than you’d think. When security asks “show me everything Bob asked across all bots,” they shouldn’t need to know that Bob is bob.smith in Slack but Bob.Smith@veradigm.me in Teams. One username format, searchable everywhere.

This also means we’re making an extra API call for every Teams interaction—but we do it after we’ve already responded to the user, so there’s no latency impact on the conversation.

Lets Build It!

The infrastructure setup is straightforward. We grant the Lambda IAM role write permissions to the shared CloudWatch log group `/aws/ai-bots/audit-logs`. All three bot Lambdas can write to this one log group, but they use different log stream prefixes to keep their logs separate.

The core audit logging function is identical across all three bots. It extracts the AWS account ID from the Lambda context, builds a structured log entry, and writes to CloudWatch:

The function handles errors silently—if CloudWatch is having issues, we don’t want to fail the user’s request.

For Slack bots

We already have the user information from building the conversation. We extract the username and flatten the conversation history:

For Teams bots

We need the extra Graph API call to fetch the User Principal Name and normalize it:

But doesn’t that slow the response down?

To avoid introducing latency in the response, we log after responding to the user. The bot sends its response first, then writes the audit log. This keeps the user experience fast. If the Graph API is slow or CloudWatch has issues, the user never sees it.

Lessons Learned

The biggest lesson I took from this is that the platform’s (read: AWS Bedrock) logs were good enough for all our purposes. Even with great search tools like Splunk or Elastic/ELK, if the data isn’t in your logs, you can’t get to it.

So we had to build it ourselves.

Username normalization matters more than you’d expect. We could have just logged whatever each platform gave us, but then searching across platforms becomes painful.

Session correlation means it’s easier to dig in if there is more detail needed. The audit log contains the full cloudtrail log group name for us to click in and read through.

Log after the user response is posted so we introduce literally no latency in the user experience.

Audit error logging should be invisible to the user - just error log for admins.

AI compliance isn’t just about model outputs. It’s about understanding who’s asking what, what information we’re providing, and being able to trace any conversation from question to answer.

Summary

In this article we talked about how Bedrock doesn’t suite our needs for audit logging, and what we’d really like instead - one cloudtrail group, shipped to Splunk or ELK, that contains all the data we need (and only the data we need) in one place.

We linked each summary / audit log to the deep log group trail we need if we want to dig further

And we shared all the code to show you how to do it!!

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In this blog post, we’re going to integrate the AWS Labs CLI MCP server with a Strands-powered AI agent running in Lambda, with support for querying resources across multiple AWS accounts using profile-based role assumption.

The specific challenge: we need our AI assistant to query AWS resources across many different accounts (dev, staging, prod, ops, etc.)… with an MCP server that appears built to support a single authentication profile, in a single account.

The Multi-Account Challenge

When you ask an AI agent “List the EKS clusters in the Prod account,” the agent needs to:

1. Determine which AWS account “Prod” refers to

2. Assume a role in that account with read permissions

3. Execute the AWS API call with the assumed credentials

4. Return the results

The obvious approach is to use `aws sts assume-role` to get temporary credentials, then use those credentials in subsequent commands. This doesn’t work with the AWS CLI MCP server because it’s stateless - each command execution creates a fresh boto3 session. The MCP server can’t “remember” credentials from a previous assume-role call.

Bummer

Another approach would be creating a separate MCP client instance, one per account we might need to speak to, each configured with a different AWS profile. This creates tool sprawl and of course doesn’t scale well.

Double bummer

What we need is profile-based role assumption where the bot includes `--profile oak` in its AWS CLI commands, and the profile configuration automatically handles assuming the correct role using the Lambda execution role as the base credential source.

Why This Is Complicated

The AWS CLI MCP server uses boto3, not shell commands. When you pass `aws eks list-clusters --profile oak`, the MCP server:

1. Parses the command string to extract the profile name

2. Creates a boto3 Session with that profile

3. Loads profile configuration from the AWS config file

4. Resolves credentials according to the profile’s settings

5. Executes the API call

Each of these steps has failure modes that produce silent errors - boto3 just fails without helpful error messages. We’ll cover the debugging later.

The key insight: the `--profile` flag does work in the AWS CLI MCP server. It extracts the profile from the parsed command and passes it to `boto3.Session(profile_name=...)`. We just need to configure profiles correctly.

Lets do it!

Understanding How the AWS CLI MCP Works

The first thing to understand is that the AWS CLI MCP server doesn’t shell out to the AWS CLI binary. It uses boto3 directly. This is important because it means we need to think about boto3 session management, not shell command execution.

When you send `aws eks list-clusters --profile oak` to the MCP server, heres what actually happens under the hood:

The server parses the command string and extracts the profile name from the `--profile` flag. I actually went digging through the source code to confirm this because the documentation wasn’t clear. In the parser.py file, theres code that does:

profile = getattr(global_args, ‘profile’, None)

Then in driver.py, when creating credentials:

credentials = credentials or get_local_credentials(
  profile=translation.command.profile or AWS_API_MCP_PROFILE_NAME
)

This means the `--profile` flag IS recognized and used. The server creates a fresh boto3 session for each command using that profile name.

Great, so profiles work

Now we need to understand where boto3 gets profile information from.

AWS Config File vs Credentials File

Boto3 looks in two places for configuration:

1. `~/.aws/credentials` - Contains access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)

2. `~/.aws/config` - Contains profile settings like region, role_arn, and credential sources

By default, boto3 only reads the credentials file. To make it read the config file, you need to set the environment variable `AWS_SDK_LOAD_CONFIG=1`.

This is critical and not obvious

Our profiles with role assumption settings live in the config file, so without this environment variable, boto3 cant see them.

Lambda Credentials and credential_source

When your Lambda function runs, AWS provides temporary credentials via environment variables:

- AWS_ACCESS_KEY_ID

- AWS_SECRET_ACCESS_KEY

- AWS_SESSION_TOKEN

These credentials belong to the Lambda execution role. In our case, thats the role `Ue1LiVeraResearchTestWorkerRole`.

In the AWS config file, you can tell a profile to use credentials from the environment with:

[profile oak]
role_arn = arn:aws:iam::123456789:role/read-only-role
credential_source = Environment

This tells boto3: “Get the base credentials from environment variables, then use those to assume the role specified in role_arn.”

The Two-Step Credential Flow

Heres the complete flow:

1. Lambda execution role provides credentials via environment variables

2. Bot sends: `aws eks list-clusters --profile oak`

3. MCP server extracts `--profile oak` from the command

4. Boto3 creates a session with `profile_name=’prod’`

5. Boto3 reads `~/.aws/config` and finds `[profile oak]`

6. Sees `credential_source = Environment` and loads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN from environment

7. Uses those credentials to call `sts:AssumeRole` for the role_arn

8. Gets back temporary credentials for the assumed role

9. Executes `eks:ListClusters` with the assumed role credentials

The key insight here is that credential_source = Environment lets us chain roles: Lambda role → assumed role.

What About source_profile?

You might think you could do this instead:

[profile default]
credential_source = Environment
[profile oak]
role_arn = arn:aws:iam::123456789:role/read-only-role
source_profile = default

This doesn’t work. When a profile has `credential_source = Environment`, it cant be used as a source_profile. AWS SDK documentation mentions this but its easy to miss.

I spent hours debugging this

Each profile needs its own `credential_source = Environment` setting directly. No chaining through a default profile.

Now that we understand how boto3 resolves profiles and credentials, lets build the actual implementation.

Building the Solution

We need to configure everything so that when the bot says `aws eks list-clusters --profile oak`, it automatically assumes the correct role in the Oak account and returns results.

Heres what we need to build:

1. An AWS config file with profiles for each account

2. A Dockerfile that pre-installs the MCP server and copies the config

3. A Python worker module that sets up the MCP client with correct environment variables

4. IAM permissions on both sides of the trust relationship

Lets go through each piece.

The AWS Config File

Create a file called `aws_config` in your Lambda project:

# AWS Config for Multi-Account Access

[profile oak]
role_arn = arn:aws:iam::111111111111:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

[profile birch]
role_arn = arn:aws:iam::222222222222:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

[profile palm]
role_arn = arn:aws:iam::333333333333:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

Key points:

- Each profile is self-contained with `credential_source = Environment`

- No `[profile default]` needed

- `role_arn` points to the read-only role in each target account

- `role_session_name` can be anything - its useful for CloudTrail auditing

- Replace the account IDs with your actual account IDs

The Dockerfile

We need to pre-install the AWS CLI MCP server and copy our config file:

# Pre-install AWS CLI MCP server

RUN mkdir -p /opt/aws-cli-mcp-server && \
 cd /opt/aws-cli-mcp-server && \
 uv venv && \
 uv pip install --python .venv/bin/python awslabs.aws-api-mcp-server && \
 chmod -R a+rX /opt/aws-cli-mcp-server

# Copy AWS CLI config with pre-configured profiles
COPY aws_config /opt/aws_config

Why `/opt` instead of `/tmp`? Lambda clears `/tmp` on every cold start, but I can read from `/opt`. The worker will copy it to `/tmp` at runtime since boto3 needs write access to the directory.

Also note: I tried copying the entire venv to `/tmp` and ran out of disk space. Lambda gives you 512MB in `/tmp` by default and the MCP server venv is huge. Running from `/opt` works fine.

The Worker Module

Create `worker_mcp_aws_cli.py`:

Critical parts:

1. Copy config to /tmp - boto3 needs the config file in a writable location

2. AWS_SDK_LOAD_CONFIG=1 - Without this, boto3 ignores the config file entirely

3. Pass through credentials - The subprocess doesn’t inherit environment automatically when you provide a custom env dict. You must explicitly pass AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN

This one got me for like 2 hours. Most lambda processes automatically inherit the IAM role assigned to the lambda. Not this tool, BIG SIGH

I initially didn’t pass through the credentials and boto3 just silently failed with no helpful error messages.

Integrating with Strands

In your main agent setup (like `worker_agent.py`), integrate the MCP client:

from worker_mcp_aws_cli import build_aws_cli_mcp_client
# Build AWS CLI MCP client
aws_cli_mcp_client, aws_cli_tools = build_aws_cli_mcp_client(aws_region=”us-east-1”)
# Prefix tool names to avoid collisions
aws_cli_tools = add_prefix_to_mcp_tools(aws_cli_tools, “aws”)
# Add to agent tools
tools.extend(aws_cli_tools)

The tool prefixing gives you `aws_call_aws` and `aws_suggest_aws_commands` instead of just `call_aws`. This prevents collisions if you have multiple MCP servers.

IAM Permissions - Both Sides Must Agree

For cross-account role assumption to work, you need permissions configured on both sides.

So the IAM role you assign to the lambda must permit assuming an IAM role, and the IAM role must permit you to assume it. If either is wrong, the assuming doesn’t work.

This is hard enough to debug in a regular application, but a HUGE pain when your AI eats errors and doesn’t print them

Testing It

At this point, you should be able to deploy your Lambda and test:

Bot receives: “List EKS clusters in the Oak account”

Bot constructs: `aws eks list-clusters --region us-east-1 --profile oak`

MCP server:

1. Parses `--profile oak`

2. Loads `/tmp/.aws/config`

3. Finds `[profile oak]` with `credential_source = Environment`

4. Gets Lambda credentials from environment variables

5. Calls `sts:AssumeRole` for `arn:aws:iam::111111111111:role/read-only-role`

6. Gets temporary credentials

7. Executes `eks:ListClusters` with assumed credentials

8. Returns results

Beautiful

Except when it doesn’t work. Lets talk about debugging.

The Debugging Journey

When I first deployed this, nothing worked. The bot would try to list resources and just get generic “Unable to locate credentials” errors. Not helpful.

The problem with boto3 and the AWS SDK is that errors are often silent or misleading. A missing environment variable produces the same error as a malformed config file or insufficient IAM permissions.

Failure 1: boto3 Cant Find the Config File

First test: bot tries `aws eks list-clusters --profile oak`.

Error: `Unable to locate credentials. You can configure credentials by running “aws configure”.`

This error is lying. The credentials exist (Lambda execution role). The real problem: boto3 isn’t reading the config file at all.

I added debug logging to print the contents of `/tmp/.aws/config` right after copying it. The file was there. The profiles were correct. boto3 just wasnt looking at it.

The fix: `AWS_SDK_LOAD_CONFIG=1`

Without this environment variable, boto3 ONLY reads `~/.aws/credentials` and completely ignores `~/.aws/config`. Since our profiles live in the config file, boto3 had no idea they existed.

After adding that one line to the env dict, profiles started working.

One down

Failure 2: Subprocess Cant See Lambda Credentials

Second test after adding `AWS_SDK_LOAD_CONFIG=1`: same error.

Wait, what? I just fixed this.

Turns out when you pass a custom `env` dictionary to `StdioServerParameters`, Python doesn’t merge it with the parent process environment. It REPLACES the environment entirely.

The MCP server subprocess was starting with ONLY the variables I explicitly provided:

- HOME

- AWS_REGION

- AWS_CONFIG_FILE

- AWS_SDK_LOAD_CONFIG

- AWS_API_MCP_WORKING_DIR

Missing: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN

Lambda provides these credentials to the parent process, but the subprocess couldnt see them. So when boto3 tried to read `credential_source = Environment`, there were no credentials in the environment.

The fix: explicitly pass through the credentials

env = {
# ... other variables ...
“AWS_ACCESS_KEY_ID”: os.environ.get(”AWS_ACCESS_KEY_ID”, “”),
“AWS_SECRET_ACCESS_KEY”: os.environ.get(”AWS_SECRET_ACCESS_KEY”, “”),
“AWS_SESSION_TOKEN”: os.environ.get(”AWS_SESSION_TOKEN”, “”),
}

This grabs the credentials from the parent Lambda environment and explicitly passes them to the MCP server subprocess.

After this change, role assumption started working.

Two down

Debugging Tips

If youre building something similar, these techniques helped me:

1. Add debug logging everywhere - Print environment variables (redact sensitive values), print file contents, print boto3 session configuration

2. Test incrementally - I tested after each change: config file copied? Environment variable set? Credentials passed through?

3. Read the source code - The AWS CLI MCP server is open source. When documentation was unclear, I read driver.py and parser.py directly

4. Use CloudTrail - Failed AssumeRole attempts show up in CloudTrail in the target account with detailed error messages

The key lesson: boto3 fails silently. You need to instrument your code with logging to understand whats actually happening.

Summary

So thats it. We now have a single AI agent running in Lambda that can query resources across any number of AWS accounts using profile-based role assumption.

The AWS CLI MCP server does support --profile flags, even though its not well documented. You just need to configure everything correctly for boto3 to work.

The full code is in my GitHub/KyMidd/SlackStrandsAgenticBot repo if you want to see the complete implementation.

Now my bot can answer “How many EKS clusters are in the Oak account?” and actually get the answer. Pretty cool.

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

I’ve been building out my Slack Strands Agentic bot, a bot that comes with built-in MCPs that could connect to:

• PagerDuty

• Confluence / Jira

• GitHub

As well as read our Bedrock knowledge base. I want the bot to be able to read the real status of resources, like in AWS or Azure.

When I added the Azure MCP, I worked through the bugs, and when I got it right… it crashed.

ERROR: Tool name ‘search’ already exists

MCP (Model Context Protocol) is how AI agents connect to external tools. It’s basically an API, but instead of you calling endpoints, the agent does. You give it access to a GitHub MCP server, and suddenly it can search code, open issues, list repos. Same with Atlassian - it gets tools for Confluence and Jira. Add PagerDuty and Azure, and you’ve got a pretty capable agent.

Except when two servers both provide a tool called search.

Apparently when you import tools to your bots, if the tool names are identical, it crashes the bot (at least for the Strands library).

Atlassian MCP has search for finding Confluence pages, and Azure MCP has search for Azure resources. Both correct, and both needed, but both have the same name.

It’s apparently not like variable scoping where context matters. This is a flat namespace.

I debugged the tools that are registering, you can see them below.

Both Atlassian and the new Azure MCP try to register a tool called “search”, which isn’t permitted.

When you connect multiple independent MCP servers to the same agent, eventually their tool names will overlap. The more servers you add, the more likely it becomes.

I don’t want to prune back those tools, I want all of them! Which meant I needed to fix the collision without modifying the MCP servers themselves (they’re first-party remote and hosted by the companies) and without breaking how the Strands SDK registers tools.

Cool! Lets do this.

The Solution: Wrap It, Prefix It, Ship It

How most tools handle this is namespaces. If a tool is imported from Azure called “search”, call it “azure_search”. That way collisions should be much more rare. And that’s what I built.

I expected I could just pass an argument to the Strands agent when I give it tools, and it’ll ingest the tool names with a prefix.

I’m working on a PR for Strands to do this.

We need a wrapper! Something that sits between the MCP tool and the Strands registry, changes the name, but otherwise acts exactly like the original tool.

Here’s what I built:

The `PrefixedMCPTool` class does three things:

1. Inherits from `AgentTool`: This is critical. The Strands SDK registry checks `isinstance(tool, AgentTool)`. If your wrapper doesn’t inherit from `AgentTool`, the registry rejects it. No exceptions.

2. Modifies the tool name: Takes the original tool’s spec, copies it, and changes just the name. Everything else stays the same - description, parameters, all of it. Just the name gets the prefix.

3. Delegates everything else: All the actual work gets passed to the original tool. The wrapper is just changing the name on the way in.

The key part is the `stream()` method:

def stream(self, tool_use, *args, **kwargs):
  return self._original_tool.stream(tool_use, *args, **kwargs)

That `*args, **kwargs` matters. The Strands framework passes extra arguments to `stream()` - stuff like `invocation_state` that the tool needs. If you don’t forward those arguments, your tools break at runtime (I learned this the hard way).

So now instead of registering a tool called `search`, you register a `PrefixedMCPTool` that wraps the original `search` tool but tells the registry its name is `atlassian_search`. The registry sees a unique name. The tool still works exactly the same way.

This ended up working SUPER WELL and I’m going to do this in ever bot I build going forward (unless the MCP itself already prefixes the platform ahead of the tool name, which they obviously should, but most don’t today).

## Making It Reusable

I didn’t want to wrap every single tool manually, or define this same function in every MCP ingestion script (which are separate files for each MCP due to filtering, authentication, and utilization differences). I needed a single helper function that could take all the tools from an MCP server and prefix them in one shot.

Here’s what I built:

The `add_prefix_to_mcp_tools()` function takes two arguments:

- `tools`: The list of tools from your MCP server

- `prefix`: The prefix you want to add (like “github” or “azure”)

It automatically adds the underscore, so you just pass `”github”` and it creates `github_search`, `github_get_issue`, etc.

Do this for each MCP server with a different prefix:

atlassian_tools = add_prefix_to_mcp_tools(atlassian_tools, “atlassian”)
github_tools = add_prefix_to_mcp_tools(azure_tools, “github”)
pagerduty_tools = add_prefix_to_mcp_tools(pagerduty_tools, “pagerduty”)

Before prefixing, I had:

- Atlassian: `search`, `fetch`

- Azure: `search`

After prefixing:

- Atlassian: `atlassian_search`, `atlassian_fetch`

- Azure: `azure_search`

- No collision, bot works

The agent now has access to 106 tools total (25 from GitHub, 19 from Atlassian, 15 from PagerDuty, 47 from Azure), and every single one has a unique name, and all are accessible at runtime.

With my luck, AWS (my next target) will also have a “search” tool, but now I won’t really care, we won’t see the collision even if it does.

Summary

If you’re using multiple MCP servers with your AI agents, you’ll hit tool name collisions eventually. It’d be amazing if some registry came out that helped folks establish a standard naming scheme, validated partners and sources, much like the GitHub Marketplace, or Azure Marketplace, Hashi Marketplace etc., but it doesn’t exist yet.

I’m excited to no longer care what tools we register - now I can just add MCPs and their tools, and go back to worrying about AIs deleting my environment, or taking over the world.

Thanks ya’ll. Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all! Today we’re tackling a problem that’s been driving me crazy since we started scaling our AWS Bedrock bot deployments across the enterprise. You’d think that when you have multiple AI bots running in production, AWS would give you some reasonable way to separate and monitor their activities. You’d be wrong.

Here’s the issue: every single AWS Bedrock model invocation in a region gets dumped into one massive CloudWatch log group. Doesn’t matter if you have five different bots serving different teams, or if Bot A is handling customer support while Bot B is doing internal documentation queries. Everything goes into `/aws/bedrock/modelinvocations` and good luck figuring out which log entry came from which bot.

This creates a monitoring nightmare. When someone asks “how much is our Slack bot costing us?” or “why did the support bot give a weird response yesterday?” you’re stuck grep-ing through thousands of log entries that all look identical. The raw Bedrock logs contain the model inputs and outputs, but zero context about which application made the call or who the actual user was.

After dealing with this mess for way too long, I finally built a solution that transforms these useless raw logs into something actually meaningful for enterprise monitoring. We’re talking about a Kinesis Firehose pipeline with a custom Lambda processor that parses the Bedrock JSON and adds the context that should have been there in the first place.

The enhancement adds four key fields that make all the difference: `botName` (extracted from the IAM role), `botOutput` (what the bot actually said to the user), `user` (who asked the question), and `query` (what they asked). Suddenly, instead of cryptic JSON blobs, you have logs that tell a story about real conversations happening in your enterprise.

This approach works for any AWS Bedrock deployment, but it’s especially valuable if you’re running multiple bots or need to track usage by team, project, or individual users. The processed logs flow into whatever monitoring system you’re already using - in our case, Splunk - where they become the foundation for dashboards, alerting, and cost attribution.

Let’s walk through how this whole system works and why it’s become essential for managing AI deployments at enterprise scale.

If you don’t care about the write-up and just want the code, you can find the modified cloudwatch to splunk lambda that adds bot information here:

gist.github.com/KyMidd/f2d216682b0c40ca06d74742f1cc56f7

The Problem: AWS Bedrock’s Logging Black Hole

To understand why this solution matters, you need to see just how broken the default AWS Bedrock logging situation really is. When you make a model invocation through Bedrock, AWS dutifully logs the entire interaction to CloudWatch - which sounds great until you realize where those logs actually end up.

Every single model invocation in a region gets dumped into `/aws/bedrock/modelinvocations`. Your customer support bot? Same log group. Your internal documentation bot? Same log group. That experimental bot someone in marketing is testing? Yep, same log group. It’s like throwing all your mail into one giant pile and then wondering why you can’t find anything.

AWS does include some metadata with each log entry - most importantly, the IAM role ARN that made the request. So technically, you can tell which bot generated which log entry. But here’s the catch: you have to parse through thousands of nearly identical JSON blobs to find the entries you care about, and even then, the raw logs don’t tell you anything useful about the actual conversation.

Here’s what a typical raw Bedrock log entry looks like - it’s hundreds of lines of nested JSON containing the full model request and response, but buried so deep in structure that extracting meaningful information requires serious JSON archaeology. You can see the complete input and output, but good luck figuring out which user asked what question or what the bot actually said back to them.

The enterprise implications are brutal. When your CFO asks “how much are we spending on the Slack bot versus the customer support bot?” you can’t answer without building custom tooling to parse through the log mess. When a user reports “the bot gave me a weird answer yesterday,” you’re looking at potentially thousands of log entries with no easy way to find their specific interaction.

Even basic monitoring becomes a nightmare. Want to set up alerts for when a specific bot starts giving errors? You’ll need to craft complex CloudWatch filters that parse the IAM role ARN and hope you can distinguish between different types of failures. Want to track usage patterns or popular queries? Hope you enjoy writing custom log analysis scripts.

The root issue is that AWS designed Bedrock logging for debugging individual model calls, not for managing multiple production AI applications. They give you all the technical details about the model invocation but none of the business context about who asked what and why. For a single-bot deployment, this might be manageable. For enterprise deployments with multiple bots serving different teams and use cases, it’s completely unworkable.

The Solution: Enhanced Log Processing Pipeline

The answer to this mess is surprisingly straightforward: transform them on their way to your monitoring system.

Here’s the architecture: CloudWatch subscription filter → Kinesis Firehose → Lambda processor → Splunk. The raw Bedrock logs stay exactly as AWS created them in CloudWatch. But when they flow to Splunk through our pipeline, they get enhanced with the context that actually matters.

The Lambda processor is where the magic happens. It takes those massive, cryptic JSON blobs and transforms them into something humans can actually understand. Instead of hundreds of lines of nested structures arriving in Splunk, you get clean log entries with the four fields that matter: `botName` (which bot generated this), `botOutput` (what it actually said), `user` (who asked the question), and `query` (what they asked).

The processor extracts the bot name from the IAM role ARN, navigates through the complex nested JSON to find the actual response text, and parses conversation formats to identify users and their questions. What used to require custom scripts and manual investigation now happens automatically for every single log entry flowing into your monitoring system.

The beauty of this approach is that it’s completely non-intrusive. Your CloudWatch logs remain untouched for AWS debugging purposes, but your enterprise monitoring gets the enriched context it needs. The whole pipeline uses standard AWS services, scales automatically, and integrates seamlessly with whatever log aggregation platform you’re already using.

Log Enhancement

The real work happens inside the Lambda processor, where we parse the Bedrock JSON and extract the four fields that actually matter for enterprise monitoring.

Bot Name Extraction

This one’s straightforward. Every Bedrock log includes the IAM role ARN that made the request. We just grab the last segment after the final slash - so `arn:aws:iam::123456789012:role/VeraSlackBot-Worker` becomes `VeraSlackBot-Worker`. Simple, but effective for distinguishing between different bots.

Bot Output Parsing

This is where things get messy. The actual response text is buried deep in the JSON structure at `output.outputBodyJson.output.message.content[0].text`. That’s five levels of nesting just to find out what the bot said to the user.

User and Query Extraction

The most complex part involves parsing the conversation format from the input messages. Our bots format user messages as “Kyler (she/her) says: Now a dog.” - so we need to split on “ says: “ and extract the username before any parentheses. The query is everything after the colon.

The code handles all the edge cases gracefully. If any field can’t be extracted, we just skip it rather than failing the entire log processing. This keeps the pipeline resilient when AWS changes log formats or when we encounter unexpected message structures.

What emerges from this parsing is a clean, searchable log entry that tells the complete story: which bot responded to which user’s query with what output. Instead of archaeological JSON excavation, you get logs that humans can actually read and monitoring systems can actually use.

Terraform Infrastructure: Making It Work

Setting up this pipeline requires several Terraform resources working together. The core piece is the Kinesis Firehose delivery stream that connects everything.

CloudWatch Subscription Filter

First, you need a subscription filter on the Bedrock log group that forwards all log events to your Firehose stream. This is the entry point that captures every Bedrock model invocation as it gets logged.

IAM Roles and Permissions

The pipeline requires two main IAM roles. One for CloudWatch to write to Firehose, and another for Firehose to invoke your Lambda processor and write to S3 for failed events. Standard AWS service-to-service permissions, but they need to be configured correctly or nothing works.

Lambda Function

Your log processing Lambda function needs the right runtime (Python 3.13), appropriate timeout settings, and environment variables for any custom configuration like Splunk source types.

Kinesis Firehose Configuration

The Firehose configuration is where everything comes together. You specify your Splunk endpoint, configure the Lambda processor, and set up S3 backup for failed events. The processing configuration is key - that’s what tells Firehose to run your custom Lambda function on every log batch before sending to Splunk.

Error Handling

Don’t forget S3 backup for failed events. When something goes wrong with log processing or Splunk delivery, you want those logs going somewhere you can investigate later rather than disappearing into the void.

Real-World Impact and Benefits

The difference this makes in day-to-day operations is dramatic.

Individual Bot Monitoring

Suddenly, you can answer basic questions about your AI deployments. “How much is the customer support bot costing us?” becomes a simple Splunk query filtering by `botName`. “Which bot is getting the most usage?” is just a matter of counting log entries by bot. No more spreadsheet gymnastics or manual log parsing.

Troubleshooting User Interactions

When someone reports “the bot gave me a weird answer,” you can actually find their interaction. Search for their username and the approximate time, and there’s the complete conversation - what they asked, what the bot responded with, and all the underlying model invocation details. This turns impossible debugging sessions into five-minute investigations.

Cost Attribution and Chargeback

Enterprise teams love being able to track costs by department or project. With botName extraction, you can build dashboards that show exactly how much each team is spending on AI. Finance gets happy, teams get visibility into their usage, and everyone makes better decisions about resource allocation.

Performance Insights

The enriched logs reveal usage patterns you never knew existed. Which types of queries take the longest to process? What time of day sees the most bot activity? Are certain users asking the same questions repeatedly (hint: maybe your documentation needs work)?

Integration with Existing Monitoring

Because the enhanced logs flow into whatever monitoring system you’re already using, all your existing dashboards, alerts, and processes just work. Set up alerts when a specific bot starts throwing errors. Create dashboards showing conversation trends across different teams. Build reports on AI adoption across the organization.

Conversation Analytics

Having the actual user queries and bot responses in searchable logs opens up possibilities for conversation analytics. You can identify common question patterns, track how bot responses change over time, and even spot opportunities for better training data or system improvements.

The end result is that AWS Bedrock goes from being a black box that costs money to being a transparent, manageable enterprise service that you can actually operate with confidence.

Summary

AWS Bedrock dumps all model invocation logs into a single regional CloudWatch log group, making enterprise monitoring nearly impossible. This creates chaos when trying to track costs, troubleshoot issues, or monitor multiple bots.

The solution: a Kinesis Firehose pipeline with a custom Lambda processor that intercepts logs flowing to Splunk and adds essential context. The processor extracts botName (from IAM role), botOutput (actual response), user (who asked), and query (what they asked) from the raw JSON.

This transforms cryptic Bedrock logs into actionable monitoring data, enabling cost attribution, user interaction troubleshooting, and conversation analytics for enterprise AI deployments.

Thanks all!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

Today we’re going to explore something pretty exciting in the world of enterprise AI applications. I recently got my hands on an evolution of the Vera genAI solution that I’ve covered extensively before - and this new version represents a significant architectural leap forward.

If you’ve been following my previous posts about Vera, you’ll remember it was built with Python directly without many external libraries, focusing on raw AWS Bedrock integration. This new implementation takes that foundation and transforms it into a proper agentic AI system using the Strands framework. Instead of just being a conversational AI, it can now take actions and use tools across your entire enterprise toolchain.

What makes this implementation particularly interesting is how it solves those everyday enterprise pain points we all know too well. You know that juggling act between Slack, PagerDuty, GitHub, and Jira just to get context on a single issue? This bot brings all those systems together in one conversational interface, powered by AWS Bedrock with Claude Sonnet 4 as the backend.

Here’s where the agentic capabilities really shine. The bot uses Model Context Protocol (MCP) to actually connect to your existing platforms. When someone asks “what incidents are assigned to me?” it goes and queries PagerDuty directly through AWS Bedrock’s processing. When they want to know about pull request status, it hits the GitHub API. The AI becomes a conversational interface to your entire toolchain, all orchestrated through AWS Bedrock’s model capabilities.

The system includes AWS Bedrock Guardrails for content filtering and even supports retrieval-augmented generation through Bedrock Knowledge Bases. It’s engineered for enterprise use with proper secret management, brand voice guidelines, and all the infrastructure automation you’d expect in a production environment.

In this article, we’ll walk through the complete architecture, examine how the Strands agentic framework transforms the original Vera approach, and look at how AWS Bedrock powers the multi-service integrations. Whether you’re thinking about building your own AI assistant or just curious about modern agentic architectures, this implementation has some great lessons to offer.

Let’s jump in and see how this all fits together!

If you want to skip the write-up and just read the code, you can find that here:

github.com/KyMidd/SlackStrandsAgenticBot

Containerizing the Worker Lambda: Why Docker Over Layers

When I started building this enhanced version of Vera, I quickly ran into a problem that anyone working with modern Python AI frameworks knows all too well: dependency hell.

The original Vera was relatively lightweight - just AWS SDK calls and some basic Slack integration. But this agentic version? We’re talking about the Strands framework, multiple MCP clients for different services, and a whole ecosystem of dependencies that each service brings along. The requirements.txt file exploded from a handful of packages to something that looked more like a small novel.

Lambda layers seemed like the obvious choice at first. After all, that’s what they’re designed for - sharing common dependencies across functions. But here’s where things got messy: we’re running on ARM64 for cost savings, and compiling some of these Python packages for ARM64 is a nightmare. Half the dependencies don’t have ARM64 wheels available, which means building from source. And building from source on ARM64? Good luck with that.

I spent way too much time trying to get a clean layer build working before I admitted defeat and switched to containers. Suddenly, all those dependency headaches disappeared. The AWS Lambda Python runtime base image handles the ARM64 compilation complexity, and I can use modern Python package managers like `uv` to handle the dependency resolution efficiently.

The container approach gives us a few key advantages. First, we can pre-install and configure the MCP servers during the build process. Notice how we’re copying the PagerDuty MCP server into `/opt/` and running `uv sync` to install its dependencies. This happens at build time, not runtime.

Second, we get much more predictable dependency resolution. The `uv` package manager handles all the complex dependency trees and gives us reproducible builds. No more “it works on my machine” issues when deploying to Lambda.

Third, the container image becomes our deployment artifact. Terraform can build, tag, and push the image to ECR, then update the Lambda function to use the new image - all in one deployment cycle. No more managing separate layer deployments or worrying about version mismatches.

The performance impact has been minimal. Cold starts are slightly longer due to the larger image size, but the warm execution performance is actually better because all dependencies are pre-resolved and ready to go. For a Slack bot that processes messages intermittently, this trade-off makes perfect sense.

Code Organization: Breaking Up the Monolith

One thing I learned from the original Vera implementation was that shoving everything into a single Python file becomes a maintenance nightmare pretty quickly. The first version worked fine when it was just basic AWS Bedrock calls and Slack responses, but as soon as I started adding MCP integrations and agentic capabilities, that single file ballooned to over 800 lines.

This time around, I split things up from the start. The worker Lambda is now organized into focused modules, each handling a specific aspect of the bot’s functionality:

• worker.py - Main Lambda handler and Slack app registration

• worker_inputs.py - Configuration, constants, and system prompts

• worker_agent.py - Strands agent setup and execution

• worker_aws.py - AWS service integrations (Bedrock, Secrets Manager)

• worker_slack.py - Slack-specific utilities and response formatting

• worker_conversation.py - Message processing and conversation flow

• worker_lambda.py - Lambda-specific utilities and event handling

• worker_mcp_*.py - Individual MCP client implementations

The import structure shows how clean this separation becomes. Each module has a single responsibility, which makes testing and debugging so much easier. When something goes wrong with PagerDuty integration, I know exactly where to look. When I need to adjust the system prompt or brand voice, it’s all in `worker_inputs.py`.

What’s particularly nice about this approach is how it handles the MCP integrations. Each service gets its own module (`worker_mcp_pagerduty.py`, `worker_mcp_github.py`, etc.) that encapsulates all the client setup, authentication, and tool registration logic. The main agent module just imports what it needs and doesn’t have to worry about the implementation details.

This modular structure also makes the container build more efficient. Docker can cache layers better when dependencies are clearly separated, and the overall codebase becomes much more approachable for anyone who needs to maintain or extend it later.

MCP Protocol Integrations: Connecting External Tools

Here’s where things get really interesting. The Model Context Protocol is what transforms this from just another chatbot into an actual agentic system that can take actions across your toolchain. Instead of the AI just talking about your GitHub issues or PagerDuty incidents, it can actually query them directly and give you real-time information.

The beauty of MCP is that it provides a standardized way for AI models to interact with external systems. Each service exposes its capabilities as “tools” that the AI can discover and use dynamically. When someone asks “what incidents are assigned to me?” the bot doesn’t just guess - it calls the PagerDuty API through the MCP interface and gets the actual data.

I filter all tools in their own file, so I can only return tools to the agent that can do READ behavior (non-modify). AIs are way too scary to give them write tools, IMO. You can say read-only = False if you want all the Write tools also.

PagerDuty Integration

The PagerDuty integration handles incident management and on-call scheduling. The MCP server runs as a pre-installed component in the container, which means zero cold-start overhead when the bot needs to query PagerDuty.

The setup is straightforward - we pass the API credentials through environment variables and let the MCP client discover what tools are available. The PagerDuty MCP server typically exposes tools for listing incidents, getting on-call schedules, and querying service status.

GitHub Integration

GitHub integration is particularly useful for development teams. The bot can pull pull request status, check repository information, and even search across codebases. I’ve implemented it with read-only access by default because, let’s be honest, you probably don’t want your Slack bot accidentally merging pull requests.

The filtering logic here is important. By restricting to read-only operations, we get all the query capabilities without the risk of destructive actions. The bot can tell you about issues, pull requests, and repository status, but it can’t modify anything.

Atlassian Integration

The Atlassian integration covers both Jira and Confluence, which makes it incredibly powerful for teams that live in the Atlassian ecosystem. The OAuth setup is a bit more complex than the other integrations, but once it’s configured, the bot becomes a conversational interface to your entire knowledge base.

What’s particularly clever about the Atlassian integration is how it handles both Jira tickets and Confluence documentation in a unified way. When someone asks about a project, the bot can pull information from both systems and present a complete picture.

The key insight with all these integrations is that they’re designed to fail gracefully. If a service is unavailable or credentials are missing, the bot continues to work - it just operates without those specific tools. This makes the system much more resilient in production environments.

Terraform Container Deployment: Building and Deploying Docker Images

One of the things I really appreciate about this architecture is how Terraform handles the entire container lifecycle. Unlike traditional Lambda deployments where you’re uploading zip files and hoping for the best, this approach treats the container image as the deployment artifact and manages everything from build to deployment in a single workflow.

The magic happens through a combination of ECR repository management and some clever null_resource provisioners that handle the Docker build and push operations. When you run `terraform apply`, it doesn’t just update your Lambda configuration - it actually builds a fresh container image, pushes it to ECR, and then updates the Lambda function to use that new image.

The ECR repository includes lifecycle policies that automatically clean up old images, which is important because container images can get pretty large. We keep the latest 10 tagged images and automatically delete untagged images after a day. This prevents your ECR costs from spiraling out of control as you iterate on the deployment.

The Docker build process is particularly clever here. The null_resource uses a trigger based on the source file hash, so it only rebuilds the image when your code actually changes. This saves a ton of time during development when you’re just tweaking Terraform configurations without touching the application code.

What’s really nice about this approach is how it handles the ARM64 build process. The `--platform linux/arm64` flag ensures we’re building for the correct architecture, and since we’re building on the same platform where Terraform runs, we avoid all those cross-compilation headaches that made Lambda layers so problematic.

The Lambda function configuration shows how seamlessly containers integrate with the rest of AWS Lambda. You still get all the same configuration options - memory size, timeout, environment variables - but now your entire dependency tree is packaged as a single, immutable artifact. No more worrying about layer compatibility or missing dependencies.

Environment variables are passed through cleanly, including the MCP feature flags that control which integrations are enabled. This means you can deploy the same container image to different environments with different capabilities just by changing the Terraform variables.

Lessons Learned and Next Steps

Building this enhanced version of Vera taught me quite a few things about modern agentic AI architectures, and honestly, some hard lessons about when to stick with what works versus when to embrace new approaches.

Container vs Layers: The Right Tool for the Job

The biggest lesson here was accepting that Lambda layers, while great for simple shared dependencies, just weren’t going to cut it for this complexity level. The original Vera worked fine with layers because it had a handful of lightweight dependencies. But once you’re dealing with the Strands framework, multiple MCP clients, and all their transitive dependencies, containers become the only sane choice.

The decision framework I’d use now: if your requirements.txt has more than 20 packages or includes any packages that need native compilation, just go straight to containers. The development experience is so much better, and the deployment reliability gains are worth the slightly larger cold start times.

Performance Considerations

Speaking of cold starts, they’re definitely longer with the container approach - usually 2-3 seconds instead of sub-second for the original Vera. But here’s the thing: for a Slack bot, this rarely matters. Most conversations happen in bursts, so the function stays warm. And when it does cold start, users are already expecting some delay for “AI thinking time.”

What surprised me was how much faster the warm execution became. All those dependencies are pre-resolved and ready to go, which makes the actual AI processing noticeably quicker. The MCP clients can establish connections faster, and there’s less Python import overhead.

Scaling Challenges and Solutions

The current architecture handles our team’s usage just fine, but I can see some potential bottlenecks as usage grows. Each Lambda invocation opens its own MCP client connections, which could become problematic if we’re hitting rate limits on external APIs.

The next evolution would probably involve connection pooling or even moving some of the MCP operations to separate services that can maintain persistent connections. But honestly, that’s a nice problem to have - it means people are actually using the bot.

Modular Architecture Payoffs

The decision to break everything into focused modules has paid dividends already. When GitHub changed their API authentication requirements, I only had to touch `worker_mcp_github.py`. When we wanted to adjust the system prompt for better Slack formatting, it was all in `worker_inputs.py`.

This modularity also makes testing much more manageable. You can unit test individual MCP integrations without spinning up the entire Lambda environment, which speeds up the development cycle considerably.

Future Enhancement Opportunities

There are several directions this could evolve. Adding more MCP integrations is the obvious one - imagine connecting to ServiceNow, Datadog, or your internal APIs. The framework makes this relatively straightforward now.

I’m also interested in exploring memory and context persistence. Right now, each conversation is stateless, but there’s real value in the bot remembering previous interactions or maintaining context across related conversations.

The Strands framework also supports more advanced agentic behaviors like multi-step planning and tool chaining. We’re barely scratching the surface of what’s possible when you give an AI system the ability to orchestrate multiple API calls to solve complex problems.

Summary

It’s been super fun to build out this new platform, and I’ve been able to re-use many of the pieces of the original genAI Vera, particularly the pieces that integrate with Slack.

The bot is quite a bit slower than genAI Vera, but can answer questions like this:

• “Show me critical incidents for my team” - pulls live PagerDuty data

• “Status of the auth PR?” - queries GitHub directly

• “Find our API rate limiting docs” - searches Confluence with links

• “What’s blocked this sprint?” - filtered Jira queries

And because of using MCP tools, the answers are less often hallucinated, and more often backed by information the tool fetches.

Instead of bouncing between Slack, PagerDuty, GitHub, and Jira for incident context, teams get comprehensive answers from one question. On-call engineers can query previous incidents, service status, and recent code changes without leaving their response channel.

Hopefully it can help you too. Good luck out there!
kyle

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

I’ve been playing around with Strands a great deal, and having a ton of fun. Strands Agents is a python library that permits writing agentic workloads with relatively little code, and supports all sorts of AI backends to provide the AI services, including Amazon Bedrock, OpenAI, Anthropic, and even local models on your computer.

Here’s one that fetches vectors from an AWS Bedrock Knowledge Base, feeds those vectors into an Agent (by default Claude 4), and streams an answer to the provided question - all in 4 lines! That’s absolutely absurd, it’s so cool.

from strands import Agent
from strands_tools import retrieve
agent = Agent(tools=[retrieve])
agent("Tell me about the cloud platform team")

One of the powers of Strands is it easily integrates with MCPs - those tool integrations that permit your agent to interact with the platforms your business uses to do stuff. Strands can use MCPs to look at:

• Your tickets in Jira? Yep!

• Your outage information from PagerDuty? Yep!

• Your code on GitHub? Well, sorta

Because “Resources” aren’t built out yet, Strands agents can’t yet read the contents of files from GitHub.

Lets talk about what’s missing and how to fix it!

Wait, What are we talking about?

There’s a lot going on here, so it’s important to understand the players.

First, there are AI clients - these provide the AI services from a platform, for example AWS Bedrock or OpenAI. You generally get a token from these platforms, export the token into your terminal, and you can then use tools that trigger jobs on these platforms.

If you want to be fancy, you can say “requests for inference” instead of “jobs”. They have the same meaning.

Then there are MCP clients. These are the software components that can talk to MCP servers, potentially authenticate, and download the tools lists from the MCP server. Often this is your IDE like VSCode or Claude Code. Strands has an MCP client built in.

Then there are MCP servers. These can exist anywhere, but we’re seeing more and more of these as remote, first party MCP servers. That means the company who hosts the MCP server (remote to your client, like over the internet) is the place you’d use those tools. Think accessing a GitHub MCP from a GitHub URL in order to download tools to interact with GitHub’s codebase.

The platforms that store enterprise data have a great motivator to support MCP - all the enterprises that are charging into AI adoption can more easily consume those platforms if they can access an MCP for the platform. Early stage companies that are much more AI focused could easily use MCP availability as a primary differentiator when choosing which platforms to use.

The problem

So there’s a GitHub MCP! And it works great. When I first started working with it, I assumed it wasn’t completely implemented - it just entered General Availability (GA) Sept 4, 2025, and I was playing with it in Strands a few weeks before that.

It had this odd ability to read ALL the data from GitHub, but it couldn’t download file contents. Weird, but whatever.

However, when I tested the same GitHub MCP with VS Code IDE, it worked great.

So I tested it with Claude Code, worked great.

It turns out ONLY Strands has a problem downloading file content from GitHub when using MCP.

I dug into the problem with some logging and found this log:

2025-08-22 07:50:55,725 | INFO | Thread-1 (_background_task) | httpx | HTTP Request: POST https://api.githubcopilot.com/mcp/ "HTTP/1.1 200 OK"
# ...
2025-08-22 07:50:55,865 | DEBUG | MainThread | strands.tools.mcp.mcp_client | [Thread: MainThread, Session: 838664df-dafb-4a06-865e-91d618eb77f6] unhandled content type: EmbeddedResource - dropping content
2025-08-22 07:50:55,865 | DEBUG | MainThread | strands.tools.mcp.mcp_client | [Thread: MainThread, Session: 838664df-dafb-4a06-865e-91d618eb77f6] tool execution completed with status: success

There’s two interesting things going on here. When the Strands agent attempts to fetch file contents, it uses a tool called get_file_contents. This tool is just an MCP instruction to use “Get Repository Contents” API to fetch a file contents.

It reads the header to identify the type of file. If it starts with `application`, it decides that file is text/ascii-based. For example, a .tfvars file (terraform variables) is text based, and the header value gets:

< content-type: application/json; charset=utf-8

After deciding the content is text, the GitHub MCP (really the Get Repository Contents REST API call), returns a payload like this:

{
  "type": "resource",
  "resource": {
    "uri": "repo://orgName/repoName/contents/path/to/file.tfvars",
    "mimeType": "application/json; charset=utf-8",
    "text": "... (plain-text contents of the file) ..."
  }
}

The Strands library expects to see direct TextResourceContents (on .text property path) or BlobResourceContents (on .blob property path) like this:

{
  "text": "... (plain-text contents of the file) ..."
}

Since the GitHub MCP is adding extra information, the Strands library totally fails to interpret it, despite the path being super duper close, and the information being nearly the same.

How To Fix It

You can’t. Well, at least not easily. The problem is in how the Strands library is written - it doesn’t yet have support for EmbeddedResources, which is the whole “.resource” property path and syntax.

I’ve created a fixed version on this PR: https://github.com/strands-agents/sdk-python/pull/726

This fixes the behavior for many types of files that are text encoded from github, but doesn’t yet work for binary/blob type files. It does however work for these types of files:

"application/json",
"application/xml",
"application/javascript",
"application/x-yaml",
"application/yaml",
"application/xhtml+xml",

And you’d be surprised how many file types are covered with these!

Try it out: Copy my version (from the PR) of this file: src/strands/tools/mcp/mcp_client.py over your version, and watch how your Strands agent can now fetch file contents:

What’s Next?

The Strands maintainers are hoping to implement full support for Resources before implementing partial support, meaning that this functionality isn’t coming immediately. Implementing the full lifecycle support for Resources will take a bit longer.

I’ll continue asking to merge this in - I think being able to fetch file contents from GitHub is a huge functionality gap for Strands, and will be relatively blocking for us.

Potentially we’ll need to implement some static tools for our bots to use to fetch file contents from GitHub using REST. I

And I’ll be sure to write about it.

Thanks all! Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In previous articles we built generative AI bots that integrate with a RAG knowledge base using the AWS Bedrock backend.

• AWS GenAI Bedrock Series

• Azure GenAI Bedrock Series

They’re awesome, but they’re not agentic.

"Agentic" describes the ability to act autonomously, make decisions, and achieve goals without constant human intervention, a concept central to agentic AI systems that can plan, execute tasks, and adapt to new information

Basically, agentic bots do 1 step, and then return that information. That generally works really well for chatbots - read this information, return a summary. One step is all you need!

However, if you want a bot to do a multi-step process, that’s when you’re looking at an “agentic workflow”. Something like “Read our PagerDuty, isolate any resources with issues, then go look at AWS and see if the resources have an error state you can recommend fixes for” is multiple step - talk to platformA, analyze the data, then use that data to talk to platformB and analyze the data from there.

Agentic agents can do it!

Strands has emerged as a language that permits building effective and comprehensive bots with little python. It comes out of AWS, but the team there has made sure to open-source it (Apache 2.0 license) and make sure it’s compatible with lots of back-ends like OpenAI/ChatGPT, Anthropic, OLlama, and others.

It support agentic workflows, MCP (tool use to talk to platforms), A2A protocol (to talk to other bots and coordinate work), and will likely be kept up to date with AWS services in AgentCore like Memory and other cool stuff shaking out of that project.

It feels like a great place to start.

Strands, find all the ec2 instances in us-east-1, and give me their IPs and AZs in a table.

I’m going to be building some enterprise Strands agentic assistants over the next few months, but for now I’m just proving out really simple workflows, and I wanted to share them. They’re all here:

github.com/KyMidd/StrandsAgenticExamples

We’ll walk through one by one for how they work.

Simple Calculator

AI Models are terrible at math. They just are. If you ask a model a math problem that it’s seen before in its data set, it’ll give you the answer it saw. Is it the right answer? It doesn’t know.

I’m also bad at math, so I use a calculator. Best idea to make an AI good at math? Give it a calculator! And that’s one of the built-in tools available to strands.

On line 1, we import the Agent library from strands, and on line 2 we import from the strands_tools library, specifically the built-in tool called “calculator”.

Then on line 3 we instantiate an agent and provide it the tool.

Then on line 4 we ask the agent to calculate the square root of 125. The agent sees a problem that it can resolve with math, so it uses the calculator.

Here’s the output:

Simple Remote MCP - AWS Docs

Not all MCP servers require authentication - some provide tools you can use to do interesting stuff, like this remote MCP endpoint from AWS that lets you ask AWS itself about their documentation, and provide your bot the ability to read AWS documentation in its operation.

On lines 2-3, we import the MCPClient from the strands tools package, and on line 3 we import the standard i/o client and parameters.

On line 5, we define the aws_docs_client as a python lambda (small function), that’s calling the uvx tool to connect to the awslambda aws-documentation-mcp-server MCP. Notably, this doesn’t “run” the MCPClient yet, this just statically defines what will happen when it does.

On line 9, we define a “with”. This step actually runs the AWS Client config.

Then on line 10, we start the agent with the tools available from that MCP - note how we’re reading the tools list available from the remote MCP with list_tools_sync().

Then on line 11, we’re asking the agent a question.

Here’s how it looks, note how it reads documentation, then decides to read further documentation. This can build a comprehensive understanding of AWS technologies, it’s pretty cool.

Simple GitHub Loop

While still simple, I added a few features to the single-MCP scripts. First, we use functions to define the MCPs.

You can see on line 6 that we define a function to build the streamable remote MCP client, just like the AWS documentation MCP.

A bit lower down, we define the github MCP client, line 2.

And then on line 5, we connect to the MCP client on line 5 with the “with”.

On line 8, we establish the bedrock model, which customizes how the model works (number of tokens permitted, etc.), and build a list of tools we can pass to the agent, line 11-12.

Then on line 15 we build the agent and pass it the tools list.

Next, we start a “while” loop where we ask the agent our question that we defined above, to find all the Repos we own. The agent works agentically to get an answer, and then we hold on line 12, where the tool asks us if we want to continue.

We can type “exit” to break the loop and exit the script, or we can ask another question. Since the Agent is still instantiated, it remembers the previous question and information fetched from MCP, and can continue the conversation.

Here’s how it looks. I think it’s super cool that it can figure out who I am, and then read stuff.

Simple PagerDuty - Local MCP

I’m a big fan of PagerDuty, and I see that they don’t yet have a remote hosted MCP for us to consume. However, they have published a local MCP that we can run ourselves. I staged that locally (and got uvx/uv installed), and then wrote this Strands script to use it.

On line 2 we establish our starter question, to find all active issues over the last 4 issues, and filter them.

Then on line 5 we define a function to utilize the local MCP server. Note that we’re not streaming anything from a remote destination, we’re literally providing the command flags that would start the MCP server if we ran them.

This script, when run, will basically call the MCP server as a child when we get to the “with” part of the function.

Note the “env” variables we inject into it on line 19-21. These are required to read PagerDuty.

see line 17, where we can enable writing to pagerduty, and with it commented out, the MCP will only execute “read” operations, which I feel is much safer with the state of AI today.

The “with” and other script info is exactly the same as the above, so skipping it, you can read the simple_pagerduty.py script if you want to see what we’re doing there.

Here’s how it looks:

Simple AWS

There is a local MCP in preview that permits the agent to run AWS commands, but it’s in beta, and not very easy to run. However, Strands has a built-in tool that permits it to do things in AWS, able to do whatever the boto3 library is capable of doing.

It’s called “use_aws”. We import it on line 3, along with a few other built-in tools.

You can see on line 8 we pass that same list of built-in tools as a list to our agent on line 11.

Here’s how it looks:

Simple AWS Bedrock Knowledge Base

I built a pretty extensive Knowledge Base on Bedrock for Vera - a GenAI bot that doesn’t operate agentically. So when I read that Strands permits it to talk to Bedrock I expected I’d have to build a ton of code, and re-ranking, and such.

However, it’s incredibly easy. I had to export the ID of the knowledge base:

export KNOWLEDGE_BASE_ID=xxxxxx

And then I had to import the “retrieve” tool, which retrieves information from Bedrock knowledge bases. I pass it like any other tool to the agent, and we’re in business.

Notably, I already have AWS creds exported into this terminal, to utilize bedrock AI services, so I’m already authenticated. If you’re using a different back-end for AI services, make sure to also export AWS creds that can read the Bedrock KB.

Here’s how it looks:

Combined MCP

The real power of MCP comes when you combine multiple MCP tools across platforms, because the model can correlate, say, outages (Pagerduty) to specific code changes (GitHub).

This is pretty intuitive once you’ve seen it. I’ve skipped some code that we covered above where we built the MCP function definitions.

On line 2-3, we call those MCPs, and on line 6 we do a “with” to open each of them.

Then on line 12-13, we sync the tools available for both, and on line 16, we build our “tool_belt” (haha), to combine the list of all the tools together.

And then on line 19 we instantiate the agent and pass it all the tools.

Here’s how it looks:

Summary

In this article we’ve covered how to build Strands agentic Agents with python. We’ve talked about how different types of MCPs work, and built them, and finished up with a multi-platform bot that’s able to:

• Talk to PagerDuty to read your incidents

• Talk to GitHub to read your code changes

• Talk to AWS to read your real infrastructure

• Talk to your AWS Bedrock Knowledge Bases to read your corporate data

And also, to correlate changes among those platforms - that’s going to be a really powerful functionality as we move forward in maturing this technology.

I’m going to keep building super bots, and I’m very interested in AgentCore to establish memory and runtime observability for these bots. I’ll be digging into that next, keep an eye out!

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

This article is part of a series of articles, because 1 article would be absolutely massive.

• Part 1: Create an Azure Bot and App Registration

• Part 2: Register Bot in Teams with Teams Developer Portal

• Part 3: Delegated Permissions and Making Lambda Stateful for Oauth2

• Part 4: Building the Receiver lambda to store tokens and state

• Part 5: Finding messages, reading conversations in Teams

• Part 6 (this article): Finding attachments and reading them into Bedrock

Hey all!

In the last article we talked about how to find the root of messages to build our conversational context in Teams. That is an incredibly challenging task, as the Teams client is several raccoons in a trench coat (well, service APIs behind a front-end, same difference). Those services are generally:

• Azure Entra/AD API - Authentication and Authorization (is Teams license assigned)

• OneDrive API - personal files. Files are shared to the person they are sent to automatically in Teams.

There is some weird behavior because of this. When building a multi-person chat, and sharing files, files are shared to each person in the chat at the time of sharing the file. If a person is added later, they can see previous messages (Teams API) but not previous files (OneDrive API), as files are not batch retroactively shared. #WeirdTeams

• SharePoint API - Team/Channel files. When files are uploaded to a Channel, they are shared with that Team automatically.

To solve this, we have a robust couple functions that find, download, and encode properly the attachments that need to be added to the conversation context.

Lets walk through it.

Iterate Through the Messages

To start with, lets catch up with where the last article covered. We went through the functions that fetch the metadata and messages around “channel” type conversations (those in Teams/Channels), “personal” type messages (those in DMs), and how we “massage” the messages to scrub any junk that Teams returns to us in those API calls. There’s a lot there, it’s worth a read if you’re skipping ahead.

But all you really need to know now is that we have a list of all conversation messages in the thread. For DMs, we just respond to the single request (I’d love to add “threading”/”grouping of topics” in DM context in future, but haven’t found an elegant way yet), and for channels we read back all the messages to the root/parent message.

So that covers us up to line 11.

On line 11 and 12, we initialize a list of conversations and context, that we’ll populate in an iterative.

On line 15, we iterate through each message package that we’ve previously built to build conversation turns - the sort of atomic unit of a conversation that bedrock wants to read. We’ll walk through this more as we dig deeper into this function.

We take that turn (line 17) and append it to the conversation (line 19).

Once we have a complete conversation built in Bedrock’s style, we return it to the parent, which will do the Bedrock conversation.

For now, lets dig into how the build_conversation_turn() function works.

Every single message from a user has from.user.displayName, and if it exists, we set the sender Role as “user”. Roles in Bedrock can be either “user” (not from the model) and “assistant” (from the model).

On line 7, if there is no displayName, we assume it’s the assistant.

On line 10, we get the message.body.content, which s the text of the message. Teams annoyingly uses html encoding, so we strip as much of that with simple regex as we can, line 13-14. I think Bedrock can interpret this fine, but it annoys the humans (read: me) when reading debug logs, so I remove it anyway.

On line 17 we start building the content block, making sure to include the sender name and text, something like:

“Kyler Middleton says: Tell me the weather”

And if there’s no sender_name (it’s the bot), we just include the text. We’ll make sure to set the role later on.

Downloading the Attachments

Next we check for any attachments in the message. This isn’t a straight-forward process (obviously, it gets it own article in this series).

The attachment always has a file name (file.name) and file_extension (everything after the last period in the file name), line 8.

We check for an attribute called the file.contentUrl which is the URL address of the content. There’s different ways to authenticate and build the URL to get the file (I miss the simplicity of slack attachments dearly here), so we pass that url to the download_file_for_users() function, line 14. We’ll dig into that shortly.

There’s a couple of other gotchas here we’ll address before we dive into downloading. Bedrock has pretty strict requirements on document names, so we file the name through regex, line 18. We’re only permitted single spaces (Weird?) so we regex that as well, line 20. We also don’t permit leading or trailing spaces, so we remove those on line 22.

Many of these text manipulations could be combined into single lines, but I abhor complex code. DRY is awful, I’d much rather understand exactly what code is doing.

Pain

It’s worth including the note for this function. All my functions have notes that describe broadly what they do, but this one is largely an apology on behalf of Teams’ APIs. I don’t know why the Teams… Team decided to make all of this such a pain to work with, but I hope they stub their toes walking to their beds after turning out the lights.

This function is more than 100 lines of code, so buckle up 😭

First, on line 9, we wrap all this in a try. Why? Because we’re doing an extraordinary amount of logic here, and if we fail any of it we want our script to continue working. If we can’t fetch the attachment, we just continue with the text only. That’s just fine.

First, we check to see if the file_url contains sharepoint.com/sites/, which means the file is stored in SharePoint. We need to fetch the file from SharePoint, which turns out to be an incredibly annoying thing to do.

First, we need the site_id. Lets build a graph call to go fetch that. We establish the primitives by splitting the URL, line 13-17, and then build the URL, line 20. We require the file, line 21, via requests, and check the call, line 22. If it worked, we set the site_id to the response.id, line 23.

With that in hand, we need the default document library ID, so we build a URL to fetch THAT, line 26-29.

Then we need to extract the file path. These sometimes work just fine, and sometimes contain characters (why the heck?). We use a URL encoding function to do the heavy lifting of encoding the full URL plus document name (spaces and characters and such get encoded a bit weird in http), line 32.

Then we FINALLY finally try to download the file, line 34.

I noticed quite a few of my test downloads had a “\u202f” which is the unicode character encoding for a “narrow non-breaking space”. Why are those in the file path? I have absolutely no idea dude. But we have a whole block where we just strip that off and see if we can download the file again.

If that doesn’t work we want to throw our hands up and retire, but there’s one more method we can use - literally fricking searching SharePoint using graph for the file name, then reading through all the results, one by one, to see if any exactly match the file name.

This a slow, serialized, and generally bad way to get this working. However, it often works, so ¯\_(ツ)_/¯

If that still fails, we just give up for SharePoint paths. I don’t know another method to find the file, other than the SharePoint/Teams API devs simply providing a link to the file (please goddess).

Next up, we try to download the file from the user’s personal OneDrive. This works if the user has uploaded the file themselves in a DM context.

Again, this is a relatively “bad” solution since it requires our own user to have uploaded the file. If the file was uploaded in a DM context by another user, I think we just won’t find it. Sigh.

Lastly, we try to directly download the file via the file_url. You’d think this would be the most reliable method for downloading files. After all, this is the only method we use for slack and it works 100% of the time.

However, I have never seen this work in my testing and live auditing. I have no idea why, but I keep it because, well, it SHOULD work, right? And maybe it’ll catch some edge case that I haven’t tested yet.

Finish the Conversation

After that, the logic is pretty much the same as the Slack Vera bot - we attach the bytecode of the files to the conversation turns, pack each one with metadata to identify it as “user” or “assistant”, and pass it to the model.

The model returns a response, and we pass it to the Teams chat as a response.

It’s quite a bit slower than the Slack Vera, largely due to the vastly greater number of API requests required to do pretty much anything - finding the conversation ID, mapping different types of identifiers together, finding and testing all different paths for downloading files, etc.

It’s hard to beat Teams’ price (FREE, or at least table stakes in your EA agreements), but it remains a bear to work with. Thank goodness it’s improving, it was utterly awful a few years ago.

Summary

We covered even further the idiosyncrasies of the various APIs that Teams uses, and how they basically operate like a raccoon in a trench coat - lots of loosely collected APIs built in different standards and with differently methodologies, trying to operate as a single entity. It sometimes even works properly.

That’s probably it for the Teams Vera series. I’ve discovered Strands, an open source tool from AWS that permits building MCP-enabled Agents to do stuff in a standard way, and I’m entirely hooked. I’m barely keeping my non-AI job right now, this stuff is just too cool.

My next articles will surely follow that trend.

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

This article is part of a series of articles, because 1 article would be absolutely massive.

• Part 1: Create an Azure Bot and App Registration

• Part 2: Register Bot in Teams with Teams Developer Portal

• Part 3: Delegated Permissions and Making Lambda Stateful for Oauth2

• Part 4: Building the Receiver lambda to store tokens and state

• Part 5 (this article): Finding messages, reading conversations in Teams

• Part 6: Finding attachments and reading them into Bedrock

Hey all!

In the last article, we covered how the Receiver lambda gets an OAuth2 token that’s compatible with the GraphAPI, and how we encrypt and store that token so that our Worker lambda can utilize it to DO STUFF - like have AI conversations and post responses from our models back to the Teams platform.

Assuming that’s all working, each time our Worker is triggered now, we’re going to get a valid OAuth2 token, and some conversation information (someone tagged the bot, or messaged it directly), and it’s now our Worker lambda’s job to do the AI magic. There’s a few parts of AI magic we’ll cover.

• Extract the message context - is this a DM or tag in a Teams.. Team (We know naming stuff is hard $msft)

• Read the previous messages if applicable

  • In a DM context, there’s no good way yet to differentiate different conversations, so we just respond directly to each question - there’s no follow-up/thread model. When Teams supports this (is that coming?) we could extend the bot to cover that model.

  • In a Teams Team context, read messages back. We have to limit this to some length, and not read ALL the messages if there’s hundreds. So we limit to like 20 messages.

• Download any attachments to any message we read

  • This is hilariously very different on context because of how Teams doesn’t REALLY exist. If you share an attachment in a DM, it’s stored in your OneDrive. If you share an attachment in a Teams Team, it’s usually stored in that Team’s SharePoint storage. The names are sometimes transformed to be SharePoint/OneDrive compatible, which makes my pull my hair out. We’ll cover this really well lower on.

• Read the user data of any post author

  • To detect pronouns, nicknames, etc.

• Get an AI response

  • We’ve covered this well in previous articles (here’s the SlackBot articles), so we won’t cover this here

• Post a response

  • Send our response back, properly formatted for Teams.

When it all works, it looks like this:

If you don’t want to walk through all this, the code we’re going to be talking about is here:

github.com/KyMidd/TeamsAIBot

Bot Bearer Token Fetching

Lets first fetch the bot’s bearer token. That token is used to post responses to the user in 1:1 DM chats, and respond to a Team’s Team/Channel in shared context.

On line 4 we fetch the json-structured secrets package, and then load it on line 7. Lines 8 - 10, we assign particular values of the secrets to variables for easy use.

Then on line 13, we call get_teams_bearer_token and send most of our secret info to it. Lets follow that thread.

This function helps us get a token to use to post responses to the user when we’re ready to. We do this immediately so we can post error messages back if anything breaks.

On line 4, we set the token_url as a static string - this is required for all BotFramework compatible bot implementations.

Then on line 7 we assign the scope - the list of permissions we’re requesting. This bot only has the permissions required, so we don’t care if we get ALL of them.

Pro tip: If the bot is assigned broad/admin permissions, and you only want this bot to get limited permissions, you can prune permissions back here by what you request.

On line 10 we build our payload, and on line 18 we request the token.

On line 19, we raise an error if we get anything other than an http/200, and if we pass this check, on line 22, we extract the “access_token” from the response and return it on line 24. This helps validate that we are the bot, and we have a right to send responses AS the bot.

Extracting the Auth Token

Before we do anything, we need the auth token so we can operate on behalf of the user. Remember, the “Receiver” lambda either had this cached or asked the user to go approve it through the SSO Card we sent. The Worker lambda (that does all this AI stuff) won’t be triggered until the auth token is valid and received.

Remember, we packed the auth token in the “event” that is passed from the Receiver (think Receptionist), as an encrypted, base64’d string.

Lets fetch the token blob on line 7 from the event payload, and then send it to the decrypt_user_auth_token() function on line 8. Lets follow that thread.

The token is base64’d after being encrypted, so lets decode it on line 4.

Then we register a KMS client on line 7, and read the KMS CMK alias that we passed in from terraform as an environmental variable on line 10.

Then we pass the token (no longer encoded as base64, but still an encrypted blob) and the key alias to the kms.decrypt() function.

On line 19, we identify the plaintext response from the KMS call, and decode it as a utf-8 string.

PHEW, that was a lot to get the token. But it does very much help us not pass plaintext access tokens from users, and to not store them in dynamo unencrypted.

On line 22, we send the decrypted access token back.