This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

Last article we built the orchestrator pattern, a single CLAUDE.md file that coordinates five agents through a full development pipeline. If you missed it:

That article covered the pipeline flow and how agents hand off work to each other. This one zooms in on how each agent is configured. The file configuration, what’s optional and what’s not, and the design decisions we made to make the autonomous development pipeline reliable.

Now lets talk how we can configure the sub-agents!

Agent Files Are (Just) Markdown

Each agent is defined in a single markdown file inside “.claude/agents/”, one file per agent. The Coder agent lives in “.claude/agents/coder.md”, the CodeReviewer in “.claude/agents/code-reviewer.md”, and so on.

The real config is in the YAML frontmatter block at the top. Here’s what the CodeReviewer’s looks like:

---
name: code-reviewer
description: Code review specialist that reviews all Python and Terraform 
  changes. Enforces quality standards, catches bugs, and reports issues. 
  Does NOT fix code - reports findings for the main agent to delegate 
  to a Coder.
tools: Read, Glob, Grep, Bash
model: opus
---

Four required fields do all the work:

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all,

In the last series I wrote for LDD, we built an agentic Slack bot on AWS AgentCore. The architecture, MCP tools, memory, deployment. If you missed it, start here:

That series ended with a teaser about multi-agent orchestration. This is that series. We’re going to teach Claude Code to develop the bot itself. Implement features, review its own code, deploy to DEV, and test the live bot via Slack. All from a single prompt.

This first article covers the orchestrator pattern: how one CLAUDE.md file coordinates five agents through a full development pipeline, and how we codified our entire SDLC into config files that Claude Code follows autonomously.

If you want to skip the words and see the code, go here:

Orchestrator claude.md

Specialized Agent files

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. If you find these useful, please consider subscribing. It’s what keeps these articles coming. <3

Hey all! I’ve been building an agentic email bot on AWS Bedrock AgentCore that reads and replies to emails autonomously. I wrote up that whole series:

• Part 1: Full Architecture and O365 Integration - EventBridge polling, Graph API delta queries, and handing off to AgentCore

• Part 2: Building the Thread, Sending the Reply - Conversation context, email-safe HTML rendering, and production operations

While building it, I wired up Microsoft Graph API for O365 access so I could have my Claude Agent test and verify changes during development. That Graph integration became the foundation for something more immediately useful: a Claude Code skill that lets you send email directly from a conversation.

Later, in a Slack channel dedicated to using AI, someone asked if anyone had a solution to allow Claude Code to send emails, specifically for automations like emailing reports on a schedule. So I pulled it out, made it portable, and published it to our internal plugin marketplace.

There are a lot of good reasons to want this. You might be testing email-related functionality, like I was, or you want to say “email Sarah and update her on our progress with this project” and have Claude just handle it in context. Or you’re running scheduled jobs and need Claude to fire off a notification or send a report when something completes. Once you have it, you find uses for it constantly. (I also use the email access in general to leverage Claude in a custom daily review process to help me stay on top of tasks and such, but that’s a story for another blog post.)

The idea sounds simple enough. The implementation had some genuinely interesting constraints that made it worth writing up.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. If you find these useful, please consider subscribing - it’s what keeps these articles coming. <3

Hey all! Alex here again. In Part 1 we covered the polling architecture: EventBridge, delta queries, idempotency, and handing off to AgentCore. That got us to the point where the AgentCore worker receives an email payload and needs to do something useful with it.

This article covers what happens next. The worker needs to understand the full conversation, not just the latest message. It needs to run the agent. And it needs to send a reply that actually looks good in Outlook, Gmail, and Apple Mail. Each of those is its own rabbit hole.

This is Part 2 of a 2-part series:

• Part 1: Full architecture and Microsoft 365 integration

• Part 2 (this article): Building the email thread, sending formatted replies, and operating in production

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support writing more of these articles <3 :)

Hey all! I’m Alex, a new contributor to Let’s Do DevOps. At work we’ve been building out a fleet of agentic AI bots backed by AWS Bedrock AgentCore: bots that live in Slack, Teams, and now email. Kyler has covered the Slack bot in depth across several articles. I want to show you the email version: same AgentCore foundation, completely different integration challenge.

The goal was to lower the friction for people to interact with our internal agentic bot. Slack and Teams work great if you’re in them all day, but not everyone is. The C-suite lives in email, and there are great email-native use cases too. Forward a vendor offer and ask the bot if accepting it would violate the company gift policy. Get CC’d on a question you don’t know the answer to? CC the bot and let it respond. The inbox is already where a lot of conversation happens, so we brought the bot there.

This is Part 1 of a 2-part series:

• Part 1 (this article): Full architecture and Microsoft 365 integration

• Part 2: Building the email thread, sending formatted replies, and operating in production

The Problem: Email Is Not Slack

If you’ve built a Slack bot before, you know how it works. Slack sends you an event when someone messages the bot. You process it, respond, done. It’s push-based so Slack does the work of notifying you.

Email is more complicated.

While Microsoft Graph supports push notifications for mail via a subscription, those subscriptions expire after 7 days and have to be renewed, you have to handle a validation handshake on creation, and if your endpoint is slow or unavailable, Graph will start dropping notifications permanently with no recovery. It’s more moving parts than it looks.

We chose polling instead. Simpler to operate, no subscription lifecycle to manage, and the delta query pattern makes it efficient.

That choice shapes the whole architecture. Instead of an event-driven Lambda that wakes up when Graph pings it, you need a polling layer that runs on a schedule, checks for new messages, and decides what to do with them.

But once you have polling, you immediately have two new problems: efficiency (you don’t want to re-fetch every email every minute) and idempotency (you don’t want to process the same email twice if something goes wrong).

We solved both. Let me show you how.

Read more

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In this 4th and last article in the AgentCore series, we’ll talk about real life deployment, operations, and lessons learned while building this project. Here are links to the other articles in the series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 (this article) - AgentCore Deployment, Operations, and Lessons Learned

Deploying an AgentCore runtime isn’t like deploying a Lambda function. There’s a container to build, a two-Lambda architecture to wire up, module boundaries to get right in Terraform, and a regional split that will trip you up if you’re not expecting it. Once it’s running, the operational model is different too. Your bot is a long-lived application now, not a function that spins up and dies. That changes how you think about health checks, debugging, cost, and failure modes.

This article walks through the Terraform patterns that hold everything together, what container lifecycle actually looks like in practice, how to debug when things go wrong, and an honest look at what I’d do differently if I started over. If you’ve been following along and thinking about building your own AgentCore agent, this is the practical stuff you’ll want before you start.

Let’s get into it.

Terraform Deployment Patterns

The whole project is structured as Terraform modules. Each component is its own module with its own variables, IAM roles, and resources. The root `main.tf` wires them together:

module “gateway”    { source = “./gateway”    }  # MCP Gateway + Cognito
module “receiver”   { source = “./receiver”   }  # Slack webhook handler
module “invoker”    { source = “./invoker”    }  # Async AgentCore caller
module “worker”     { source = “./worker”     }  # AgentCore runtime
module “auth_portal” { source = “./auth_portal” }  # Per-user OAuth flows

Data flows one direction: gateway outputs feed into the worker, the worker’s runtime ARN feeds into the invoker, and the invoker’s function name feeds into the receiver. Each module only gets the permissions and configuration it needs.

# main.tf - wiring it together
module “invoker” {
  source            = “./invoker”
  agent_runtime_arn = module.worker.runtime_arn  # Worker feeds invoker
}
module “receiver” {
  source                = “./receiver”
  invoker_function_name = module.invoker.invoker_function_name  # Invoker feeds receiver
}
module “worker” {
  source            = “./worker”
  gateway_arn       = module.gateway.gateway_arn       # Gateway feeds worker
  gateway_client_id = module.gateway.cognito_client_id
  gateway_url       = module.gateway.gateway_url
  memory_id         = aws_bedrockagentcore_memory.vera_memory.id
  # ... 20+ more variables
}

That worker module has a lot of inputs. Gateway auth, memory config, knowledge base IDs, guardrail IDs, OAuth table references, audit log settings. It all gets passed as environment variables on the AgentCore runtime resource. This is the glue between infrastructure and application code. The worker’s Python code reads `os.environ.get(”GATEWAY_CLIENT_ID”)` and never knows or cares that Terraform set it.

The two Lambdas are deliberately minimal. The receiver is a single Python file zipped and deployed directly, no Docker, no build pipeline. 512MB memory, 10-second timeout, ARM64. Its only job is to validate the Slack signature and asynchronously invoke the invoker. The invoker is even smaller, 256MB, and its only job is to call `InvokeAgentRuntime`. Both run on ARM64 for the cost savings since Graviton Lambda is about 20% cheaper than x86.

# Receiver: tiny, fast, validates Slack webhooks
resource “aws_lambda_function” “receiver” {
  memory_size   = 512
  timeout       = 10
  runtime       = “python3.12”
  architectures = [”arm64”]
}
# Invoker: even tinier, calls AgentCore
resource “aws_lambda_function” “invoker” {
  memory_size   = 256
  timeout       = 900  # Matches Lambda max, waits for AgentCore
  runtime       = “python3.12”
  architectures = [”arm64”]
}

The invoker’s 900-second timeout might look surprising. It’s set to the Lambda maximum because it’s making a synchronous call to AgentCore and waiting for the response. The agent can take minutes on complex queries. If the invoker times out before the agent finishes, the response gets lost. Setting it to max is cheap insurance since the invoker uses almost no memory and you only pay for execution time.

The worker container build is where things get more interesting. We use a content-hash tagging strategy so the container only rebuilds when the code actually changes:

locals {
  # Hash all Python source files
  python_files_hash = sha256(join(”“, [
    for f in fileset(”${path.module}/src”, “*.py”) :
    filesha256(”${path.module}/src/${f}”)
  ]))
  # First 12 chars of hash as image tag
  image_tag = substr(local.python_files_hash, 0, 12)
  image_uri = “${var.ecr_repository_url}:${local.image_tag}”
}

Every Python file, the Dockerfile, and `requirements.txt` are hashed. If nothing changes, the tag stays the same and `null_resource.build_and_push_worker_image` doesn’t fire. If you change a single line of Python, you get a new hash, a new tag, and a new build. No “latest” tag guessing games, no manual version bumps. Terraform knows exactly when to rebuild.

The build itself targets ARM64 using `docker buildx`:

provisioner “local-exec” {
  command = <<EOF
    docker buildx build \
      --platform linux/arm64 \
      --output=type=registry,compression=gzip,force-compression=true \
      -t ${local.image_uri} \
      --push .
  EOF

ARM64 containers on AgentCore use Graviton instances, same cost savings as the Lambdas. The `--output=type=registry` pushes directly to ECR without storing a local image, which keeps CI/CD clean.

One thing that caught me off guard: the regional split. AgentCore, memory, guardrails, and the gateway all run in us-east-1. But the Bedrock Knowledge Base is in us-west-2. At the time I set this up, not all Bedrock features were available in every region. The knowledge base needed a region where the embedding model was available, and that was us-west-2. The worker handles this with a separate region variable:

# providers.tf - two regions
provider “aws” {
  region = var.region  # us-east-1: AgentCore, gateway, memory
}
provider “aws” {
  alias  = “west2”
  region = “us-west-2”  # Knowledge Base only
}

It works, but it means IAM policies need to cover both regions, and you need to be deliberate about which provider each resource uses. If you’re starting fresh, check if your region supports everything first. A single-region deployment is much simpler.

Container Lifecycle and Operational Reality

In Article 1, I talked about AgentCore’s “living application” model in the abstract. Now that I’ve been running it in production for a few months, here’s what it actually looks like.

The lifecycle is controlled by two values:

variable “idle_timeout” {
  description = “Idle runtime session timeout in seconds”
  type        = number
  default     = 900   # 15 minutes
}
variable “max_lifetime” {
  description = “Maximum runtime lifetime in seconds”
  type        = number
  default     = 28800 # 8 hours
}

`idle_timeout` is how long the container sits idle before AWS shuts it down. `max_lifetime` is the absolute ceiling, no matter how busy it is. After 8 hours, the container gets recycled and the next request spins up a fresh one.

The 15-minute idle timeout might seem short, but it’s a balance between responsiveness and cost. During business hours, Vera gets enough traffic to stay warm. After hours, the container shuts down within 15 minutes and you stop paying. The next morning’s first message takes a cold start penalty, about 30-45 seconds while the container boots, loads dependencies, fetches secrets, initializes MCP connections, and registers with AgentCore. After that, requests are fast.

Here’s the thing about cold starts though. With Lambda, a cold start was maybe 5-10 seconds for a lightweight function. AgentCore’s cold start is heavier because you’re booting a full Docker container with Python, MCP servers, Node.js (for Azure MCP), the OpenTelemetry instrumentation, and the AWS CLI MCP server. Look at the Dockerfile and you can see why it takes a bit:

# System deps: Node.js for Azure MCP, .NET libs, curl, etc.
RUN apt-get update && apt-get install -y \
    ca-certificates curl tar gzip nodejs npm \
    libicu-dev libssl3 zlib1g libgcc-s1 libstdc++6 libc6
# Azure MCP server
RUN npm install -g @azure/mcp@0.5.8
# Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
# AWS CLI MCP server in its own venv
RUN mkdir -p /opt/aws-cli-mcp-server && \
    cd /opt/aws-cli-mcp-server && \
    uv venv && \
    uv pip install --python .venv/bin/python awslabs.aws-api-mcp-server

That’s a lot of stuff in one container. Vera talks to six or seven different platforms, and each one has its own runtime dependencies. Having these all ready to go means each reqeust is faster.

Once the container is warm, the application loop is straightforward. The `BedrockAgentCoreApp` handles health checks and request routing. When a request comes in, it hits the `@app.entrypoint` handler:

@app.entrypoint
def handle_slack_message(payload):
    # Start a background thread so we don’t block health checks
    task_id = app.add_async_task(”slack_message_processing”, {...})
    def process_in_background():
        try:
            handle_message_event(...)
        except Exception as error:
            # Post error to Slack so users aren’t left hanging
            slack_app.client.chat_postMessage(
                channel=channel_id, text=error_message, thread_ts=thread_ts
            )
        finally:
            app.complete_async_task(task_id)
    thread = threading.Thread(target=process_in_background, daemon=True)
    thread.start()
    return {”status”: “processing”, “task_id”: task_id}

I didn’t want to do background threads - they were required by AgentCore’s architecture. My lambda-built design was built to handle a single thread at a time, because who cares, each request uses a single lambda. There’s no need to have background threads and multi-tenancy, because... well, there isn’t multi-tenancy. That isn’t the case here. The blocking processing of a thread meant the heartbeat that AWS AgentCore sends to our application stops responding, and AgentCore thinks our application has died, and AgentCore’s service recycles the app - not great.

A background thread has a great benefit - it’s worked asynchronously, and the heartbeat process can continue responding even when working many requests.

Now, the cost question. Lambda is pay-per-invocation. You pay for exactly the compute time your function uses. AgentCore is pay-per-container-hour. You pay for the container running, whether it’s processing a request or sitting idle.

For a bot like Vera with steady weekday traffic, the costs end up similar. During business hours, the container is warm and handling requests efficiently. After hours and weekends, it shuts down after 15 minutes of inactivity. You’re not paying for 24/7 uptime.

Where Lambda wins is low-traffic bots. If your bot handles 5 requests a day, Lambda’s pay-per-invocation model is cheaper than keeping a container warm. AgentCore’s idle timeout helps, but there’s still the overhead of cold starting multiple times a day.

For Vera’s usage pattern of hundreds of users, dozens of requests per hour during business hours, and near-zero after hours, the cost is roughly the same. I didn’t migrate for cost savings. I migrated for the features.

Debugging and Observability

Debugging in AgentCore was a HUGE PAIN. When I first migrated, I couldn’t get the stdout messages to appear in cloudwatch at all. I thought I had to depend on x-ray and OpenTelemetry (OTEL), and that situation is just horrible. I wanted to like the OTEL observability stack and how it’s currently implemented in AWS but I just can’t. Clicking through literally thousands of events to find the failure straight up sucks.

After debugging for hours, I found out that python BUFFERS ITS OUTPUT to cloudwatch, sometimes for the entire duration of the container, and then it dies without writing to cloudwatch.

Adding the `python -u` to the invocation of the app makes it logs to stdout without buffering. This has a (very tiny) impact on perforamcne, but it means that the APP GENERATES LOGS which are absolutely necessary to run any damn thing.

The OpenTelemetry setup is one line in the Dockerfile:

CMD [”opentelemetry-instrument”, “python”, “-u”, “-m”, “worker_agentcore”]

That `opentelemetry-instrument` wrapper auto-instruments boto3 calls, HTTP requests, and other common libraries. You get traces in X-Ray showing the full lifecycle of a request: Bedrock Converse API calls, Secrets Manager fetches, MCP tool invocations, Slack API posts. When a request takes 4 minutes and you want to know why, X-Ray shows you which tool call was slow.

I don’t like to use .logging(), I literally just have a `DEBUG_ENABLED` environment variable that turns on stdout print() logging. In production it’s off, but when something weird happens, you can flip it on and get full request/response bodies logged:

debug_enabled = os.environ.get(”DEBUG_ENABLED”, “False”)
# In the Converse API call
if debug_enabled == “True”:
    print(f”converse_body: {converse_body}”)

Error handling follows the same philosophy of telling users what happened. Vera categorizes errors into three buckets and gives specific guidance for each:

THROTTLING_KEYWORDS = [”throttl”, “rate limit”, “too many requests”, ...]
CONTEXT_OVERFLOW_KEYWORDS = [”context window”, “too many tokens”, ...]
if is_context_overflow:
    return “Request was too large... try narrowing your request”
elif is_throttling:
    return “High demand, please try again in a few minutes”
else:
    return “Unexpected error, please try again”

Context window overflows happen more often than you’d expect. When Vera analyzes a large Jira board with hundreds of tickets, the accumulated tool responses can blow past the model’s context limit. The error message tells users to narrow their request, like limiting to a specific sprint or date range, which has cut down on repeat failures.

The Strands SDK quirk I mentioned in the memory article also shows up in debugging. Short-term memory events that contain malformed or empty messages cause the Strands memory interface to crash rather than gracefully discarding them. The workaround is to use unique session IDs per message so that corrupted short-term memory from one conversation can’t bleed into the next. It’s not elegant, but it’s reliable. If you see mysterious crashes during message processing with no obvious cause, check your session ID strategy first.

This bug will likely be fixed in the Strands Memory interfact client soon, but until then you can remove conversation turns that are empty which MCP tools sometimes generate, then memory didn’t crash. It only took me 5 hours of debugging to figure that out.

What I’d Do Differently and What’s Next

I’ve been running Vera on AgentCore in production for a few months now, serving hundreds of users across Slack, Teams, and Email.

Things I’d do the same: the module-per-component Terraform structure, the hash-based image tagging, the background thread pattern, ARM64 everywhere, and the three-Lambda architecture (despite this being quirky as all get out, it works great). These patterns have held up well and I’d use them again on a new project without hesitation.

Things I’d do differently:

Start with a single region. The us-east-1/us-west-2 split was necessary at the time because of Bedrock feature availability, but it added complexity to every IAM policy and every debugging session. Check regional availability before you commit to a multi-region setup. If you can fit everything in one region, do it.

Don’t underestimate the Dockerfile. Every MCP server you add brings its own runtime: Node.js for Azure, a Python venv for AWS CLI, pip packages for everything else. The Dockerfile grew fast and so did the cold start time. If I were starting fresh, I’d think harder about which MCP servers truly need to be local (stdio) versus remote, because every local server is another dependency baked into your container image.

Now, what’s next. AgentCore supports multi-agent orchestration, where a supervisor agent delegates to specialist agents, each with their own tools and context windows. I haven’t implemented this yet, but the architecture is clearly heading there. Imagine a “manager” agent that routes questions to a GitHub specialist, a Jira specialist, and an incident response specialist, each optimized for their domain. The pieces are all there in AgentCore. I just haven’t needed it yet because Vera handles the current workload as a single agent.

The MCP Gateway is also evolving. When AWS adds broader bearer token support for gateway targets, the direct MCP connections for GitHub, Atlassian, and Splunk can all move to the gateway. That means less auth code in the application, fewer secrets to manage, and adding a new tool provider becomes a Terraform resource instead of a code change.

If you’re considering building an agentic bot on AWS today: if your workflows fit within Lambda’s 15-minute timeout and you don’t need persistent memory or a managed tool gateway, Lambda with the Strands SDK is simpler and cheaper. It works great. But if you’re hitting the timeout wall, or you want memory that persists across conversations, or you’re tired of managing credentials for six different MCP servers in application code, AgentCore is worth the migration.

The code for this entire project is open source: https://github.com/KyMidd/AgentCore_AgenticSlackBot. Feel free to poke around, steal ideas, or open issues when things don’t make sense.

Happy building!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In the last article, we covered how Vera connects to tool providers through the MCP Gateway and direct connections. Now let’s talk about something that makes AgentCore feel genuinely different from Lambda: memory.

This article is part of a series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 (this article) - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

This is persistent memory that survives across conversations, across days, across sessions. Not only can users teach the bot to call them a nickname, users can teach Vera multi-step business processes: “When I ask for standup notes, check Jira for my tickets, pull my recent PRs from GitHub, and summarize my Confluence updates from the past week.” Once learned, Vera executes that workflow every time without the user repeating the instructions. In the first month of deployment, Vera memorized roughly 2,000 facts across our user base, including work patterns, team structures, preferred workflows, and procedural preferences.

This is one of those features where the “learning AI” model really pays off. In Lambda, every invocation starts with a blank slate. You can bolt on memory by writing to DynamoDB or S3, but you’re building and maintaining that whole system yourself. AgentCore gives you a managed memory service with strategies, namespaces, and semantic search out of the box.

AgentCore memory has two layers that work together. Short-term memory events capture each conversation as it happens. A backend process then analyzes those events and extracts durable facts and preferences into long-term memory records. This article covers both layers, how we use the user preference strategy to give Vera per-user memory, the identity model that ties it all together, and the tools we built so users can see and control what’s been remembered about them.

Let’s get into it.

If you’d prefer to skip the write-up and just read the code, the entire codebase is public and MIT open sourced here.

Two Layers of Memory: Events and Records

AgentCore memory has two distinct layers, and understanding the difference is important because they serve very different purposes.

Short-term memory (events) are created during each interaction with the bot. When a user sends a message and Vera responds, that exchange gets written as an event. Events are the raw material. They capture what happened in a specific conversation, and they expire based on your configured TTL (Usually 7 days or so).

Long-term memory (records) are created by AgentCore’s backend processing system based on the memory strategy/strategies you configure. After events are written, AgentCore asynchronously analyzes them and extracts durable facts, preferences, and patterns. These get stored as memory records in a namespace you define. Records are the refined output. They’re what the bot actually retrieves and uses in future conversations.

• Events (short-term): Created by each interaction. Contains raw conversation exchanges. Lifespan is a configurable TTL (we use 30 days).

• Records (long-term): Created by the AgentCore backend. Contains extracted facts, preferences, and procedures. Lifespan is also a configurable TTL (we use 30 days).

Here’s the flow in practice:

1. A user tells Vera: “When I ask for standup notes, check Jira for my tickets and pull my recent GitHub PRs”

2. Vera writes that conversation as a short-term memory event

3. AgentCore’s backend asynchronously processes the event and extracts a long-term record: “User’s standup procedure: query Jira tickets, fetch GitHub PRs”

4. Days later, the user says: “Standup notes please”

5. Vera retrieves the relevant memory record, recognizes the learned workflow, and executes it

You don’t (and can’t!) write long-term memories directly. You have conversations, events get created, and AgentCore’s backend figures out what’s worth remembering. Your job is to configure the strategy that tells AgentCore what kind of things to extract.

Pro tip: It takes about a minute after the event is created for the record to be created. That’s usually fast enough, but I try to tell my users what to expect for the short turn-around time.

What Gets Remembered - Memory Strategies

AgentCore supports multiple memory strategies that you can configure on the AWS Memory resource. Each strategy tells the backend what to look for when processing events. I’ve only explored one so far: user preferences.

User preferences extract facts about individual users. Things like timezone, preferred tools, communication style, team membership, and procedural workflows. The strategy is keyed by actor ID, so each user gets their own isolated memory space.

Here’s the Terraform:

# The memory resource itself - shared across all strategies
resource “aws_bedrockagentcore_memory” “vera_memory” {
  name                      = “${title(var.region_short_code)}${title(var.account_short_code)}VeraMemory”
  event_expiry_duration     = var.session_ttl_days
  memory_execution_role_arn = module.worker.worker_task_role_arn
}
# User preferences strategy
# Extracts facts, preferences, and patterns from conversations
# Distinct for each user (keyed by actorId)
resource “aws_bedrockagentcore_memory_strategy” “user_preferences” {
  name                      = “${title(var.region_short_code)}${title(var.account_short_code)}UserPreferences”
  memory_id                 = aws_bedrockagentcore_memory.vera_memory.id
  type                      = “USER_PREFERENCE”
  namespaces                = [”/preferences/{actorId}”]
  memory_execution_role_arn = module.worker.worker_task_role_arn
}

Two things to notice:

The `event_expiry_duration` controls how long short-term events stick around. Events older than that get cleaned up automatically. This is the raw conversation data, and you probably don’t want it living forever.

The namespace pattern `/preferences/{actorId}` is where the magic happens. AgentCore replaces `{actorId}` with the actual user identifier at runtime, creating per-user memory isolation. My preferences live in `/preferences/kyler_middleton`. Someone else’s live in their own namespace.

Users don’t have the ability to fetch memories for others users - that’d be a scary thing to deploy, and is something I had to be careful not to permit

We plan to add a semantic knowledge strategy in the future. Where user preferences capture things specific to an individual (”my standup process”, “my timezone”), semantic knowledge would capture broader organizational facts that come up in conversations: “the deploy window is 2-4 PM EST” or “the staging environment uses us-west-2.” Knowledge that isn’t specific to a single user but is useful for the whole team. Adding it would mean a new strategy resource in Terraform with a different namespace path and including that namespace in the retrieval config.

Identity: Sessions and Actor IDs

Memory needs two identifiers to work correctly: a session ID that scopes short-term events, and an actor ID that scopes long-term records. Getting these right matters more than you’d think.

Here’s something that wasn’t obvious and cost me some debugging time. AgentCore uses the session ID to scope short-term memory. If you reuse the same session ID across multiple messages, the short-term memory accumulates every previous exchange. That sounds useful until you realize the agent replays all of that accumulated context on every new message, and it can corrupt the conversation state when messages from different contexts bleed together.

This shouldn’t have been necessary, but the memory interfact for the Strands agentic library that Vera is based on has some limitations for processing blank or malformed messages, and crashes rather than discarding them. I might try and fix this in future, but for now I’m just pruning invalid messages and separate short term memory streams.

I fixed it by giving every inbound message its own unique session ID.

def generate_session_id(body):
    event = body.get(”event”, {})
    # Use the unique message timestamp from Slack
    ts = event.get(”ts”, “”)
    if ts:
        # Replace periods with underscores for AWS regex compliance
        return f”msg_{ts.replace(’.’, ‘_’)}”
    # Fallback to UUID if no timestamp
    import uuid
    return f”msg_{uuid.uuid4()}”

Slack gives every message a unique timestamp like `1719432000.123456`. We convert that to `msg_1719432000_123456` (replacing periods with underscores because AWS’s regex validation doesn’t allow dots in session IDs).

This means every message is its own session from AgentCore’s perspective. Short-term memory doesn’t carry over between messages. But long-term memory absolutely does, because preferences are keyed by actor ID, not session ID.

And that’s the other half of the equation. The actor ID needs to be consistent across sessions. If I get actor ID `kyler_middleton` on Monday and `U12345678` on Tuesday, those memories live in different namespaces and the bot won’t remember Monday’s conversation.

We derive the actor ID from the user’s Slack profile name:

# Fetch the user’s Slack ID as fallback
actor_id = body[”event”][”user”]
# Extract real_name_normalized from Slack profile
real_name = (
    user_info_json.get(”user”, {})
    .get(”profile”, {})
    .get(”real_name_normalized”, “”)
)
if real_name:
    # Convert to lowercase and replace spaces with underscores
    actor_id = real_name.lower().replace(” “, “_”)

“Kyler Middleton” becomes `kyler_middleton`. This has a nice side effect: the namespace path `/preferences/kyler_middleton` is human-readable when you’re debugging. But it’s not perfect. Someone with a name change, or a name that normalizes differently between Slack and Teams, would end up with split memories. For now it works, and the fallback to the raw Slack user ID (`U12345678`) means we never fail entirely.

I use a “first_last” as the actor ID because it’s something I can correlate between Slack, Teams, and Email, so memory can be shared among all those platforms. And that is pretty darn cool.

Both identifiers get packed into the memory config that flows through the rest of the system:

The design is stateless short-term memory with stateful long-term memory. Each message gets a fresh session (no short-term memory event accumulation), but user preferences persist across every session via the actor ID.

Wiring Memory Into the Agent

The memory config flows into the agent through the Strands SDK’s session manager. This is where you configure which namespaces to retrieve from and how selective the retrieval should be:

from bedrock_agentcore.memory.integrations.strands.config import (
    AgentCoreMemoryConfig,
    RetrievalConfig,
)
from bedrock_agentcore.memory.integrations.strands.session_manager import (
    AgentCoreMemorySessionManager,
)
# Memory configuration
agentcore_config = AgentCoreMemoryConfig(
    memory_id=memory_config[”memory_id”],
    session_id=memory_config[”session_id”],
    actor_id=memory_config[”actor_id”],
    retrieval_config={
        # User preferences only - high relevance threshold
        “/preferences/{actorId}”: RetrievalConfig(
            top_k=5, relevance_score=0.7
        ),
    },
)
# Create session manager
session_manager = AgentCoreMemorySessionManager(
    agentcore_memory_config=agentcore_config, region_name=memory_region
)
# Pass to agent
agent = Agent(
    model=bedrock_model,
    system_prompt=system_prompt,
    tools=tools,
    session_manager=session_manager,
)

The `retrieval_config` is a dictionary keyed by namespace pattern. Memory is just a vector-based knowledge base (just like every other bedrock knowledge base you’ll use), so for each namespace, we set two parameters:

`top_k=5` means retrieve the 5 most relevant memory records. You don’t want to flood the agent’s context with every memory ever stored. Five is usually enough to capture the important preferences without noise.

`relevance_score=0.7` is the minimum similarity threshold. Memory retrieval uses semantic search under the hood, so this filters out memories that are vaguely related but not actually useful. If you’re finding the bot isn’t retrieving memories it should, lower this. If it’s pulling in irrelevant ones, raise it.

The session manager handles everything automatically. When the agent starts processing a message, it retrieves relevant long-term memories and injects them into context. When the conversation ends, it writes events back to short-term memory. You don’t call any memory APIs in your agent logic. It just works.

Giving Users Control: Memory Management Tools

When I first rolled this out, someone asked if the bot could delete wrong long-term memories, and I promised it could. I didn’t realize that was a lie - the Strands bot natively *cannot* delete memory records.

To address users wanting to remove some or all memories, I wrote a few Strands memory tools that let users remove specific or all memories.

I’m really loving strands tools. They’re not MCP based (which is a bummer) but they have deep integration into the user context of who launched them, so I can do sensitive things like fetch and delete memories.

I built these tools:

• `list_my_memory_records` - A user can say “what do you remember about me?” and Vera will list every preference and fact that’s been extracted from their conversations.

• `search_my_memories` - Runs semantic search against stored memories. “Do you remember my standup process?” will find the relevant record even if the stored text is worded differently.

• `get_memory_record` - Retrieves the full details of a specific record by ID.

• `delete_memory_record` - Removes a single memory. “Forget my standup workflow, I have a new process” will find and delete that specific record.

• `delete_all_my_memories_in_namespace` - The nuclear option. This one has a two-step confirmation flow built into the tool itself:

All of these tools are scoped to the current user’s namespace by default:

default_namespace = f”/preferences/{actor_id}”

The same actor ID that scopes memory retrieval also scopes the management tools.

Summary and What’s Next

This article walked through how AgentCore memory works at both layers: short-term events created by each conversation, and long-term records extracted asynchronously by the backend. We covered the user preference strategy and namespace isolation, how the Strands SDK session manager works, and the memory management tools that give users visibility and control over what the bot remembers.

Memory is the feature that transforms Vera from a tool into a teammate. The bot moves on from GenAI Q&A bot to teaching Vera how they work, and Vera remembers. With 2,000 facts learned in the first month across nearly 500 users, it’s clear that people want to teach their bot. They just need the capability.

The code for this entire project is open source: https://github.com/KyMidd/AgentCore_AgenticSlackBot. Feel free to poke around, steal ideas, or open issues when things don’t make sense.

Happy building!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In the last article, we talked about why I migrated from Lambda to AWS Bedrock AgentCore, the architecture with two Lambdas feeding into an AgentCore runtime, and the features you get out of the box.

This article is part of a series:

• Part 1 - AgentCore: Full Architecture and Code

• Part 2 (this article) - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

Now let’s talk about tools.

Vera (our Slack bot) currently has access to tools from PagerDuty, GitHub, Atlassian (Jira + Confluence), Azure, AWS CLI, and Splunk. That’s a lot of tool providers, and each one has its own authentication mechanism, transport protocol, and quirks. Managing all of that in application code can cause a lot of sprawl.

AgentCore’s MCP Gateway is designed to solve this - a single, managed endpoint for all your tool providers. But today, the gateway doesn’t support every auth pattern, so I’m running a hybrid: some tools go through the gateway, others connect directly. This article covers both patterns, how they coexist, and where this is all heading.

If you want a deeper look at MCP tool integration with the Strands SDK specifically, I covered that in a previous article. We’ll be lighter on that here and focus more on the gateway pattern.

Let’s get into it.

Local MCP Connections

Before the gateway existed, every tool provider was a direct MCP connection managed in application code. That’s still how most of Vera’s tools work today, and it’s worth understanding the pattern before we talk about what the gateway changes.

MCP (Model Context Protocol) supports multiple transport types. Vera uses three of them:

Stdio (local process)

The MCP server runs as a subprocess inside the container. AWS CLI MCP works this way—it’s a Python process that Vera spawns and communicates with over stdin/stdout.

# AWS CLI: local process, stdio transport
aws_cli_mcp_client = MCPClient(
  lambda: stdio_client(
    StdioServerParameters(
      command=f”{opt_aws_cli_mcp_dir}/.venv/bin/awslabs.aws-api-mcp-server”,
      env=env,
    )
  ),
  prefix=”aws”,
)

Streamable HTTP (remote)

The MCP server is hosted remotely and you connect over HTTP. GitHub and Splunk use this pattern, each with their own bearer token.

# GitHub: remote, streamable HTTP with bearer token
github_mcp_client = MCPClient(
lambda: streamablehttp_client(
“https://api.githubcopilot.com/mcp/”,
headers={”Authorization”: f”Bearer {github_token}”},
timeout=timedelta(seconds=30),
sse_read_timeout=timedelta(seconds=300),
    ),
tool_filters=tool_filters,
prefix=”github”,
)

SSE (remote, server-sent events)

An older transport pattern. Atlassian’s MCP endpoint still uses this. It has its own token refresh dance—exchange a refresh token for an access token before connecting.

# Atlassian: remote, SSE with refresh token flow
access_token = get_access_token(refresh_token, client_id)
atlassian_mcp_client = MCPClient(
lambda: sse_client(
“https://mcp.atlassian.com/v1/sse”,
headers={”Authorization”: f”Bearer {access_token}”},
timeout=300.0,
    ),
tool_filters=tool_filters,
prefix=”atlassian”,
)

This works. Each client connects independently, each manages its own auth, and they all feed tools into the same agent. But there are friction points:

Credential sprawl. Every provider needs its own secret stored, fetched, and refreshed. GitHub needs a PAT. Atlassian needs a client ID and refresh token. PagerDuty needs an API token. Splunk needs a bearer token. Azure needs a tenant ID, client ID, and client secret. That’s a lot of secrets to manage.

Per-provider auth code. Each provider has its own authentication pattern. Atlassian’s refresh token flow looks nothing like GitHub’s static PAT. Every new provider means writing another auth handler.

No centralized management. Want to see which tools your bot has access to? You need to read the code. Want to add a new provider? You need to write code, build a new container, and deploy. There’s no infrastructure-level view of “here are all the tools.”

The gateway changes this - for providers that support it.

AgentCore MCP Gateway Pattern

The AgentCore MCP Gateway is an AWS-managed service that sits between your agent and your tool providers. Instead of connecting directly to each provider, your agent connects to one gateway endpoint. The gateway handles routing, authentication, and tool discovery.

Tool registration moves from your application code to shared infrastructure. You define providers as gateway “targets” in Terraform. The gateway handles authentication to those providers using credentials stored in AgentCore’s Identity and token vault system. Your application code just connects to the gateway.

If authentication fails, the gateway can remove those tools without crashing your application. If you add tools, the gateway can provide those to any number of applications that use that gateway. This establishes a really scalable pattern for MCPs for your agents.

Here’s what that looks like in practice.

Setting Up Cognito for Gateway Auth

The gateway needs to authenticate your agent. It uses Cognito OAuth2 with the client credentials flow -machine-to-machine auth, no human login involved.

The setup has four pieces, all in Terraform:

1. A Cognito User Pool - the identity authority.

resource “aws_cognito_user_pool” “gateway” {
  name = “${var.gateway_name}-${var.environment}”
  password_policy {
    minimum_length    = 16
    require_uppercase = true
    require_lowercase = true
    require_numbers   = true
    require_symbols   = true
  }
}

2. Cognito Domain - provides the OAuth token endpoint.

resource “aws_cognito_user_pool_domain” “gateway” {
  domain       = lower(”${var.gateway_name}-${var.environment}”)
  user_pool_id = aws_cognito_user_pool.gateway.id
}

3. A Resource Server - defines the OAuth scope your agent needs.

resource “aws_cognito_resource_server” “gateway” {
  identifier   = var.gateway_name
  name         = var.gateway_name
  user_pool_id = aws_cognito_user_pool.gateway.id
  scope {
    scope_name        = “invoke”
    scope_description = “Scope for invoking the ${var.gateway_name} AgentCore gateway”
  }
}

4. An App Client - generates the credentials your agent uses.

resource “aws_cognito_user_pool_client” “gateway” {
name         = “${var.gateway_name}-client-${var.environment}”
  user_pool_id = aws_cognito_user_pool.gateway.id
  # OAuth 2.0 client credentials flow (machine-to-machine)
  generate_secret                      = true
  allowed_oauth_flows_user_pool_client = true
  allowed_oauth_flows                  = [”client_credentials”]
  allowed_oauth_scopes = [
    “${aws_cognito_resource_server.gateway.identifier}/invoke”
  ]
  refresh_token_validity        = 30
  enable_token_revocation       = true
  supported_identity_providers  = [”COGNITO”]
}

The Cognito client ID gets passed to the worker as an environment variable. The client secret goes into Secrets Manager and gets fetched at runtime. The token URL is constructed from the domain:

https://{domain}.auth.{region}.amazoncognito.com/oauth2/token

All of these outputs flow from the gateway module into the worker module:

# main.tf - wiring gateway outputs to worker
module “worker” {
  source = “./worker”
  # Gateway configurations
  gateway_arn       = module.gateway.gateway_arn
  gateway_client_id = module.gateway.cognito_client_id
  gateway_url       = module.gateway.gateway_url
  gateway_token_url = module.gateway.cognito_token_url
  gateway_scope     = module.gateway.gateway_scope
  # ... other config
}

The AgentCore Gateway Resource Itself

With Cognito in place, the gateway resource is straightforward:

resource “aws_bedrockagentcore_gateway” “main” {
  name        = var.gateway_name
  description = “Unified MCP gateway for ${var.runtime_name}”
  protocol_type = “MCP”
  protocol_configuration {
    mcp {
      supported_versions = [”2025-03-26”, “2025-06-18”, “2025-11-25”]
      search_type        = “SEMANTIC”
    }
  }
  # JWT authentication via Cognito
  authorizer_type = “CUSTOM_JWT”
  authorizer_configuration {
    custom_jwt_authorizer {
      discovery_url   = “https://cognito-idp.(region).amazonaws.com/${aws_cognito_user_pool.gateway.id}/.well-known/openid-configuration”
      allowed_clients = [aws_cognito_user_pool_client.gateway.id]
    }
  }
  role_arn        = aws_iam_role.gateway.arn
  exception_level = “DEBUG”
}

There’s a few things to note.

Semantic search

The gateway supports `SEMANTIC` search type, which enables on-demand tool discovery based on what the user is asking rather than loading every tool upfront. In our current architecture, we load all gateway tools at startup since there are only a handful registered. But as more providers migrate to the gateway, on-demand semantic discovery will make more sense than loading hundreds of tools into the agent’s context on every invocation.

MCP protocol versions

The gateway supports multiple MCP versions. This is important because the protocol is evolving and different clients may use different versions.

CUSTOM_JWT authorizer

The gateway validates tokens by fetching the OIDC discovery document from Cognito. No custom validation code needed.

The gateway also needs IAM permissions -access to Secrets Manager for provider credentials, S3 for OpenAPI schemas, and the token vault for outbound authentication:

# Gateway needs access to provider credentials in token vault
resource “aws_iam_role_policy” “gateway_workload_identity” {
  name = “WorkloadIdentityAccess”
  role = aws_iam_role.gateway.id
  policy = jsonencode({
    Version = “2012-10-17”
    Statement = [
      {
        Effect = “Allow”
        Action = [”bedrock-agentcore:GetWorkloadAccessToken”]
        Resource = [
          “arn:aws:bedrock-agentcore:(region):(account ID):workload-identity-directory/default”,
          “arn:aws:bedrock-agentcore:(region):(account ID):workload-identity-directory/default/workload-identity/*”
        ]
      },
      {
        Effect = “Allow”
        Action = [”bedrock-agentcore:GetResourceApiKey”]
        Resource = [
          “arn:aws:bedrock-agentcore:(region):(account ID):workload-identity-directory/default”,
          “arn:aws:bedrock-agentcore:(region):(account ID):workload-identity-directory/default/workload-identity/*”,
          “arn:aws:bedrock-agentcore:(region):(account ID):token-vault/default”,
          “arn:aws:bedrock-agentcore:(region):(account ID):token-vault/default/apikeycredentialprovider/*”
        ]
      }
    ]
  })
}

PagerDuty on the Gateway

You thought this would be intuitive? lol

Registering PagerDuty as a gateway target looks simple in Terraform:

resource “aws_bedrockagentcore_gateway_target” “pagerduty” {
  gateway_identifier = aws_bedrockagentcore_gateway.main.gateway_id
  name               = “pagerduty”
  target_configuration {
    mcp {
      open_api_schema {
        s3 {
          uri = “s3://amazonbedrockagentcore-built-sampleschemas455e0815-oj7jujcd8xiu/pagerduty-open-api.json”
        }
      }
    }
  }
  credential_provider_configuration {
    api_key {
      provider_arn        = “arn:aws:bedrock-agentcore:(region):(account ID):token-vault/default/apikeycredentialprovider/pagerduty-vera-read-only”
      credential_prefix   = “Token”
      credential_location = “HEADER”
    }
  }
}

The URI I only got by building it by hand in the console and then importing it ¯\_(ツ)_/¯

Looks clean. But there are two things worth calling out.

First, the `provider_arn` references a credential provider in AgentCore’s token vault. That credential provider - an “outbound authentication” item in AgentCore Identity - has to be created separately. I did this manually through the AWS console rather than Terraform, because the token vault stores actual API keys and I didn’t want those in Terraform state. I always keep secrets separate from Terraform when I can.

The workflow is:

1. Manually create an outbound authentication credential provider in AgentCore Identity

2. Store your PagerDuty API token in that credential provider

3. Reference the credential provider’s ARN in your Terraform gateway target configuration

4. Deploy the gateway and target with Terraform

This separation makes sense from a security perspective - infrastructure definition stays in code, secrets stay out of it. You’re wiring together three things: the gateway target, a credential provider in the token vault, and the actual API key stored in that provider.

And then it didn’t work. Sweet. Turns out that even though this is an Integration, which I would think implies this is a vetted and tested pattern from AWS, it isn’t built correctly. Most APIs that auth with a bearer token use a header key of “bearer”, but PagerDuty doesn’t - it uses “Token”.

You can totally fix this in the Gateway Target (yay!) but I didn’t expect the special integration with PagerDuty to be broken out of the box (lol, boo).

Overrode the header value with credential_prefix = “Token” to fix it. Until we did, every PagerDuty API call failed with 401.

For an Integration that’s supposed to be turnkey, having to dig into PagerDuty’s auth docs to figure out their non-standard header format was a surprise.

Token Caching Keeping the Gateway Fast

Once the gateway is set up, your agent needs to authenticate to it on every request. That means fetching a JWT from Cognito. You don’t want to hit Cognito on every single tool call, so we cache tokens in memory.

# Token cache with expiry tracking
_token_cache = {”token”: None, “expires_at”: 0}
# Refresh 5 minutes before expiry to prevent race conditions
TOKEN_REFRESH_BUFFER = 300
def get_gateway_token(secrets_json):
    current_time = time.time()
# Check if token is expired or near expiry
if current_time >= (_token_cache[”expires_at”] - TOKEN_REFRESH_BUFFER):
        client_id = os.environ.get(”GATEWAY_CLIENT_ID”)
        client_secret = secrets_json.get(”GATEWAY_CLIENT_SECRET”)
# Fetch new token from Cognito
        token_data = _fetch_new_token(client_id, client_secret)
# Update cache
        _token_cache[”token”] = token_data[”access_token”]
        _token_cache[”expires_at”] = current_time + int(
            token_data.get(”expires_in”, 3600)
        )
return _token_cache[”token”]

The 5-minute buffer (`TOKEN_REFRESH_BUFFER = 300`) is important. Without it, you’d have a window where the token is technically valid when you check but expires before the request completes. Cognito tokens typically last an hour, so refreshing at 55 minutes is plenty safe.

This caching is especially effective in AgentCore because the container stays alive between requests. In Lambda, the cache only survived if you hit a warm container. In AgentCore, the cache persists for the lifetime of the runtime -up to 8 hours. One Cognito call can serve dozens of agent invocations.

The token gets injected into the transport via a factory function:

def create_transport():
“”“Create streamable HTTP transport with fresh JWT token”“”
    token = get_gateway_token(secrets_json)
return streamablehttp_client(
        gateway_url,
headers={”Authorization”: f”Bearer {token}”},
timeout=timedelta(seconds=30),
sse_read_timeout=timedelta(seconds=300),
    )
gateway_mcp_client = MCPClient(
    create_transport,  # Factory called each time transport is needed
tool_filters=tool_filters,
)

The factory pattern means each new transport connection gets a fresh (or cached) token automatically. No manual token management in the agent logic.

Combine MCP Client Tools Together

Today, Vera uses both patterns simultaneously. The agent initialization in `execute_agent()` builds a single tool list from all sources:

def execute_agent(secrets_json, conversation, memory_config=None):
    tools = []
    opened_clients = {}
# Built-in tools (no MCP, bundled with Strands SDK)
from strands_tools import calculator, current_time, retrieve
    tools.extend([calculator, current_time, retrieve])
# Gateway MCP (PagerDuty today, more providers later)
try:
        gateway_mcp_client = build_gateway_mcp_client(secrets_json, mode=”read_only”)
        opened_clients[”Gateway”] = gateway_mcp_client
        tools.append(gateway_mcp_client)
except Exception as error:
print(f”Error setting up gateway MCP client: {str(error)}”)
# Direct MCP: GitHub
try:
        github_mcp_client = build_github_mcp_client(secrets_json[”GITHUB_TOKEN”], “read_only”)
        opened_clients[”GitHub”] = github_mcp_client
        tools.append(github_mcp_client)
except Exception as error:
print(f”Error setting up GitHub MCP client: {str(error)}”)
# All others MCPs also - Azure, AWS, etc.
# Build agent with ALL tools from ALL sources
    agent = Agent(
model=BedrockModel(model_id=model_id, ...),
system_prompt=system_prompt,
tools=tools,
    )

The agent doesn’t know or care whether a tool came from the gateway or a direct connection. It sees a flat list of tools and uses whatever’s relevant to the user’s question. Ask about PagerDuty incidents? The gateway-routed PagerDuty tools handle it. Ask about a GitHub PR? The direct GitHub MCP client handles it.

The try/except means that if Atlassian’s MCP server is having a bad day, the agent still has PagerDuty, GitHub, AWS, and Splunk. Graceful degradation means you lose one provider, not all of them.

Each MCP target’s configuration is in its own file so we can do tool filtering (Vera is read-only) and makes it easy to sort out targets and authentication.

The Future: When the Gateway Absorbs Everything

Right now, the AgentCore MCP Gateway supports “Integrations” -providers that AWS has specifically built support for. PagerDuty is one of these. The gateway knows how to authenticate to PagerDuty, how to parse its OpenAPI schema, and how to route tool calls.

But GitHub, Atlassian, Splunk, and Azure aren’t Integrations yet. They use bearer token authentication over streamable HTTP or SSE, and the gateway’s credential provider system doesn’t support generic bearer token auth for arbitrary MCP endpoints today. That’s why these providers still need direct connections.

When AWS adds broader bearer token support to gateway targets -and the architecture is clearly heading that direction -the picture changes dramatically:

Today (hybrid):

Agent → Gateway → PagerDuty (Integration)
Agent → Direct  → GitHub (bearer token, streamable HTTP)
Agent → Direct  → Atlassian (bearer token, SSE)
Agent → Direct  → Splunk (bearer token, streamable HTTP)
Agent → Direct  → AWS CLI (local stdio)

Future (consolidated):

Agent → Gateway → PagerDuty
Agent → Gateway → GitHub
Agent → Gateway → Atlassian (Jira + Confluence)
Agent → Gateway → Splunk
Agent → Direct  → AWS CLI (local stdio, will always be direct)

When this migration is fully done, we’ll have:

• IAM-only auth - each agent runtime won’t have to manage MCP target authentication secrets at all

• Infrastructure-level tool management. Adding a new provider means adding a Terraform resource - a new gateway target. No application code changes at all.

• Centralized tool discovery. The gateway knows about all registered tools across all providers. With semantic search, the agent can find the right tool even if it doesn’t know the exact name.

What’s Next

This article covered how Vera connects to its tool providers -the three MCP transport patterns, the AgentCore gateway as a managed alternative, the Cognito auth setup, and the PagerDuty configuration gotchas. We’re running a hybrid today: gateway where AWS supports it, direct connections everywhere else.

In the next article, we’ll cover AgentCore Memory - how Vera remembers things across conversations. Not just conversation history, but actual persistent knowledge: user preferences, learned facts, and session summaries. It’s one of the features that makes the AgentCore model feel genuinely different from Lambda.

The code for this entire project is open source:

GitHub/KyMidd/AgentCore_AgenticSlackBot

Feel free to poke around, steal ideas, or open issues when things don’t make sense.

Happy building!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

It’s been a while! I’ve been busy migrating Vera, my agentic AI helper, over to AWS AgentCore.

This article is part of a series:

• Part 1 (this article) - AgentCore: Full Architecture and Code

• Part 2 - AgentCore MCP Gateway

• Part 3 - AgentCore Memory and Tools

• Part 4 - AgentCore Deployment, Operations, and Lessons Learned

AgentCore has a few very cool features that made it attractive for migration, including:

• Longer running jobs: Can run for 8 hours! Much longer than Lambda’s 15 minute hard limit

• Avoiding cold starts: Lambda starts from scratch each time (with some exceptions for warm more expert-tier stuff), so having the bot ready to go is helpful

I’ve been building AI bots for enterprise use for a while now, and if you’ve been following along, you’ve seen the evolution. We started with a genAI Slack bot (https://www.letsdodevops.com/p/solving-aws-bedrocks-enterprise-logging) that could answer questions using RAG and knowledge bases. Then we leveled it up to an agentic Slack bot (https://www.letsdodevops.com/p/building-agentic-slack-bot) that could actually do things - query GitHub, check PagerDuty, search Jira.

Here’s the thing: that Lambda-based agentic bot? It works great. Seriously. Lambda with the Strands SDK running Claude can handle complex multi-tool workflows without breaking a sweat. If you’re building an agentic bot today and Lambda fits your needs, go for it.

But.

Lambda has a hard 15-minute timeout. For most agentic workflows, that’s plenty. But what happens when someone asks your bot to analyze a quarter’s worth of incidents, cross-reference them with deployment logs, and generate a report? Or when the agent needs to iterate through dozens of GitHub repos looking for a specific pattern? Those workflows can push past 15 minutes, and when Lambda hits that wall, it hits it hard.

Beyond the timeout, I kept hearing about AWS Bedrock AgentCore and its growing feature set. Memory that persists across conversations? A managed MCP gateway for tool access? Multi-agent orchestration? These aren’t things you can’t build on Lambda, but they’re things AgentCore gives you out of the box.

So I decided to migrate. Not because Lambda failed me, but because AgentCore offered a different model - one where my bot isn’t a function that spins up and dies, but an application that stays warm, maintains state, and processes requests like a real service.

This is the first article in a series documenting that migration. We’ll cover:

1. This article: What AgentCore is and why the “living application” model matters

2. MCP Gateway - one gateway to rule all your tools

3. AgentCore Memory - teaching your bot to remember

4. Guardrails and Knowledge Bases - safety and smarts

5. Deployment, operations, and lessons learned

Let’s do it.

If you want to skip all this architecture and just see the code, it’s all published here under an MIT license:

GitHub.com/KyMidd/AgentCore_AgenticSlackBot

The Problem with Lambda for AI Agents

Don’t get me wrong - Lambda is fantastic for AI workloads. I’ve been running agentic bots on Lambda for months, and the model works well:

• Pay-per-invocation: You only pay when the bot is actually working

• Auto-scaling: Slack blows up with requests? Lambda handles it

• Simple deployment: Push code, done

But there are friction points when you’re running long-lived agentic workflows:

The 15-minute ceiling. Most agent tasks complete in 2-5 minutes. But “most” isn’t “all.” When your agent needs to iterate through large datasets, make dozens of tool calls, or wait for slow external APIs, you can hit that timeout. And when you do, everything stops. No graceful degradation, no “let me finish this thought” - just death.

Cold starts during conversations. Lambda containers can be reused, but there’s no guarantee. If your user sends a follow-up message and it hits a cold container, you’re paying the initialization cost again - reloading models, re-establishing MCP connections, re-fetching secrets. It’s not catastrophic, but it’s not elegant either.

No persistent state. Every Lambda invocation starts fresh. Want to remember what you talked about 5 minutes ago? You need to build that yourself - fetch conversation history, reconstruct context, hope you got it right. Lambda doesn’t help you here.

The invoke-and-die pattern. This is the big mental model issue. Lambda wants you to do a thing and exit. Agents want to exist - to maintain connections, hold context, be ready for the next question. These models are fundamentally different.

None of these are dealbreakers. I worked around all of them. But when AgentCore offered a different approach, I was curious.

Enter AgentCore: An Application, Not a Function

AWS Bedrock AgentCore flips the model. Instead of invoking a function that dies after each request, you’re running a container that stays alive.

Think of it like this:

| Aspect | Lambda | AgentCore |
|--------|--------|-----------|
| Lifecycle | Invoke → Execute → Die | Start → Stay warm → Process many requests → Eventually die |
| State | None (rebuild each time) | Maintained in memory |
| Timeout | 15 minutes hard limit | Configurable (hours) |
| Connections | Re-establish each invocation | Keep alive |
| Mental model | Function | Application |

Your AgentCore runtime is a Docker container running on AWS-managed infrastructure. You define how long it can sit idle before AWS kills it (idle_timeout), and how long it can live total (max_lifetime). Between requests, it just... waits. Connections stay open. Memory stays loaded. When the next request comes in, you’re already warm.

This is a fundamental shift in how you think about your bot. You’re not writing a function anymore. You’re writing an application - one that handles HTTP health checks, processes incoming requests on a loop, and maintains state between calls.

The lambda pattern is to handle one single event and then die. The AgentCore pattern is your application spins up at the first request, and then it hangs around to handle other requests.

It’s fully warm, tokens are cached, MCP clients are prepped, and so we now need to understand which pieces of the application are shared vs distinct, and even which pieces of data are sensitive and need to be kept separate between requests.

The runtime stays warm based on your configuration:

# Runtime lifecycle configuration
idle_timeout = 900   # 15 minutes idle before shutdown
max_lifetime = 28800 # 8 hours max before forced restart

AWS manages the health checks. If your container becomes unhealthy, AgentCore restarts it. If traffic spikes, AgentCore can scale. You focus on the application logic; AWS handles the infrastructure.

Architecture Overview

Here’s the thing about AgentCore: you can’t just point Slack directly at it. Slack requires a webhook response within 3 seconds, and your AI agent is definitely not responding in 3 seconds. So we still need Lambda - but in a different role.

The architecture looks like this:

Receiver Lambda

Catches the Slack webhook, validates the signature, and immediately returns 200. Slack is happy. Then it asynchronously invokes the Invoker Lambda. This function is tiny and fast - 128MB, sub-second execution.

Invoker Lambda

Takes the payload and calls InvokeAgentRuntime. This is a synchronous call to AgentCore, but the Invoker doesn’t wait for it to complete - it fires and forgets. The Invoker exists mainly to provide IAM isolation; it has only the permission to invoke AgentCore, nothing else.

AgentCore Runtime

The actual bot. Receives the request, loads conversation history, spins up MCP connections, runs the agent loop with Claude, and posts the response directly back to Slack. This is where the magic happens, and it can take as long as it needs.

Why the two-Lambda dance? Separation of concerns. The Receiver needs to be fast and needs access to validate Slack signatures. The Invoker needs permission to call AgentCore. By splitting them, we follow least-privilege principles and keep each component focused.

What You Get Out of the Box

AgentCore isn’t just “Lambda but longer.” It comes with a suite of features that would take significant effort to build yourself:

MCP Gateway

A managed gateway for MCP (Model Context Protocol) servers with OAuth2 authentication via Cognito. Register your tools once, authenticate once, access everything. We’ll dive deep into this in Article 2.

AgentCore Memory

Persistent memory across conversations. Not just “what did we talk about in this thread” but “what are this user’s preferences” and “what facts have we learned.” Three memory strategies out of the box: session summaries, user preferences, and semantic knowledge. Article 3 covers this in detail.

Multi-Agent Orchestration

This one’s interesting. AgentCore supports agent-to-agent communication and hierarchical agent patterns - think a “manager” agent that delegates to specialist agents. I haven’t implemented this yet, but the capability is there. A supervisor agent could coordinate between a “GitHub specialist” and a “Jira specialist,” each with their own tools and context. Something to explore in a future article.

Container Lifecycle Management

AWS handles health checks, restarts, and scaling. You define the parameters; AWS keeps things running.

The Entrypoint Pattern: From Handler to Application

We were already using Docker containers for the Lambda-based bot, so the Dockerfile doesn’t change dramatically. What changes is how the code runs inside that container.

With Lambda, you write a handler function that processes one event:

def lambda_handler(event, context):
# Process this one request
# Return response
# Container may or may not stick around

With AgentCore, you write an application that stays alive and processes many requests:

from strands.multimodal.application import Application
app = Application()
@app.entrypoint
async def handle_slack_message(payload: dict):
# Process this request
# Container keeps running after we return
pass
if __name__ == “__main__”:
    app.run()  # Start HTTP server, wait for requests

That app.run() call is the key difference. It starts an HTTP server inside the container that:

• Responds to health checks from AgentCore

• Routes incoming requests to your handler

• Keeps the process alive between requests

Your handler gets called for each incoming request, but the container - and everything you’ve initialized - stays warm. MCP connections persist. Cached tokens stay valid. The mental model shifts from “spin up, do work, die” to “stay ready, handle work as it comes.”

What’s Next

This article covered the “why” and “what” of AgentCore. We talked about Lambda’s limitations for long-running agents, how AgentCore’s application model differs, and what features come bundled in.

In the next article, we’ll go deep on the MCP Gateway - how to set up unified authentication with Cognito, register multiple tool providers, and handle the quirks of different MCP servers (spoiler: Azure MCP on ARM64 is... special).

Until then, if you’re running agentic bots on Lambda and they’re working fine, don’t let this article convince you to migrate for migration’s sake. But if you’re hitting the 15-minute wall, or you want persistent memory, or you’re tired of rebuilding MCP clients on every invocation - AgentCore is worth a look.

The code for this entire project is open source: [GitHub link]. Feel free to poke around, steal ideas, or open issues when things don’t make sense.

Happy building!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

Here’s the thing about building custom solutions: they work great until the library you depend on ships an update. Then you’re stuck choosing betIen “maintain my clever workaround” or “rewrite everything to match the new API.” Neither option sounds fun.

VeraResearch is an agentic Slack bot built with AWS Strands that connects to six different MCP (Model Context Protocol) servers—GitHub, Atlassian, PagerDuty, Azure, AWS CLI, and Splunk. Each MCP server provides tools the agent can use to ansIr questions and complete tasks. Pull incident data from PagerDuty. Search Jira tickets. List AWS resources. You get the idea.

When you load multiple MCP servers, they inevitably have overlapping tool names. GitHub has get_user. PagerDuty has get_user. The next MCP you install probably has get_user. Without prefixing, the agent crashes when it tries to load the second one.

So I built a solution: a custom PrefixedMCPTool wrapper class that manually prefixed every tool name. github_get_user, pagerduty_get_user, azure_get_user. Problem solved. It worked perfectly.

Except now I owned ~75 lines of code that reached into MCPClient internals, manually managed client lifecycles, extracted tool lists, wrapped them, and returned tuples. Every time Strands updated how MCPClient worked, I had to check if our custom wrapper still functioned. Maintenance debt.

Then Strands PR #895 landed: first-class support for tool prefixing and filtering directly in MCPClient. The library now provides prefix and tool_filters parameters. Just pass a prefix string and optional tool filter instructions, and MCPClient handles everything.

If I’m maintaining custom abstractions, I’m on my own when breaking changes happen. If Strands provides the feature natively, their updates won’t break my code because they own the abstraction.

Let’s walk through removing my custom cruft and use the code the Strands maintainers team wrote.

To skip right to the code changes, see the commit here.

The Old Way: Custom Tool Wrapping with PrefixedMCPTool

The custom solution had three parts: a wrapper class, builder functions for each MCP server, and consumption code in worker_agent.py.

Here’s the PrefixedMCPTool wrapper class:

Every MCP client builder followed the same pattern. Here’s GitHub:

Then in worker_agent.py, I consumed these tuples:

This pattern repeated six times for six MCP servers.

The problems: I’m calling __enter__() to manually start clients, but Agent expects unstarted clients and calls load_tools() itself. I’m reaching into MCPClient internals with list_tools_sync(). Every builder function returns a tuple that needs unpacking, prefixing, and extending. When Strands changes MCPClient behavior, this breaks.

The New Way: What Strands Added and What I Deleted

Strands PR #895 added three things to MCPClient: a prefix parameter, a tool_filters parameter, and ToolProvider interface support. Each addition let me delete chunks of custom code.

Prefix Parameter

Strands now adds a prefix to every tool name automatically. Pass prefix=”github” and the client handles the rest.

Before:

After:

I deleted the entire PrefixedMCPTool class (25 lines) and the add_prefix_to_mcp_tools() function.

tool_filters Parameter

Strands now filters tools using lambda functions you provide. Want only read-only tools? Pass a filter that checks tool names.

Before:

After:

I deleted the manual filtering loops from six MCP client files.

ToolProvider Interface

MCPClient now implements ToolProvider, which means Agent starts the client automatically when it needs tools. I don’t call __enter__() or list_tools_sync() anymore.

Before:

After:

Then in worker_agent.py, instead of unpacking tuples and extending tool lists, I just append the client:

Before:

After:

Agent calls load_tools() on the client when it needs to, which starts the connection and extracts tools automatically.

Gotchas

• Prefix separator: Strands adds an underscore between prefix and tool name automatically. I initially set TOOLS_PREFIX = “pagerduty_” (with trailing underscore) and got tool names like pagerduty__list_incidents (double underscore). Remove the trailing underscore from your prefix constants.

• Filter timing: tool_filters run after the prefix is applied. Check for github_get_user, not get_user.

Summary

In this blog we walked through how to move from my custom-built MCP management interface to the MCP tool management pattern that the Strands maintainers built. It’s both less code for us to manage, as well as quite a bit faster to get all those MCPs built in parallel instead of in sequence.

Remember, every custom abstraction is maintenance debt when the upstream library changes. When the library ships the feature natively, future updates are forward compatible by default.

Always let someone else do the hard work.

Full code here: https://github.com/KyMidd/SlackStrandsAgenticBot

Good luck out there!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

We’ve built three AI bots across our organization: VeraSlack, VeraTeams, and VeraResearch. All three run on AWS Bedrock using Claude Sonnet. They answer questions, search knowledge bases, and help employees find information.

When we first deployed these bots, we relied on AWS Bedrock’s built-in logging. Every API call generates a log entry with timestamps and request IDs. Our Lambda functions logged to CloudWatch. We figured we had full visibility. If security needed to audit a conversation, they’d just look at the logs.

The first time our security team asked “What did Bob from Finance ask the bot last Tuesday?” we couldn’t answer it. We had thousands of Bedrock API log entries, but no way to connect them to Bob’s question.

An employee asks: “What’s our process for requesting SSL certificates?” That single question triggers the bot to query the knowledge base, call a reranking service, search Confluence, check PagerDuty, maybe query Jira, synthesize the results with Bedrock, and format a response. That’s fifteen separate API calls across multiple services.

Each API call logs separately. Each log entry is atomic—just a request ID, timestamp, and parameters. No user name. No original question. No conversation context.

To reconstruct what Bob asked, you’d need to find his username in Slack’s event logs, correlate that to a Lambda execution timestamp, connect it to fifteen different Bedrock API calls scattered across log groups, and piece together the conversation.

Pro tip: This is awful, I don’t want to spend all my time reading logs

The problem gets worse with conversational bots. VeraResearch uses Bedrock Agents and can have multi-turn conversations. A five-minute troubleshooting session generates hundreds of log entries. Bedrock logs API requests, but has no concept of “user,” “conversation,” or “session.”

We needed to answer basic questions: What is Bob asking? What information are we providing about sensitive topics? Which bot is being used most? Bedrock’s logging couldn’t answer these.

So we built our own audit logging system. One that captures user identity, the original question, conversation context, the bot’s response, and ties it together with a single session ID.

In this article, I’ll walk you through why platform logging fails for agentic bots, how we designed an audit system across Slack and Teams, and the platform-specific challenges we solved.

It’s Harder Than It Looks

Good audit logging needs specific fields: who asked (user_id), what they asked (user_query), the conversation context, what we told them (response), when it happened (timestamp), and which bot answered (bot_name). We also need the AWS account ID for environment tracking and a session ID that ties audit logs directly to Lambda execution logs.

We’re writing these logs to CloudWatch Logs using a shared log group called /aws/ai-bots/audit-logs. Each bot gets its own log stream pattern: VeraSlack/{session_id}, VeraTeams/{session_id}, VeraResearch/{session_id}. The session ID is the Lambda log stream name, which means we can correlate audit logs with execution logs instantly. No more hunting through multiple log groups trying to piece together what happened.

Here’s where it gets interesting: Slack and Teams are completely different platforms with different user identity models.

Slack is straightforward

When a user messages the bot, we get a Slack event that includes their user ID. We already call the Slack API’s users.info endpoint to fetch the user’s profile for conversation building. That API returns a username like kyler.middleton. We just reuse that data for audit logging. No extra API calls, no complexity.

Teams is … different

Honestly, I would have been surprised if Teams made this easy. Read through the whole Teams Vera series to learn all the awful patterns I had to build to make it work. This one isn’t as terrible as all that, but still not great.

When a user messages the bot through Microsoft Teams, we get a Bot Framework event with an Azure AD Object ID—basically a GUID like d4bxxxx-xxxx-xxxx-xxxx-xxxxxxxxcbb. We also get a display name like “Kyler Middleton,” but that’s not useful for audit correlation. Display names aren’t unique and they change.

To get a consistent username from Teams, we need to call the Microsoft Graph API with that Azure AD Object ID. The Graph API returns the User Principal Name, which is typically an email address: Kyler.Middleton@veradigm.me. That’s better, but still not consistent with Slack’s format.

So we normalize it: split on the @ symbol, take everything before it, convert to lowercase. Kyler.Middleton@veradigm.me becomes kyler.middleton. Now it matches Slack’s username format, and we can search audit logs across platforms using consistent identifiers.

The normalization matters more than you’d think. When security asks “show me everything Bob asked across all bots,” they shouldn’t need to know that Bob is bob.smith in Slack but Bob.Smith@veradigm.me in Teams. One username format, searchable everywhere.

This also means we’re making an extra API call for every Teams interaction—but we do it after we’ve already responded to the user, so there’s no latency impact on the conversation.

Lets Build It!

The infrastructure setup is straightforward. We grant the Lambda IAM role write permissions to the shared CloudWatch log group `/aws/ai-bots/audit-logs`. All three bot Lambdas can write to this one log group, but they use different log stream prefixes to keep their logs separate.

The core audit logging function is identical across all three bots. It extracts the AWS account ID from the Lambda context, builds a structured log entry, and writes to CloudWatch:

The function handles errors silently—if CloudWatch is having issues, we don’t want to fail the user’s request.

For Slack bots

We already have the user information from building the conversation. We extract the username and flatten the conversation history:

For Teams bots

We need the extra Graph API call to fetch the User Principal Name and normalize it:

But doesn’t that slow the response down?

To avoid introducing latency in the response, we log after responding to the user. The bot sends its response first, then writes the audit log. This keeps the user experience fast. If the Graph API is slow or CloudWatch has issues, the user never sees it.

Lessons Learned

The biggest lesson I took from this is that the platform’s (read: AWS Bedrock) logs were good enough for all our purposes. Even with great search tools like Splunk or Elastic/ELK, if the data isn’t in your logs, you can’t get to it.

So we had to build it ourselves.

Username normalization matters more than you’d expect. We could have just logged whatever each platform gave us, but then searching across platforms becomes painful.

Session correlation means it’s easier to dig in if there is more detail needed. The audit log contains the full cloudtrail log group name for us to click in and read through.

Log after the user response is posted so we introduce literally no latency in the user experience.

Audit error logging should be invisible to the user - just error log for admins.

AI compliance isn’t just about model outputs. It’s about understanding who’s asking what, what information we’re providing, and being able to trace any conversation from question to answer.

Summary

In this article we talked about how Bedrock doesn’t suite our needs for audit logging, and what we’d really like instead - one cloudtrail group, shipped to Splunk or ELK, that contains all the data we need (and only the data we need) in one place.

We linked each summary / audit log to the deep log group trail we need if we want to dig further

And we shared all the code to show you how to do it!!

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In this blog post, we’re going to integrate the AWS Labs CLI MCP server with a Strands-powered AI agent running in Lambda, with support for querying resources across multiple AWS accounts using profile-based role assumption.

The specific challenge: we need our AI assistant to query AWS resources across many different accounts (dev, staging, prod, ops, etc.)… with an MCP server that appears built to support a single authentication profile, in a single account.

The Multi-Account Challenge

When you ask an AI agent “List the EKS clusters in the Prod account,” the agent needs to:

1. Determine which AWS account “Prod” refers to

2. Assume a role in that account with read permissions

3. Execute the AWS API call with the assumed credentials

4. Return the results

The obvious approach is to use `aws sts assume-role` to get temporary credentials, then use those credentials in subsequent commands. This doesn’t work with the AWS CLI MCP server because it’s stateless - each command execution creates a fresh boto3 session. The MCP server can’t “remember” credentials from a previous assume-role call.

Bummer

Another approach would be creating a separate MCP client instance, one per account we might need to speak to, each configured with a different AWS profile. This creates tool sprawl and of course doesn’t scale well.

Double bummer

What we need is profile-based role assumption where the bot includes `--profile oak` in its AWS CLI commands, and the profile configuration automatically handles assuming the correct role using the Lambda execution role as the base credential source.

Why This Is Complicated

The AWS CLI MCP server uses boto3, not shell commands. When you pass `aws eks list-clusters --profile oak`, the MCP server:

1. Parses the command string to extract the profile name

2. Creates a boto3 Session with that profile

3. Loads profile configuration from the AWS config file

4. Resolves credentials according to the profile’s settings

5. Executes the API call

Each of these steps has failure modes that produce silent errors - boto3 just fails without helpful error messages. We’ll cover the debugging later.

The key insight: the `--profile` flag does work in the AWS CLI MCP server. It extracts the profile from the parsed command and passes it to `boto3.Session(profile_name=...)`. We just need to configure profiles correctly.

Lets do it!

Understanding How the AWS CLI MCP Works

The first thing to understand is that the AWS CLI MCP server doesn’t shell out to the AWS CLI binary. It uses boto3 directly. This is important because it means we need to think about boto3 session management, not shell command execution.

When you send `aws eks list-clusters --profile oak` to the MCP server, heres what actually happens under the hood:

The server parses the command string and extracts the profile name from the `--profile` flag. I actually went digging through the source code to confirm this because the documentation wasn’t clear. In the parser.py file, theres code that does:

profile = getattr(global_args, ‘profile’, None)

Then in driver.py, when creating credentials:

credentials = credentials or get_local_credentials(
  profile=translation.command.profile or AWS_API_MCP_PROFILE_NAME
)

This means the `--profile` flag IS recognized and used. The server creates a fresh boto3 session for each command using that profile name.

Great, so profiles work

Now we need to understand where boto3 gets profile information from.

AWS Config File vs Credentials File

Boto3 looks in two places for configuration:

1. `~/.aws/credentials` - Contains access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)

2. `~/.aws/config` - Contains profile settings like region, role_arn, and credential sources

By default, boto3 only reads the credentials file. To make it read the config file, you need to set the environment variable `AWS_SDK_LOAD_CONFIG=1`.

This is critical and not obvious

Our profiles with role assumption settings live in the config file, so without this environment variable, boto3 cant see them.

Lambda Credentials and credential_source

When your Lambda function runs, AWS provides temporary credentials via environment variables:

- AWS_ACCESS_KEY_ID

- AWS_SECRET_ACCESS_KEY

- AWS_SESSION_TOKEN

These credentials belong to the Lambda execution role. In our case, thats the role `Ue1LiVeraResearchTestWorkerRole`.

In the AWS config file, you can tell a profile to use credentials from the environment with:

[profile oak]
role_arn = arn:aws:iam::123456789:role/read-only-role
credential_source = Environment

This tells boto3: “Get the base credentials from environment variables, then use those to assume the role specified in role_arn.”

The Two-Step Credential Flow

Heres the complete flow:

1. Lambda execution role provides credentials via environment variables

2. Bot sends: `aws eks list-clusters --profile oak`

3. MCP server extracts `--profile oak` from the command

4. Boto3 creates a session with `profile_name=’prod’`

5. Boto3 reads `~/.aws/config` and finds `[profile oak]`

6. Sees `credential_source = Environment` and loads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN from environment

7. Uses those credentials to call `sts:AssumeRole` for the role_arn

8. Gets back temporary credentials for the assumed role

9. Executes `eks:ListClusters` with the assumed role credentials

The key insight here is that credential_source = Environment lets us chain roles: Lambda role → assumed role.

What About source_profile?

You might think you could do this instead:

[profile default]
credential_source = Environment
[profile oak]
role_arn = arn:aws:iam::123456789:role/read-only-role
source_profile = default

This doesn’t work. When a profile has `credential_source = Environment`, it cant be used as a source_profile. AWS SDK documentation mentions this but its easy to miss.

I spent hours debugging this

Each profile needs its own `credential_source = Environment` setting directly. No chaining through a default profile.

Now that we understand how boto3 resolves profiles and credentials, lets build the actual implementation.

Building the Solution

We need to configure everything so that when the bot says `aws eks list-clusters --profile oak`, it automatically assumes the correct role in the Oak account and returns results.

Heres what we need to build:

1. An AWS config file with profiles for each account

2. A Dockerfile that pre-installs the MCP server and copies the config

3. A Python worker module that sets up the MCP client with correct environment variables

4. IAM permissions on both sides of the trust relationship

Lets go through each piece.

The AWS Config File

Create a file called `aws_config` in your Lambda project:

# AWS Config for Multi-Account Access

[profile oak]
role_arn = arn:aws:iam::111111111111:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

[profile birch]
role_arn = arn:aws:iam::222222222222:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

[profile palm]
role_arn = arn:aws:iam::333333333333:role/read-only-role
role_session_name = MyBot
region = us-east-1
credential_source = Environment

Key points:

- Each profile is self-contained with `credential_source = Environment`

- No `[profile default]` needed

- `role_arn` points to the read-only role in each target account

- `role_session_name` can be anything - its useful for CloudTrail auditing

- Replace the account IDs with your actual account IDs

The Dockerfile

We need to pre-install the AWS CLI MCP server and copy our config file:

# Pre-install AWS CLI MCP server

RUN mkdir -p /opt/aws-cli-mcp-server && \
 cd /opt/aws-cli-mcp-server && \
 uv venv && \
 uv pip install --python .venv/bin/python awslabs.aws-api-mcp-server && \
 chmod -R a+rX /opt/aws-cli-mcp-server

# Copy AWS CLI config with pre-configured profiles
COPY aws_config /opt/aws_config

Why `/opt` instead of `/tmp`? Lambda clears `/tmp` on every cold start, but I can read from `/opt`. The worker will copy it to `/tmp` at runtime since boto3 needs write access to the directory.

Also note: I tried copying the entire venv to `/tmp` and ran out of disk space. Lambda gives you 512MB in `/tmp` by default and the MCP server venv is huge. Running from `/opt` works fine.

The Worker Module

Create `worker_mcp_aws_cli.py`:

Critical parts:

1. Copy config to /tmp - boto3 needs the config file in a writable location

2. AWS_SDK_LOAD_CONFIG=1 - Without this, boto3 ignores the config file entirely

3. Pass through credentials - The subprocess doesn’t inherit environment automatically when you provide a custom env dict. You must explicitly pass AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN

This one got me for like 2 hours. Most lambda processes automatically inherit the IAM role assigned to the lambda. Not this tool, BIG SIGH

I initially didn’t pass through the credentials and boto3 just silently failed with no helpful error messages.

Integrating with Strands

In your main agent setup (like `worker_agent.py`), integrate the MCP client:

from worker_mcp_aws_cli import build_aws_cli_mcp_client
# Build AWS CLI MCP client
aws_cli_mcp_client, aws_cli_tools = build_aws_cli_mcp_client(aws_region=”us-east-1”)
# Prefix tool names to avoid collisions
aws_cli_tools = add_prefix_to_mcp_tools(aws_cli_tools, “aws”)
# Add to agent tools
tools.extend(aws_cli_tools)

The tool prefixing gives you `aws_call_aws` and `aws_suggest_aws_commands` instead of just `call_aws`. This prevents collisions if you have multiple MCP servers.

IAM Permissions - Both Sides Must Agree

For cross-account role assumption to work, you need permissions configured on both sides.

So the IAM role you assign to the lambda must permit assuming an IAM role, and the IAM role must permit you to assume it. If either is wrong, the assuming doesn’t work.

This is hard enough to debug in a regular application, but a HUGE pain when your AI eats errors and doesn’t print them

Testing It

At this point, you should be able to deploy your Lambda and test:

Bot receives: “List EKS clusters in the Oak account”

Bot constructs: `aws eks list-clusters --region us-east-1 --profile oak`

MCP server:

1. Parses `--profile oak`

2. Loads `/tmp/.aws/config`

3. Finds `[profile oak]` with `credential_source = Environment`

4. Gets Lambda credentials from environment variables

5. Calls `sts:AssumeRole` for `arn:aws:iam::111111111111:role/read-only-role`

6. Gets temporary credentials

7. Executes `eks:ListClusters` with assumed credentials

8. Returns results

Beautiful

Except when it doesn’t work. Lets talk about debugging.

The Debugging Journey

When I first deployed this, nothing worked. The bot would try to list resources and just get generic “Unable to locate credentials” errors. Not helpful.

The problem with boto3 and the AWS SDK is that errors are often silent or misleading. A missing environment variable produces the same error as a malformed config file or insufficient IAM permissions.

Failure 1: boto3 Cant Find the Config File

First test: bot tries `aws eks list-clusters --profile oak`.

Error: `Unable to locate credentials. You can configure credentials by running “aws configure”.`

This error is lying. The credentials exist (Lambda execution role). The real problem: boto3 isn’t reading the config file at all.

I added debug logging to print the contents of `/tmp/.aws/config` right after copying it. The file was there. The profiles were correct. boto3 just wasnt looking at it.

The fix: `AWS_SDK_LOAD_CONFIG=1`

Without this environment variable, boto3 ONLY reads `~/.aws/credentials` and completely ignores `~/.aws/config`. Since our profiles live in the config file, boto3 had no idea they existed.

After adding that one line to the env dict, profiles started working.

One down

Failure 2: Subprocess Cant See Lambda Credentials

Second test after adding `AWS_SDK_LOAD_CONFIG=1`: same error.

Wait, what? I just fixed this.

Turns out when you pass a custom `env` dictionary to `StdioServerParameters`, Python doesn’t merge it with the parent process environment. It REPLACES the environment entirely.

The MCP server subprocess was starting with ONLY the variables I explicitly provided:

- HOME

- AWS_REGION

- AWS_CONFIG_FILE

- AWS_SDK_LOAD_CONFIG

- AWS_API_MCP_WORKING_DIR

Missing: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN

Lambda provides these credentials to the parent process, but the subprocess couldnt see them. So when boto3 tried to read `credential_source = Environment`, there were no credentials in the environment.

The fix: explicitly pass through the credentials

env = {
# ... other variables ...
“AWS_ACCESS_KEY_ID”: os.environ.get(”AWS_ACCESS_KEY_ID”, “”),
“AWS_SECRET_ACCESS_KEY”: os.environ.get(”AWS_SECRET_ACCESS_KEY”, “”),
“AWS_SESSION_TOKEN”: os.environ.get(”AWS_SESSION_TOKEN”, “”),
}

This grabs the credentials from the parent Lambda environment and explicitly passes them to the MCP server subprocess.

After this change, role assumption started working.

Two down

Debugging Tips

If youre building something similar, these techniques helped me:

1. Add debug logging everywhere - Print environment variables (redact sensitive values), print file contents, print boto3 session configuration

2. Test incrementally - I tested after each change: config file copied? Environment variable set? Credentials passed through?

3. Read the source code - The AWS CLI MCP server is open source. When documentation was unclear, I read driver.py and parser.py directly

4. Use CloudTrail - Failed AssumeRole attempts show up in CloudTrail in the target account with detailed error messages

The key lesson: boto3 fails silently. You need to instrument your code with logging to understand whats actually happening.

Summary

So thats it. We now have a single AI agent running in Lambda that can query resources across any number of AWS accounts using profile-based role assumption.

The AWS CLI MCP server does support --profile flags, even though its not well documented. You just need to configure everything correctly for boto3 to work.

The full code is in my GitHub/KyMidd/SlackStrandsAgenticBot repo if you want to see the complete implementation.

Now my bot can answer “How many EKS clusters are in the Oak account?” and actually get the answer. Pretty cool.

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

I’ve been building out my Slack Strands Agentic bot, a bot that comes with built-in MCPs that could connect to:

• PagerDuty

• Confluence / Jira

• GitHub

As well as read our Bedrock knowledge base. I want the bot to be able to read the real status of resources, like in AWS or Azure.

When I added the Azure MCP, I worked through the bugs, and when I got it right… it crashed.

ERROR: Tool name ‘search’ already exists

MCP (Model Context Protocol) is how AI agents connect to external tools. It’s basically an API, but instead of you calling endpoints, the agent does. You give it access to a GitHub MCP server, and suddenly it can search code, open issues, list repos. Same with Atlassian - it gets tools for Confluence and Jira. Add PagerDuty and Azure, and you’ve got a pretty capable agent.

Except when two servers both provide a tool called search.

Apparently when you import tools to your bots, if the tool names are identical, it crashes the bot (at least for the Strands library).

Atlassian MCP has search for finding Confluence pages, and Azure MCP has search for Azure resources. Both correct, and both needed, but both have the same name.

It’s apparently not like variable scoping where context matters. This is a flat namespace.

I debugged the tools that are registering, you can see them below.

Both Atlassian and the new Azure MCP try to register a tool called “search”, which isn’t permitted.

When you connect multiple independent MCP servers to the same agent, eventually their tool names will overlap. The more servers you add, the more likely it becomes.

I don’t want to prune back those tools, I want all of them! Which meant I needed to fix the collision without modifying the MCP servers themselves (they’re first-party remote and hosted by the companies) and without breaking how the Strands SDK registers tools.

Cool! Lets do this.

The Solution: Wrap It, Prefix It, Ship It

How most tools handle this is namespaces. If a tool is imported from Azure called “search”, call it “azure_search”. That way collisions should be much more rare. And that’s what I built.

I expected I could just pass an argument to the Strands agent when I give it tools, and it’ll ingest the tool names with a prefix.

I’m working on a PR for Strands to do this.

We need a wrapper! Something that sits between the MCP tool and the Strands registry, changes the name, but otherwise acts exactly like the original tool.

Here’s what I built:

The `PrefixedMCPTool` class does three things:

1. Inherits from `AgentTool`: This is critical. The Strands SDK registry checks `isinstance(tool, AgentTool)`. If your wrapper doesn’t inherit from `AgentTool`, the registry rejects it. No exceptions.

2. Modifies the tool name: Takes the original tool’s spec, copies it, and changes just the name. Everything else stays the same - description, parameters, all of it. Just the name gets the prefix.

3. Delegates everything else: All the actual work gets passed to the original tool. The wrapper is just changing the name on the way in.

The key part is the `stream()` method:

def stream(self, tool_use, *args, **kwargs):
  return self._original_tool.stream(tool_use, *args, **kwargs)

That `*args, **kwargs` matters. The Strands framework passes extra arguments to `stream()` - stuff like `invocation_state` that the tool needs. If you don’t forward those arguments, your tools break at runtime (I learned this the hard way).

So now instead of registering a tool called `search`, you register a `PrefixedMCPTool` that wraps the original `search` tool but tells the registry its name is `atlassian_search`. The registry sees a unique name. The tool still works exactly the same way.

This ended up working SUPER WELL and I’m going to do this in ever bot I build going forward (unless the MCP itself already prefixes the platform ahead of the tool name, which they obviously should, but most don’t today).

## Making It Reusable

I didn’t want to wrap every single tool manually, or define this same function in every MCP ingestion script (which are separate files for each MCP due to filtering, authentication, and utilization differences). I needed a single helper function that could take all the tools from an MCP server and prefix them in one shot.

Here’s what I built:

The `add_prefix_to_mcp_tools()` function takes two arguments:

- `tools`: The list of tools from your MCP server

- `prefix`: The prefix you want to add (like “github” or “azure”)

It automatically adds the underscore, so you just pass `”github”` and it creates `github_search`, `github_get_issue`, etc.

Do this for each MCP server with a different prefix:

atlassian_tools = add_prefix_to_mcp_tools(atlassian_tools, “atlassian”)
github_tools = add_prefix_to_mcp_tools(azure_tools, “github”)
pagerduty_tools = add_prefix_to_mcp_tools(pagerduty_tools, “pagerduty”)

Before prefixing, I had:

- Atlassian: `search`, `fetch`

- Azure: `search`

After prefixing:

- Atlassian: `atlassian_search`, `atlassian_fetch`

- Azure: `azure_search`

- No collision, bot works

The agent now has access to 106 tools total (25 from GitHub, 19 from Atlassian, 15 from PagerDuty, 47 from Azure), and every single one has a unique name, and all are accessible at runtime.

With my luck, AWS (my next target) will also have a “search” tool, but now I won’t really care, we won’t see the collision even if it does.

Summary

If you’re using multiple MCP servers with your AI agents, you’ll hit tool name collisions eventually. It’d be amazing if some registry came out that helped folks establish a standard naming scheme, validated partners and sources, much like the GitHub Marketplace, or Azure Marketplace, Hashi Marketplace etc., but it doesn’t exist yet.

I’m excited to no longer care what tools we register - now I can just add MCPs and their tools, and go back to worrying about AIs deleting my environment, or taking over the world.

Thanks ya’ll. Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all! Today we’re tackling a problem that’s been driving me crazy since we started scaling our AWS Bedrock bot deployments across the enterprise. You’d think that when you have multiple AI bots running in production, AWS would give you some reasonable way to separate and monitor their activities. You’d be wrong.

Here’s the issue: every single AWS Bedrock model invocation in a region gets dumped into one massive CloudWatch log group. Doesn’t matter if you have five different bots serving different teams, or if Bot A is handling customer support while Bot B is doing internal documentation queries. Everything goes into `/aws/bedrock/modelinvocations` and good luck figuring out which log entry came from which bot.

This creates a monitoring nightmare. When someone asks “how much is our Slack bot costing us?” or “why did the support bot give a weird response yesterday?” you’re stuck grep-ing through thousands of log entries that all look identical. The raw Bedrock logs contain the model inputs and outputs, but zero context about which application made the call or who the actual user was.

After dealing with this mess for way too long, I finally built a solution that transforms these useless raw logs into something actually meaningful for enterprise monitoring. We’re talking about a Kinesis Firehose pipeline with a custom Lambda processor that parses the Bedrock JSON and adds the context that should have been there in the first place.

The enhancement adds four key fields that make all the difference: `botName` (extracted from the IAM role), `botOutput` (what the bot actually said to the user), `user` (who asked the question), and `query` (what they asked). Suddenly, instead of cryptic JSON blobs, you have logs that tell a story about real conversations happening in your enterprise.

This approach works for any AWS Bedrock deployment, but it’s especially valuable if you’re running multiple bots or need to track usage by team, project, or individual users. The processed logs flow into whatever monitoring system you’re already using - in our case, Splunk - where they become the foundation for dashboards, alerting, and cost attribution.

Let’s walk through how this whole system works and why it’s become essential for managing AI deployments at enterprise scale.

If you don’t care about the write-up and just want the code, you can find the modified cloudwatch to splunk lambda that adds bot information here:

gist.github.com/KyMidd/f2d216682b0c40ca06d74742f1cc56f7

The Problem: AWS Bedrock’s Logging Black Hole

To understand why this solution matters, you need to see just how broken the default AWS Bedrock logging situation really is. When you make a model invocation through Bedrock, AWS dutifully logs the entire interaction to CloudWatch - which sounds great until you realize where those logs actually end up.

Every single model invocation in a region gets dumped into `/aws/bedrock/modelinvocations`. Your customer support bot? Same log group. Your internal documentation bot? Same log group. That experimental bot someone in marketing is testing? Yep, same log group. It’s like throwing all your mail into one giant pile and then wondering why you can’t find anything.

AWS does include some metadata with each log entry - most importantly, the IAM role ARN that made the request. So technically, you can tell which bot generated which log entry. But here’s the catch: you have to parse through thousands of nearly identical JSON blobs to find the entries you care about, and even then, the raw logs don’t tell you anything useful about the actual conversation.

Here’s what a typical raw Bedrock log entry looks like - it’s hundreds of lines of nested JSON containing the full model request and response, but buried so deep in structure that extracting meaningful information requires serious JSON archaeology. You can see the complete input and output, but good luck figuring out which user asked what question or what the bot actually said back to them.

The enterprise implications are brutal. When your CFO asks “how much are we spending on the Slack bot versus the customer support bot?” you can’t answer without building custom tooling to parse through the log mess. When a user reports “the bot gave me a weird answer yesterday,” you’re looking at potentially thousands of log entries with no easy way to find their specific interaction.

Even basic monitoring becomes a nightmare. Want to set up alerts for when a specific bot starts giving errors? You’ll need to craft complex CloudWatch filters that parse the IAM role ARN and hope you can distinguish between different types of failures. Want to track usage patterns or popular queries? Hope you enjoy writing custom log analysis scripts.

The root issue is that AWS designed Bedrock logging for debugging individual model calls, not for managing multiple production AI applications. They give you all the technical details about the model invocation but none of the business context about who asked what and why. For a single-bot deployment, this might be manageable. For enterprise deployments with multiple bots serving different teams and use cases, it’s completely unworkable.

The Solution: Enhanced Log Processing Pipeline

The answer to this mess is surprisingly straightforward: transform them on their way to your monitoring system.

Here’s the architecture: CloudWatch subscription filter → Kinesis Firehose → Lambda processor → Splunk. The raw Bedrock logs stay exactly as AWS created them in CloudWatch. But when they flow to Splunk through our pipeline, they get enhanced with the context that actually matters.

The Lambda processor is where the magic happens. It takes those massive, cryptic JSON blobs and transforms them into something humans can actually understand. Instead of hundreds of lines of nested structures arriving in Splunk, you get clean log entries with the four fields that matter: `botName` (which bot generated this), `botOutput` (what it actually said), `user` (who asked the question), and `query` (what they asked).

The processor extracts the bot name from the IAM role ARN, navigates through the complex nested JSON to find the actual response text, and parses conversation formats to identify users and their questions. What used to require custom scripts and manual investigation now happens automatically for every single log entry flowing into your monitoring system.

The beauty of this approach is that it’s completely non-intrusive. Your CloudWatch logs remain untouched for AWS debugging purposes, but your enterprise monitoring gets the enriched context it needs. The whole pipeline uses standard AWS services, scales automatically, and integrates seamlessly with whatever log aggregation platform you’re already using.

Log Enhancement

The real work happens inside the Lambda processor, where we parse the Bedrock JSON and extract the four fields that actually matter for enterprise monitoring.

Bot Name Extraction

This one’s straightforward. Every Bedrock log includes the IAM role ARN that made the request. We just grab the last segment after the final slash - so `arn:aws:iam::123456789012:role/VeraSlackBot-Worker` becomes `VeraSlackBot-Worker`. Simple, but effective for distinguishing between different bots.

Bot Output Parsing

This is where things get messy. The actual response text is buried deep in the JSON structure at `output.outputBodyJson.output.message.content[0].text`. That’s five levels of nesting just to find out what the bot said to the user.

User and Query Extraction

The most complex part involves parsing the conversation format from the input messages. Our bots format user messages as “Kyler (she/her) says: Now a dog.” - so we need to split on “ says: “ and extract the username before any parentheses. The query is everything after the colon.

The code handles all the edge cases gracefully. If any field can’t be extracted, we just skip it rather than failing the entire log processing. This keeps the pipeline resilient when AWS changes log formats or when we encounter unexpected message structures.

What emerges from this parsing is a clean, searchable log entry that tells the complete story: which bot responded to which user’s query with what output. Instead of archaeological JSON excavation, you get logs that humans can actually read and monitoring systems can actually use.

Terraform Infrastructure: Making It Work

Setting up this pipeline requires several Terraform resources working together. The core piece is the Kinesis Firehose delivery stream that connects everything.

CloudWatch Subscription Filter

First, you need a subscription filter on the Bedrock log group that forwards all log events to your Firehose stream. This is the entry point that captures every Bedrock model invocation as it gets logged.

IAM Roles and Permissions

The pipeline requires two main IAM roles. One for CloudWatch to write to Firehose, and another for Firehose to invoke your Lambda processor and write to S3 for failed events. Standard AWS service-to-service permissions, but they need to be configured correctly or nothing works.

Lambda Function

Your log processing Lambda function needs the right runtime (Python 3.13), appropriate timeout settings, and environment variables for any custom configuration like Splunk source types.

Kinesis Firehose Configuration

The Firehose configuration is where everything comes together. You specify your Splunk endpoint, configure the Lambda processor, and set up S3 backup for failed events. The processing configuration is key - that’s what tells Firehose to run your custom Lambda function on every log batch before sending to Splunk.

Error Handling

Don’t forget S3 backup for failed events. When something goes wrong with log processing or Splunk delivery, you want those logs going somewhere you can investigate later rather than disappearing into the void.

Real-World Impact and Benefits

The difference this makes in day-to-day operations is dramatic.

Individual Bot Monitoring

Suddenly, you can answer basic questions about your AI deployments. “How much is the customer support bot costing us?” becomes a simple Splunk query filtering by `botName`. “Which bot is getting the most usage?” is just a matter of counting log entries by bot. No more spreadsheet gymnastics or manual log parsing.

Troubleshooting User Interactions

When someone reports “the bot gave me a weird answer,” you can actually find their interaction. Search for their username and the approximate time, and there’s the complete conversation - what they asked, what the bot responded with, and all the underlying model invocation details. This turns impossible debugging sessions into five-minute investigations.

Cost Attribution and Chargeback

Enterprise teams love being able to track costs by department or project. With botName extraction, you can build dashboards that show exactly how much each team is spending on AI. Finance gets happy, teams get visibility into their usage, and everyone makes better decisions about resource allocation.

Performance Insights

The enriched logs reveal usage patterns you never knew existed. Which types of queries take the longest to process? What time of day sees the most bot activity? Are certain users asking the same questions repeatedly (hint: maybe your documentation needs work)?

Integration with Existing Monitoring

Because the enhanced logs flow into whatever monitoring system you’re already using, all your existing dashboards, alerts, and processes just work. Set up alerts when a specific bot starts throwing errors. Create dashboards showing conversation trends across different teams. Build reports on AI adoption across the organization.

Conversation Analytics

Having the actual user queries and bot responses in searchable logs opens up possibilities for conversation analytics. You can identify common question patterns, track how bot responses change over time, and even spot opportunities for better training data or system improvements.

The end result is that AWS Bedrock goes from being a black box that costs money to being a transparent, manageable enterprise service that you can actually operate with confidence.

Summary

AWS Bedrock dumps all model invocation logs into a single regional CloudWatch log group, making enterprise monitoring nearly impossible. This creates chaos when trying to track costs, troubleshoot issues, or monitor multiple bots.

The solution: a Kinesis Firehose pipeline with a custom Lambda processor that intercepts logs flowing to Splunk and adds essential context. The processor extracts botName (from IAM role), botOutput (actual response), user (who asked), and query (what they asked) from the raw JSON.

This transforms cryptic Bedrock logs into actionable monitoring data, enabling cost attribution, user interaction troubleshooting, and conversation analytics for enterprise AI deployments.

Thanks all!
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

Today we’re going to explore something pretty exciting in the world of enterprise AI applications. I recently got my hands on an evolution of the Vera genAI solution that I’ve covered extensively before - and this new version represents a significant architectural leap forward.

If you’ve been following my previous posts about Vera, you’ll remember it was built with Python directly without many external libraries, focusing on raw AWS Bedrock integration. This new implementation takes that foundation and transforms it into a proper agentic AI system using the Strands framework. Instead of just being a conversational AI, it can now take actions and use tools across your entire enterprise toolchain.

What makes this implementation particularly interesting is how it solves those everyday enterprise pain points we all know too well. You know that juggling act between Slack, PagerDuty, GitHub, and Jira just to get context on a single issue? This bot brings all those systems together in one conversational interface, powered by AWS Bedrock with Claude Sonnet 4 as the backend.

Here’s where the agentic capabilities really shine. The bot uses Model Context Protocol (MCP) to actually connect to your existing platforms. When someone asks “what incidents are assigned to me?” it goes and queries PagerDuty directly through AWS Bedrock’s processing. When they want to know about pull request status, it hits the GitHub API. The AI becomes a conversational interface to your entire toolchain, all orchestrated through AWS Bedrock’s model capabilities.

The system includes AWS Bedrock Guardrails for content filtering and even supports retrieval-augmented generation through Bedrock Knowledge Bases. It’s engineered for enterprise use with proper secret management, brand voice guidelines, and all the infrastructure automation you’d expect in a production environment.

In this article, we’ll walk through the complete architecture, examine how the Strands agentic framework transforms the original Vera approach, and look at how AWS Bedrock powers the multi-service integrations. Whether you’re thinking about building your own AI assistant or just curious about modern agentic architectures, this implementation has some great lessons to offer.

Let’s jump in and see how this all fits together!

If you want to skip the write-up and just read the code, you can find that here:

github.com/KyMidd/SlackStrandsAgenticBot

Containerizing the Worker Lambda: Why Docker Over Layers

When I started building this enhanced version of Vera, I quickly ran into a problem that anyone working with modern Python AI frameworks knows all too well: dependency hell.

The original Vera was relatively lightweight - just AWS SDK calls and some basic Slack integration. But this agentic version? We’re talking about the Strands framework, multiple MCP clients for different services, and a whole ecosystem of dependencies that each service brings along. The requirements.txt file exploded from a handful of packages to something that looked more like a small novel.

Lambda layers seemed like the obvious choice at first. After all, that’s what they’re designed for - sharing common dependencies across functions. But here’s where things got messy: we’re running on ARM64 for cost savings, and compiling some of these Python packages for ARM64 is a nightmare. Half the dependencies don’t have ARM64 wheels available, which means building from source. And building from source on ARM64? Good luck with that.

I spent way too much time trying to get a clean layer build working before I admitted defeat and switched to containers. Suddenly, all those dependency headaches disappeared. The AWS Lambda Python runtime base image handles the ARM64 compilation complexity, and I can use modern Python package managers like `uv` to handle the dependency resolution efficiently.

The container approach gives us a few key advantages. First, we can pre-install and configure the MCP servers during the build process. Notice how we’re copying the PagerDuty MCP server into `/opt/` and running `uv sync` to install its dependencies. This happens at build time, not runtime.

Second, we get much more predictable dependency resolution. The `uv` package manager handles all the complex dependency trees and gives us reproducible builds. No more “it works on my machine” issues when deploying to Lambda.

Third, the container image becomes our deployment artifact. Terraform can build, tag, and push the image to ECR, then update the Lambda function to use the new image - all in one deployment cycle. No more managing separate layer deployments or worrying about version mismatches.

The performance impact has been minimal. Cold starts are slightly longer due to the larger image size, but the warm execution performance is actually better because all dependencies are pre-resolved and ready to go. For a Slack bot that processes messages intermittently, this trade-off makes perfect sense.

Code Organization: Breaking Up the Monolith

One thing I learned from the original Vera implementation was that shoving everything into a single Python file becomes a maintenance nightmare pretty quickly. The first version worked fine when it was just basic AWS Bedrock calls and Slack responses, but as soon as I started adding MCP integrations and agentic capabilities, that single file ballooned to over 800 lines.

This time around, I split things up from the start. The worker Lambda is now organized into focused modules, each handling a specific aspect of the bot’s functionality:

• worker.py - Main Lambda handler and Slack app registration

• worker_inputs.py - Configuration, constants, and system prompts

• worker_agent.py - Strands agent setup and execution

• worker_aws.py - AWS service integrations (Bedrock, Secrets Manager)

• worker_slack.py - Slack-specific utilities and response formatting

• worker_conversation.py - Message processing and conversation flow

• worker_lambda.py - Lambda-specific utilities and event handling

• worker_mcp_*.py - Individual MCP client implementations

The import structure shows how clean this separation becomes. Each module has a single responsibility, which makes testing and debugging so much easier. When something goes wrong with PagerDuty integration, I know exactly where to look. When I need to adjust the system prompt or brand voice, it’s all in `worker_inputs.py`.

What’s particularly nice about this approach is how it handles the MCP integrations. Each service gets its own module (`worker_mcp_pagerduty.py`, `worker_mcp_github.py`, etc.) that encapsulates all the client setup, authentication, and tool registration logic. The main agent module just imports what it needs and doesn’t have to worry about the implementation details.

This modular structure also makes the container build more efficient. Docker can cache layers better when dependencies are clearly separated, and the overall codebase becomes much more approachable for anyone who needs to maintain or extend it later.

MCP Protocol Integrations: Connecting External Tools

Here’s where things get really interesting. The Model Context Protocol is what transforms this from just another chatbot into an actual agentic system that can take actions across your toolchain. Instead of the AI just talking about your GitHub issues or PagerDuty incidents, it can actually query them directly and give you real-time information.

The beauty of MCP is that it provides a standardized way for AI models to interact with external systems. Each service exposes its capabilities as “tools” that the AI can discover and use dynamically. When someone asks “what incidents are assigned to me?” the bot doesn’t just guess - it calls the PagerDuty API through the MCP interface and gets the actual data.

I filter all tools in their own file, so I can only return tools to the agent that can do READ behavior (non-modify). AIs are way too scary to give them write tools, IMO. You can say read-only = False if you want all the Write tools also.

PagerDuty Integration

The PagerDuty integration handles incident management and on-call scheduling. The MCP server runs as a pre-installed component in the container, which means zero cold-start overhead when the bot needs to query PagerDuty.

The setup is straightforward - we pass the API credentials through environment variables and let the MCP client discover what tools are available. The PagerDuty MCP server typically exposes tools for listing incidents, getting on-call schedules, and querying service status.

GitHub Integration

GitHub integration is particularly useful for development teams. The bot can pull pull request status, check repository information, and even search across codebases. I’ve implemented it with read-only access by default because, let’s be honest, you probably don’t want your Slack bot accidentally merging pull requests.

The filtering logic here is important. By restricting to read-only operations, we get all the query capabilities without the risk of destructive actions. The bot can tell you about issues, pull requests, and repository status, but it can’t modify anything.

Atlassian Integration

The Atlassian integration covers both Jira and Confluence, which makes it incredibly powerful for teams that live in the Atlassian ecosystem. The OAuth setup is a bit more complex than the other integrations, but once it’s configured, the bot becomes a conversational interface to your entire knowledge base.

What’s particularly clever about the Atlassian integration is how it handles both Jira tickets and Confluence documentation in a unified way. When someone asks about a project, the bot can pull information from both systems and present a complete picture.

The key insight with all these integrations is that they’re designed to fail gracefully. If a service is unavailable or credentials are missing, the bot continues to work - it just operates without those specific tools. This makes the system much more resilient in production environments.

Terraform Container Deployment: Building and Deploying Docker Images

One of the things I really appreciate about this architecture is how Terraform handles the entire container lifecycle. Unlike traditional Lambda deployments where you’re uploading zip files and hoping for the best, this approach treats the container image as the deployment artifact and manages everything from build to deployment in a single workflow.

The magic happens through a combination of ECR repository management and some clever null_resource provisioners that handle the Docker build and push operations. When you run `terraform apply`, it doesn’t just update your Lambda configuration - it actually builds a fresh container image, pushes it to ECR, and then updates the Lambda function to use that new image.

The ECR repository includes lifecycle policies that automatically clean up old images, which is important because container images can get pretty large. We keep the latest 10 tagged images and automatically delete untagged images after a day. This prevents your ECR costs from spiraling out of control as you iterate on the deployment.

The Docker build process is particularly clever here. The null_resource uses a trigger based on the source file hash, so it only rebuilds the image when your code actually changes. This saves a ton of time during development when you’re just tweaking Terraform configurations without touching the application code.

What’s really nice about this approach is how it handles the ARM64 build process. The `--platform linux/arm64` flag ensures we’re building for the correct architecture, and since we’re building on the same platform where Terraform runs, we avoid all those cross-compilation headaches that made Lambda layers so problematic.

The Lambda function configuration shows how seamlessly containers integrate with the rest of AWS Lambda. You still get all the same configuration options - memory size, timeout, environment variables - but now your entire dependency tree is packaged as a single, immutable artifact. No more worrying about layer compatibility or missing dependencies.

Environment variables are passed through cleanly, including the MCP feature flags that control which integrations are enabled. This means you can deploy the same container image to different environments with different capabilities just by changing the Terraform variables.

Lessons Learned and Next Steps

Building this enhanced version of Vera taught me quite a few things about modern agentic AI architectures, and honestly, some hard lessons about when to stick with what works versus when to embrace new approaches.

Container vs Layers: The Right Tool for the Job

The biggest lesson here was accepting that Lambda layers, while great for simple shared dependencies, just weren’t going to cut it for this complexity level. The original Vera worked fine with layers because it had a handful of lightweight dependencies. But once you’re dealing with the Strands framework, multiple MCP clients, and all their transitive dependencies, containers become the only sane choice.

The decision framework I’d use now: if your requirements.txt has more than 20 packages or includes any packages that need native compilation, just go straight to containers. The development experience is so much better, and the deployment reliability gains are worth the slightly larger cold start times.

Performance Considerations

Speaking of cold starts, they’re definitely longer with the container approach - usually 2-3 seconds instead of sub-second for the original Vera. But here’s the thing: for a Slack bot, this rarely matters. Most conversations happen in bursts, so the function stays warm. And when it does cold start, users are already expecting some delay for “AI thinking time.”

What surprised me was how much faster the warm execution became. All those dependencies are pre-resolved and ready to go, which makes the actual AI processing noticeably quicker. The MCP clients can establish connections faster, and there’s less Python import overhead.

Scaling Challenges and Solutions

The current architecture handles our team’s usage just fine, but I can see some potential bottlenecks as usage grows. Each Lambda invocation opens its own MCP client connections, which could become problematic if we’re hitting rate limits on external APIs.

The next evolution would probably involve connection pooling or even moving some of the MCP operations to separate services that can maintain persistent connections. But honestly, that’s a nice problem to have - it means people are actually using the bot.

Modular Architecture Payoffs

The decision to break everything into focused modules has paid dividends already. When GitHub changed their API authentication requirements, I only had to touch `worker_mcp_github.py`. When we wanted to adjust the system prompt for better Slack formatting, it was all in `worker_inputs.py`.

This modularity also makes testing much more manageable. You can unit test individual MCP integrations without spinning up the entire Lambda environment, which speeds up the development cycle considerably.

Future Enhancement Opportunities

There are several directions this could evolve. Adding more MCP integrations is the obvious one - imagine connecting to ServiceNow, Datadog, or your internal APIs. The framework makes this relatively straightforward now.

I’m also interested in exploring memory and context persistence. Right now, each conversation is stateless, but there’s real value in the bot remembering previous interactions or maintaining context across related conversations.

The Strands framework also supports more advanced agentic behaviors like multi-step planning and tool chaining. We’re barely scratching the surface of what’s possible when you give an AI system the ability to orchestrate multiple API calls to solve complex problems.

Summary

It’s been super fun to build out this new platform, and I’ve been able to re-use many of the pieces of the original genAI Vera, particularly the pieces that integrate with Slack.

The bot is quite a bit slower than genAI Vera, but can answer questions like this:

• “Show me critical incidents for my team” - pulls live PagerDuty data

• “Status of the auth PR?” - queries GitHub directly

• “Find our API rate limiting docs” - searches Confluence with links

• “What’s blocked this sprint?” - filtered Jira queries

And because of using MCP tools, the answers are less often hallucinated, and more often backed by information the tool fetches.

Instead of bouncing between Slack, PagerDuty, GitHub, and Jira for incident context, teams get comprehensive answers from one question. On-call engineers can query previous incidents, service status, and recent code changes without leaving their response channel.

Hopefully it can help you too. Good luck out there!
kyle

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

I’ve been playing around with Strands a great deal, and having a ton of fun. Strands Agents is a python library that permits writing agentic workloads with relatively little code, and supports all sorts of AI backends to provide the AI services, including Amazon Bedrock, OpenAI, Anthropic, and even local models on your computer.

Here’s one that fetches vectors from an AWS Bedrock Knowledge Base, feeds those vectors into an Agent (by default Claude 4), and streams an answer to the provided question - all in 4 lines! That’s absolutely absurd, it’s so cool.

from strands import Agent
from strands_tools import retrieve
agent = Agent(tools=[retrieve])
agent("Tell me about the cloud platform team")

One of the powers of Strands is it easily integrates with MCPs - those tool integrations that permit your agent to interact with the platforms your business uses to do stuff. Strands can use MCPs to look at:

• Your tickets in Jira? Yep!

• Your outage information from PagerDuty? Yep!

• Your code on GitHub? Well, sorta

Because “Resources” aren’t built out yet, Strands agents can’t yet read the contents of files from GitHub.

Lets talk about what’s missing and how to fix it!

Wait, What are we talking about?

There’s a lot going on here, so it’s important to understand the players.

First, there are AI clients - these provide the AI services from a platform, for example AWS Bedrock or OpenAI. You generally get a token from these platforms, export the token into your terminal, and you can then use tools that trigger jobs on these platforms.

If you want to be fancy, you can say “requests for inference” instead of “jobs”. They have the same meaning.

Then there are MCP clients. These are the software components that can talk to MCP servers, potentially authenticate, and download the tools lists from the MCP server. Often this is your IDE like VSCode or Claude Code. Strands has an MCP client built in.

Then there are MCP servers. These can exist anywhere, but we’re seeing more and more of these as remote, first party MCP servers. That means the company who hosts the MCP server (remote to your client, like over the internet) is the place you’d use those tools. Think accessing a GitHub MCP from a GitHub URL in order to download tools to interact with GitHub’s codebase.

The platforms that store enterprise data have a great motivator to support MCP - all the enterprises that are charging into AI adoption can more easily consume those platforms if they can access an MCP for the platform. Early stage companies that are much more AI focused could easily use MCP availability as a primary differentiator when choosing which platforms to use.

The problem

So there’s a GitHub MCP! And it works great. When I first started working with it, I assumed it wasn’t completely implemented - it just entered General Availability (GA) Sept 4, 2025, and I was playing with it in Strands a few weeks before that.

It had this odd ability to read ALL the data from GitHub, but it couldn’t download file contents. Weird, but whatever.

However, when I tested the same GitHub MCP with VS Code IDE, it worked great.

So I tested it with Claude Code, worked great.

It turns out ONLY Strands has a problem downloading file content from GitHub when using MCP.

I dug into the problem with some logging and found this log:

2025-08-22 07:50:55,725 | INFO | Thread-1 (_background_task) | httpx | HTTP Request: POST https://api.githubcopilot.com/mcp/ "HTTP/1.1 200 OK"
# ...
2025-08-22 07:50:55,865 | DEBUG | MainThread | strands.tools.mcp.mcp_client | [Thread: MainThread, Session: 838664df-dafb-4a06-865e-91d618eb77f6] unhandled content type: EmbeddedResource - dropping content
2025-08-22 07:50:55,865 | DEBUG | MainThread | strands.tools.mcp.mcp_client | [Thread: MainThread, Session: 838664df-dafb-4a06-865e-91d618eb77f6] tool execution completed with status: success

There’s two interesting things going on here. When the Strands agent attempts to fetch file contents, it uses a tool called get_file_contents. This tool is just an MCP instruction to use “Get Repository Contents” API to fetch a file contents.

It reads the header to identify the type of file. If it starts with `application`, it decides that file is text/ascii-based. For example, a .tfvars file (terraform variables) is text based, and the header value gets:

< content-type: application/json; charset=utf-8

After deciding the content is text, the GitHub MCP (really the Get Repository Contents REST API call), returns a payload like this:

{
  "type": "resource",
  "resource": {
    "uri": "repo://orgName/repoName/contents/path/to/file.tfvars",
    "mimeType": "application/json; charset=utf-8",
    "text": "... (plain-text contents of the file) ..."
  }
}

The Strands library expects to see direct TextResourceContents (on .text property path) or BlobResourceContents (on .blob property path) like this:

{
  "text": "... (plain-text contents of the file) ..."
}

Since the GitHub MCP is adding extra information, the Strands library totally fails to interpret it, despite the path being super duper close, and the information being nearly the same.

How To Fix It

You can’t. Well, at least not easily. The problem is in how the Strands library is written - it doesn’t yet have support for EmbeddedResources, which is the whole “.resource” property path and syntax.

I’ve created a fixed version on this PR: https://github.com/strands-agents/sdk-python/pull/726

This fixes the behavior for many types of files that are text encoded from github, but doesn’t yet work for binary/blob type files. It does however work for these types of files:

"application/json",
"application/xml",
"application/javascript",
"application/x-yaml",
"application/yaml",
"application/xhtml+xml",

And you’d be surprised how many file types are covered with these!

Try it out: Copy my version (from the PR) of this file: src/strands/tools/mcp/mcp_client.py over your version, and watch how your Strands agent can now fetch file contents:

What’s Next?

The Strands maintainers are hoping to implement full support for Resources before implementing partial support, meaning that this functionality isn’t coming immediately. Implementing the full lifecycle support for Resources will take a bit longer.

I’ll continue asking to merge this in - I think being able to fetch file contents from GitHub is a huge functionality gap for Strands, and will be relatively blocking for us.

Potentially we’ll need to implement some static tools for our bots to use to fetch file contents from GitHub using REST. I

And I’ll be sure to write about it.

Thanks all! Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

Hey all!

In previous articles we built generative AI bots that integrate with a RAG knowledge base using the AWS Bedrock backend.

• AWS GenAI Bedrock Series

• Azure GenAI Bedrock Series

They’re awesome, but they’re not agentic.

"Agentic" describes the ability to act autonomously, make decisions, and achieve goals without constant human intervention, a concept central to agentic AI systems that can plan, execute tasks, and adapt to new information

Basically, agentic bots do 1 step, and then return that information. That generally works really well for chatbots - read this information, return a summary. One step is all you need!

However, if you want a bot to do a multi-step process, that’s when you’re looking at an “agentic workflow”. Something like “Read our PagerDuty, isolate any resources with issues, then go look at AWS and see if the resources have an error state you can recommend fixes for” is multiple step - talk to platformA, analyze the data, then use that data to talk to platformB and analyze the data from there.

Agentic agents can do it!

Strands has emerged as a language that permits building effective and comprehensive bots with little python. It comes out of AWS, but the team there has made sure to open-source it (Apache 2.0 license) and make sure it’s compatible with lots of back-ends like OpenAI/ChatGPT, Anthropic, OLlama, and others.

It support agentic workflows, MCP (tool use to talk to platforms), A2A protocol (to talk to other bots and coordinate work), and will likely be kept up to date with AWS services in AgentCore like Memory and other cool stuff shaking out of that project.

It feels like a great place to start.

Strands, find all the ec2 instances in us-east-1, and give me their IPs and AZs in a table.

I’m going to be building some enterprise Strands agentic assistants over the next few months, but for now I’m just proving out really simple workflows, and I wanted to share them. They’re all here:

github.com/KyMidd/StrandsAgenticExamples

We’ll walk through one by one for how they work.

Simple Calculator

AI Models are terrible at math. They just are. If you ask a model a math problem that it’s seen before in its data set, it’ll give you the answer it saw. Is it the right answer? It doesn’t know.

I’m also bad at math, so I use a calculator. Best idea to make an AI good at math? Give it a calculator! And that’s one of the built-in tools available to strands.

On line 1, we import the Agent library from strands, and on line 2 we import from the strands_tools library, specifically the built-in tool called “calculator”.

Then on line 3 we instantiate an agent and provide it the tool.

Then on line 4 we ask the agent to calculate the square root of 125. The agent sees a problem that it can resolve with math, so it uses the calculator.

Here’s the output:

Simple Remote MCP - AWS Docs

Not all MCP servers require authentication - some provide tools you can use to do interesting stuff, like this remote MCP endpoint from AWS that lets you ask AWS itself about their documentation, and provide your bot the ability to read AWS documentation in its operation.

On lines 2-3, we import the MCPClient from the strands tools package, and on line 3 we import the standard i/o client and parameters.

On line 5, we define the aws_docs_client as a python lambda (small function), that’s calling the uvx tool to connect to the awslambda aws-documentation-mcp-server MCP. Notably, this doesn’t “run” the MCPClient yet, this just statically defines what will happen when it does.

On line 9, we define a “with”. This step actually runs the AWS Client config.

Then on line 10, we start the agent with the tools available from that MCP - note how we’re reading the tools list available from the remote MCP with list_tools_sync().

Then on line 11, we’re asking the agent a question.

Here’s how it looks, note how it reads documentation, then decides to read further documentation. This can build a comprehensive understanding of AWS technologies, it’s pretty cool.

Simple GitHub Loop

While still simple, I added a few features to the single-MCP scripts. First, we use functions to define the MCPs.

You can see on line 6 that we define a function to build the streamable remote MCP client, just like the AWS documentation MCP.

A bit lower down, we define the github MCP client, line 2.

And then on line 5, we connect to the MCP client on line 5 with the “with”.

On line 8, we establish the bedrock model, which customizes how the model works (number of tokens permitted, etc.), and build a list of tools we can pass to the agent, line 11-12.

Then on line 15 we build the agent and pass it the tools list.

Next, we start a “while” loop where we ask the agent our question that we defined above, to find all the Repos we own. The agent works agentically to get an answer, and then we hold on line 12, where the tool asks us if we want to continue.

We can type “exit” to break the loop and exit the script, or we can ask another question. Since the Agent is still instantiated, it remembers the previous question and information fetched from MCP, and can continue the conversation.

Here’s how it looks. I think it’s super cool that it can figure out who I am, and then read stuff.

Simple PagerDuty - Local MCP

I’m a big fan of PagerDuty, and I see that they don’t yet have a remote hosted MCP for us to consume. However, they have published a local MCP that we can run ourselves. I staged that locally (and got uvx/uv installed), and then wrote this Strands script to use it.

On line 2 we establish our starter question, to find all active issues over the last 4 issues, and filter them.

Then on line 5 we define a function to utilize the local MCP server. Note that we’re not streaming anything from a remote destination, we’re literally providing the command flags that would start the MCP server if we ran them.

This script, when run, will basically call the MCP server as a child when we get to the “with” part of the function.

Note the “env” variables we inject into it on line 19-21. These are required to read PagerDuty.

see line 17, where we can enable writing to pagerduty, and with it commented out, the MCP will only execute “read” operations, which I feel is much safer with the state of AI today.

The “with” and other script info is exactly the same as the above, so skipping it, you can read the simple_pagerduty.py script if you want to see what we’re doing there.

Here’s how it looks:

Simple AWS

There is a local MCP in preview that permits the agent to run AWS commands, but it’s in beta, and not very easy to run. However, Strands has a built-in tool that permits it to do things in AWS, able to do whatever the boto3 library is capable of doing.

It’s called “use_aws”. We import it on line 3, along with a few other built-in tools.

You can see on line 8 we pass that same list of built-in tools as a list to our agent on line 11.

Here’s how it looks:

Simple AWS Bedrock Knowledge Base

I built a pretty extensive Knowledge Base on Bedrock for Vera - a GenAI bot that doesn’t operate agentically. So when I read that Strands permits it to talk to Bedrock I expected I’d have to build a ton of code, and re-ranking, and such.

However, it’s incredibly easy. I had to export the ID of the knowledge base:

export KNOWLEDGE_BASE_ID=xxxxxx

And then I had to import the “retrieve” tool, which retrieves information from Bedrock knowledge bases. I pass it like any other tool to the agent, and we’re in business.

Notably, I already have AWS creds exported into this terminal, to utilize bedrock AI services, so I’m already authenticated. If you’re using a different back-end for AI services, make sure to also export AWS creds that can read the Bedrock KB.

Here’s how it looks:

Combined MCP

The real power of MCP comes when you combine multiple MCP tools across platforms, because the model can correlate, say, outages (Pagerduty) to specific code changes (GitHub).

This is pretty intuitive once you’ve seen it. I’ve skipped some code that we covered above where we built the MCP function definitions.

On line 2-3, we call those MCPs, and on line 6 we do a “with” to open each of them.

Then on line 12-13, we sync the tools available for both, and on line 16, we build our “tool_belt” (haha), to combine the list of all the tools together.

And then on line 19 we instantiate the agent and pass it all the tools.

Here’s how it looks:

Summary

In this article we’ve covered how to build Strands agentic Agents with python. We’ve talked about how different types of MCPs work, and built them, and finished up with a multi-platform bot that’s able to:

• Talk to PagerDuty to read your incidents

• Talk to GitHub to read your code changes

• Talk to AWS to read your real infrastructure

• Talk to your AWS Bedrock Knowledge Bases to read your corporate data

And also, to correlate changes among those platforms - that’s going to be a really powerful functionality as we move forward in maturing this technology.

I’m going to keep building super bots, and I’m very interested in AgentCore to establish memory and runtime observability for these bots. I’ll be digging into that next, keep an eye out!

Good luck out there.
kyler

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

These articles are supported by readers, please consider subscribing to support me writing more of these articles <3 :)

This article is part of a series of articles, because 1 article would be absolutely massive.

• Part 1: Create an Azure Bot and App Registration

• Part 2: Register Bot in Teams with Teams Developer Portal

• Part 3: Delegated Permissions and Making Lambda Stateful for Oauth2

• Part 4: Building the Receiver lambda to store tokens and state

• Part 5: Finding messages, reading conversations in Teams

• Part 6 (this article): Finding attachments and reading them into Bedrock

Hey all!

In the last article we talked about how to find the root of messages to build our conversational context in Teams. That is an incredibly challenging task, as the Teams client is several raccoons in a trench coat (well, service APIs behind a front-end, same difference). Those services are generally:

• Azure Entra/AD API - Authentication and Authorization (is Teams license assigned)

• OneDrive API - personal files. Files are shared to the person they are sent to automatically in Teams.

There is some weird behavior because of this. When building a multi-person chat, and sharing files, files are shared to each person in the chat at the time of sharing the file. If a person is added later, they can see previous messages (Teams API) but not previous files (OneDrive API), as files are not batch retroactively shared. #WeirdTeams

• SharePoint API - Team/Channel files. When files are uploaded to a Channel, they are shared with that Team automatically.

To solve this, we have a robust couple functions that find, download, and encode properly the attachments that need to be added to the conversation context.

Lets walk through it.

Iterate Through the Messages

To start with, lets catch up with where the last article covered. We went through the functions that fetch the metadata and messages around “channel” type conversations (those in Teams/Channels), “personal” type messages (those in DMs), and how we “massage” the messages to scrub any junk that Teams returns to us in those API calls. There’s a lot there, it’s worth a read if you’re skipping ahead.

But all you really need to know now is that we have a list of all conversation messages in the thread. For DMs, we just respond to the single request (I’d love to add “threading”/”grouping of topics” in DM context in future, but haven’t found an elegant way yet), and for channels we read back all the messages to the root/parent message.

So that covers us up to line 11.

On line 11 and 12, we initialize a list of conversations and context, that we’ll populate in an iterative.

On line 15, we iterate through each message package that we’ve previously built to build conversation turns - the sort of atomic unit of a conversation that bedrock wants to read. We’ll walk through this more as we dig deeper into this function.

We take that turn (line 17) and append it to the conversation (line 19).

Once we have a complete conversation built in Bedrock’s style, we return it to the parent, which will do the Bedrock conversation.

For now, lets dig into how the build_conversation_turn() function works.

Every single message from a user has from.user.displayName, and if it exists, we set the sender Role as “user”. Roles in Bedrock can be either “user” (not from the model) and “assistant” (from the model).

On line 7, if there is no displayName, we assume it’s the assistant.

On line 10, we get the message.body.content, which s the text of the message. Teams annoyingly uses html encoding, so we strip as much of that with simple regex as we can, line 13-14. I think Bedrock can interpret this fine, but it annoys the humans (read: me) when reading debug logs, so I remove it anyway.

On line 17 we start building the content block, making sure to include the sender name and text, something like:

“Kyler Middleton says: Tell me the weather”

And if there’s no sender_name (it’s the bot), we just include the text. We’ll make sure to set the role later on.

Downloading the Attachments

Next we check for any attachments in the message. This isn’t a straight-forward process (obviously, it gets it own article in this series).

The attachment always has a file name (file.name) and file_extension (everything after the last period in the file name), line 8.

We check for an attribute called the file.contentUrl which is the URL address of the content. There’s different ways to authenticate and build the URL to get the file (I miss the simplicity of slack attachments dearly here), so we pass that url to the download_file_for_users() function, line 14. We’ll dig into that shortly.

There’s a couple of other gotchas here we’ll address before we dive into downloading. Bedrock has pretty strict requirements on document names, so we file the name through regex, line 18. We’re only permitted single spaces (Weird?) so we regex that as well, line 20. We also don’t permit leading or trailing spaces, so we remove those on line 22.

Many of these text manipulations could be combined into single lines, but I abhor complex code. DRY is awful, I’d much rather understand exactly what code is doing.

Pain

It’s worth including the note for this function. All my functions have notes that describe broadly what they do, but this one is largely an apology on behalf of Teams’ APIs. I don’t know why the Teams… Team decided to make all of this such a pain to work with, but I hope they stub their toes walking to their beds after turning out the lights.

This function is more than 100 lines of code, so buckle up 😭

First, on line 9, we wrap all this in a try. Why? Because we’re doing an extraordinary amount of logic here, and if we fail any of it we want our script to continue working. If we can’t fetch the attachment, we just continue with the text only. That’s just fine.

First, we check to see if the file_url contains sharepoint.com/sites/, which means the file is stored in SharePoint. We need to fetch the file from SharePoint, which turns out to be an incredibly annoying thing to do.

First, we need the site_id. Lets build a graph call to go fetch that. We establish the primitives by splitting the URL, line 13-17, and then build the URL, line 20. We require the file, line 21, via requests, and check the call, line 22. If it worked, we set the site_id to the response.id, line 23.

With that in hand, we need the default document library ID, so we build a URL to fetch THAT, line 26-29.

Then we need to extract the file path. These sometimes work just fine, and sometimes contain characters (why the heck?). We use a URL encoding function to do the heavy lifting of encoding the full URL plus document name (spaces and characters and such get encoded a bit weird in http), line 32.

Then we FINALLY finally try to download the file, line 34.

I noticed quite a few of my test downloads had a “\u202f” which is the unicode character encoding for a “narrow non-breaking space”. Why are those in the file path? I have absolutely no idea dude. But we have a whole block where we just strip that off and see if we can download the file again.

If that doesn’t work we want to throw our hands up and retire, but there’s one more method we can use - literally fricking searching SharePoint using graph for the file name, then reading through all the results, one by one, to see if any exactly match the file name.

This a slow, serialized, and generally bad way to get this working. However, it often works, so ¯\_(ツ)_/¯

If that still fails, we just give up for SharePoint paths. I don’t know another method to find the file, other than the SharePoint/Teams API devs simply providing a link to the file (please goddess).

Next up, we try to download the file from the user’s personal OneDrive. This works if the user has uploaded the file themselves in a DM context.

Again, this is a relatively “bad” solution since it requires our own user to have uploaded the file. If the file was uploaded in a DM context by another user, I think we just won’t find it. Sigh.

Lastly, we try to directly download the file via the file_url. You’d think this would be the most reliable method for downloading files. After all, this is the only method we use for slack and it works 100% of the time.

However, I have never seen this work in my testing and live auditing. I have no idea why, but I keep it because, well, it SHOULD work, right? And maybe it’ll catch some edge case that I haven’t tested yet.

Finish the Conversation

After that, the logic is pretty much the same as the Slack Vera bot - we attach the bytecode of the files to the conversation turns, pack each one with metadata to identify it as “user” or “assistant”, and pass it to the model.

The model returns a response, and we pass it to the Teams chat as a response.

It’s quite a bit slower than the Slack Vera, largely due to the vastly greater number of API requests required to do pretty much anything - finding the conversation ID, mapping different types of identifiers together, finding and testing all different paths for downloading files, etc.

It’s hard to beat Teams’ price (FREE, or at least table stakes in your EA agreements), but it remains a bear to work with. Thank goodness it’s improving, it was utterly awful a few years ago.

Summary

We covered even further the idiosyncrasies of the various APIs that Teams uses, and how they basically operate like a raccoon in a trench coat - lots of loosely collected APIs built in different standards and with differently methodologies, trying to operate as a single entity. It sometimes even works properly.

That’s probably it for the Teams Vera series. I’ve discovered Strands, an open source tool from AWS that permits building MCP-enabled Agents to do stuff in a standard way, and I’m entirely hooked. I’m barely keeping my non-AI job right now, this stuff is just too cool.

My next articles will surely follow that trend.

Good luck out there.
kyler