How We Built a Slack AI Agent That Answers Engineering Questions in Seconds

The Problem: Documentation Exists, Nobody Can Find It

At VibeFactory.ai, we run a complex SaaS platform — dozens of microservices, multiple databases, payment integrations, CI/CD pipelines, and an AI-powered feature set that's constantly evolving. All of it documented across hundreds of pages.

But documentation is only useful if people can find it. Our engineers were doing one of three things:

• Searching Confluence/Notion/GitHub — 10-15 minutes per question, often finding outdated docs
• Asking a senior engineer — interrupts their deep work, creates a knowledge bottleneck
• Reading source code directly — works but slow, and you need to know where to look

We needed something that could answer "How does the deployment pipeline handle private projects?" in seconds, grounded in our actual documentation, without hallucinating.

The Solution: A RAG-Powered Slack Agent

We built an AI agent that lives in Slack. Engineers ask a question in natural language, and the agent:

Average response time: under 10 seconds. Average accuracy: 95%+ (because it can only answer from docs, not hallucinate).

Architecture: Three Services, Zero Complexity

The entire system has three components. No Kubernetes, no microservices, no message queues:

  Engineer asks in Slack
         |
         v
  +------------------+
  |    COMPOSIO       |  Handles Slack auth, triggers, and actions
  |  (Action Layer)   |  No webhook URL needed - outbound WebSocket
  +--------+---------+
           |
           v
  +------------------+       +------------------+
  |   YOUR AGENT     | ----> |   ZEROENTROPY     |
  |  (Python, ~200   |       |  (RAG Service)    |
  |   lines of code) |       |  Semantic search   |
  +--------+---------+       |  + reranking       |
           |                  +------------------+
           v
  +------------------+
  |   CLAUDE AI       |
  |  (Anthropic)      |
  |  Answer generation |
  +------------------+
           |
           v
  Reply posted in Slack thread
                

Component 1: ZeroEntropy (Knowledge Retrieval)

ZeroEntropy is a managed RAG service that handles the hard parts of document retrieval: chunking, embedding, indexing, and reranking. We feed it our markdown documentation and it returns the most relevant snippets for any query.

Why not just stuff everything into the LLM context? Three reasons:

• Cost — sending 100 pages of docs with every question costs 10-50x more than retrieving 5 relevant snippets
• Accuracy — LLMs get worse with more context. Focused retrieval produces better answers
• Scale — your knowledge base can grow to thousands of docs without hitting context limits

ZeroEntropy uses hybrid search (BM25 keyword matching + dense vector embeddings) with a reranker, so it handles both "what's the schema for project_shares table?" (exact match) and "how do we prevent double-charging users?" (semantic match).

Component 2: Composio (Slack Integration)

Composio is the action layer — it handles Slack OAuth, message triggers, and reply actions through a single SDK. No webhook URLs, no ngrok, no public endpoints required.

This is the key architectural decision that made deployment trivial: Composio uses outbound WebSocket connections (via Pusher) instead of inbound webhooks. Your agent connects to Composio, not the other way around. This means:

• No public URL needed — run behind a corporate firewall, on a Raspberry Pi, or on your laptop
• No SSL certificates — Composio handles TLS
• No Slack app configuration — Composio manages OAuth tokens and scopes
• 100+ integrations available — same SDK works for GitHub, Jira, Linear, Notion, and more

Component 3: Claude AI (Answer Generation)

Claude Sonnet 4 generates the actual answer from retrieved snippets. The system prompt is carefully crafted:

• Only answer from provided context — if the docs don't cover it, say so
• Always cite sources — include document paths so engineers can verify
• Ask for clarification — if the question is ambiguous, ask before guessing
• Flag stale docs — if context contains contradictions, mention it
• Format for Slack — use bold, code blocks, and lists (not markdown headers)

Real-World Example

Here's an actual screenshot from our Slack workspace — the agent answering questions about platform architecture and pricing in real time:

Answers arrive in under 10 seconds. The agent pulls from internal documentation, cites sources, and formats responses for Slack. No hallucination, no guessing — if the docs don't cover it, the agent says so and creates a GitHub issue to track the gap.

The Self-Healing Feature: Auto-Detecting Documentation Gaps

This is the feature we didn't plan but turned out to be the most valuable: when the agent can't find relevant documentation for a question, it automatically creates a GitHub issue tagged [Doc Gap].

This turns every unanswered question into an actionable documentation task. After two weeks of running the agent, we had a prioritized list of exactly what was missing from our docs — ranked by how often engineers asked about it.

What It Costs

Full transparency on running costs for a team of 10-50 engineers:

Compare that to one senior engineer spending 30 minutes per day answering architecture questions: at $150K/year salary, that's $3,125/month in lost productivity. The agent pays for itself in the first week.

What Else Can This Architecture Power?

The same three-component pattern (RAG retrieval + LLM generation + action layer) applies to dozens of business use cases:

Why RAG Beats Fine-Tuning for Enterprise Knowledge

Teams often ask: "Should we fine-tune an LLM on our data instead?" For internal knowledge use cases, RAG wins decisively:

How Long Does It Take to Build?

If you have your documentation ready, here's a realistic timeline:

Total: ~2 weeks from zero to production, including documentation preparation. The agent code itself is about 200 lines of Python.

Frequently Asked Questions