Why a Leading IT Company Needed an AI Bot
As the company’s teams grew and projects multiplied, internal knowledge became the engine of productivity — yet in practice, it was hard to access.
Documents were scattered across network drives, abandoned wikis, sprawling Confluence spaces with broken links, disorganized Notion databases, auto-deleting Slack threads, and endless email chains. Finding reliable, actionable information was a daily challenge.
This caused tangible problems:
- New hires couldn’t find answers. Onboarding stalled as newcomers struggled with basic questions like “How do I get access to staging?” or “Which proposal template is current?” Weeks were lost before meaningful contributions began.
- Experts repeated themselves endlessly. Senior engineers and product leads fielded the same questions over and over: password rotation policies, MFA configuration, escalation rules. Each interruption chipped away at time meant for architecture, delivery, and strategy.
- Keyword search failed in practice. Built-in search tools returned irrelevant noise or missed critical documents entirely. Real questions rarely matched the exact wording used in documentation.
The company hoped an AI bot could help: employees could ask questions in plain English and get precise answers, reducing interruptions and manual support. Leadership approved the project, expecting significant time savings for engineering teams.
The First Version: “Let’s Just Plug in an LLM”
The engineering team built a prototype using a common RAG-style architecture:
- Knowledge base ingestion: Documents from wikis, drives, Notion, and Slack were stored in a vector database like FAISS or Weaviate.
- Embeddings layer: Models such as OpenAI’s text-embedding-ada-002 or Sentence Transformers converted text into semantic vectors for meaning-based retrieval.
- LLM generation: GPT-3.5-turbo retrieved relevant chunks and synthesized conversational responses with source citations.
On paper, this stack seemed enough. Frameworks like LangChain or LlamaIndex offered ready-made pipelines, and a Slack-integrated interface allowed immediate testing.
At first, the bot performed well on simple queries like vacation policies or approved cloud providers. Demos impressed stakeholders — responses were instant, sources linked, and the interface polished. Confidence was high; a beta rollout to 50 users followed.
Where It Started Breaking Down
Once users tried real-world problems, the bot often failed. Examples included:
- Troubleshooting a persistent 502 error in Kubernetes under peak traffic.
- Following exact escalation steps for GDPR incidents.
- Integrating Stripe webhooks with legacy systems without disrupting live payments.
Failures were predictable:
- Generic, non-actionable answers: “Check best practices and consult your lead.”
- Dead-ends: About 60% of complex queries returned no useful information.
- Hallucinations or evasions: Occasionally the bot invented instructions or skirted specifics.
Prompt engineering and retrieval tweaks helped slightly but did not solve the core problem. The team realized the issue wasn’t the AI model — it was the lack of documented knowledge. The base contained high-level overviews and summaries, but essential operational procedures existed only in employees’ heads.
Juggling Engines: Improvements Without Solving the Core Issue
The team tried:
- Swapping LLMs: GPT-4-class models, Anthropic Claude, Llama 3.1 for on-prem control.
- Increasing context windows: From 4K to 128K+ tokens for richer retrieval.
- Advanced retrieval: Hybrid search combining BM25 keyword matching with semantic vectors, plus rerankers.
Metrics improved: retrieval precision rose, user satisfaction increased, hallucinations dropped. Yet the limit remained: if content didn’t exist, no AI could produce it. Queries like “Handling Q2 fiscal closeout variances under SOX compliance” returned blanks. Technology could polish, not invent, missing knowledge.
The Unexpected Pivot: Bot as a Diagnostic Tool
The breakthrough came when the team examined query logs rather than focusing solely on answers:
- Unanswered questions: Instances where confidence was too low or retrieval failed.
- Repeated queries: Frequently asked questions across teams, often phrased differently.
This revealed:
- Largest knowledge gaps, heatmapped by frequency, team, and business impact.
- Precise content needs, e.g., interactive flowcharts for OAuth, video walkthroughs for CI/CD pitfalls, or regional compliance decision trees.
Standard search tools had failed here — keywords alone didn’t capture intent. The bot now functioned as a practical diagnostic instrument, highlighting what employees actually needed.
The Manual Work That Made the Difference
With insight from logs, cross-functional teams of experts and writers produced targeted documentation:
- Focused on the top 20 query clusters, covering 70% of recurring issues.
- Written for both humans and machines: structured Markdown with headings, numbered steps, code blocks, comparison tables, cautionary notes, and FAQs.
The effect was immediate: within days, the unanswered query rate dropped sharply. By the end of the quarter, satisfaction reached 92%. The paradox: the AI only became truly useful after the content existed. It did not create knowledge — it amplified and surfaced it.
Main Takeaway: AI Highlights Knowledge, It Doesn’t Replace It
This story revealed the deep-seated myths. AI stands not as a fount of knowledge, but as a potent amplifier, brilliantly remixing, contextualizing, and surfacing documented truths while ruthlessly flagging omissions.
Key lessons:
- Knowledge base is primary: well-structured content is the foundation.
- AI is secondary: its effectiveness scales with the quality of input.
- Content is the real investment: careful documentation pays dividends through self-service and faster decision-making.
Without the bot, identifying what to document would have been guesswork.
Practical Takeaways for Those Wanting to Do the Same
For teams implementing AI atop internal knowledge bases:
- Plan for content creation: dedicate 20–30% of project time to writing post-launch.
- Use the bot as a “gap detector”: log queries from day one, treat failed responses as clues.
- Monitor carefully: unanswered questions, weak answers, and repeated query patterns.
- Visualize gaps: dashboards with heatmaps, owners, and fill-rate tracking improve efficiency.
Approached this way, an AI bot becomes less of an answer engine and more a tool for systematically identifying missing knowledge.
Conclusion
AI isn’t magic. It can’t invent knowledge. Solid documentation is the foundation; AI amplifies it. When applied thoughtfully, a bot accelerates knowledge creation, reveals missing content, and guides teams to write exactly what is needed — based on real data, not guesses.
The IT company described turned a simple question bot into a powerful tool for spotting knowledge gaps, and you can do the same.
Good luck with your technical writing!
Author, host and deliver documentation across platforms and devices
FAQ
Why didn’t the AI bot work as an answer engine initially?
The bot could only work with the knowledge that existed in the documentation. Most operational details lived in employees’ heads and were not written down, so even the most advanced AI models couldn’t produce accurate answers.
Did switching AI models or improving retrieval fix the problem?
Not completely. Swapping LLMs, increasing context windows, and fine-tuning retrieval improved accuracy and reduced hallucinations, but no AI can generate knowledge that isn’t documented. The core issue remained missing content.
How did the AI bot help if it couldn’t answer many questions?
By analyzing logs of failed and repeated queries, the bot highlighted exactly where knowledge gaps existed. This allowed the team to prioritize content creation effectively, turning the bot into a knowledge audit tool rather than a pure answer engine.
What kind of content proved most effective?
Clear, structured documentation works best. Examples include step-by-step guides, code snippets, annotated diagrams, FAQs, and decision trees. Content should be optimized both for humans and the AI retrieval process.
How long did it take to see improvements?
Within days of adding the first set of targeted documentation, the number of unanswered queries dropped significantly. Within a few months, user satisfaction exceeded 90%.
Can AI replace the need for writing documentation?
No. AI amplifies existing knowledge but cannot invent it. Without solid, structured content, AI cannot provide reliable answers. The real value comes from combining content creation with AI-driven insights.
How should companies use AI bots for internal knowledge bases?
Treat AI as a tool for discovering gaps in knowledge. Log queries from day one, track unanswered or poorly answered questions, cluster similar queries, and use dashboards or visualizations to guide content creation. AI should guide what to write, not replace writing itself.
What is the main takeaway from this experience?
Knowledge comes first; AI comes second. Well-structured content is the foundation. A bot becomes valuable not by magically answering questions, but by highlighting what knowledge is missing and helping teams prioritize documentation.
