# Beyond Bots
Recently a customer needed a chatbot developed that was safe, accurate, and capable of near real-time responses. In simple terms, they needed an AI system that could answer support queries flawlessly, without exposing sensitive data or drifting away from the company’s voice. Requests like this are now more frequent, as data security, latency, and response quality directly impact a company’s bottom line. AI systems that get these requirements wrong lose trust quickly and cost the business money. IBM’s 2025 report estimates the average global cost of a data breach at \$4.44 million.
But generic chatbots and off-the-shelf large language models (LLMs) often fail to meet enterprise expectations. There’s no doubt that LLMs are powerful. But these models face real constraints around token limits, context utilization, and hallucinations. These limitations are even more evident with the increasing demand for domain-specific knowledge and strict response formats. When all these aspects are at work, how do you build an AI that knows how to answer like an expert, knows what to answer based on real data, and still remains fast, safe, and controllable?
From my experience in developing models, I can say that the answer is not a single model or technique. It requires a broader architectural approach that separates what the model knows from how it responds, while combining learning with retrieval.
# The Core Challenges
Early in the design phase, four fundamental challenges became clear.
// Effective Context Limits (Beyond Token Counts)
Modern LLMs advertise context windows of 16K, 32K, or even 128K tokens. But in real-world use, anyone who works closely with these models knows their attention starts to break down much sooner. When large volumes of text are passed as context, models often underutilize information in the middle of the prompt — a phenomenon known as primacy-recency bias.
Increasing context size does not guarantee better answers. For enterprise environments, where knowledge bases can span millions of tokens, this is not a solution.
// Weak Utilization of Long-Tail Information
LLMs may ignore relevant information, misinterpret it, or overweight irrelevant sections even when the right content is present in the prompt. Works such as Lost in the Middle support this theory. That paper highlights how long-context inputs often lead to incomplete reasoning if not carefully controlled. This makes naive “dump everything into the prompt” strategies unreliable for complex, domain-heavy support systems.
// Precision vs. Performance Trade-offs in Retrieval
Retrieval introduces real-world latency and compute costs. If you retrieve too much, the increased context raises response time and dilutes model attention. If you retrieve too little, you increase the risk of hallucinations. The real challenge is not retrieval itself, but precision retrieval — guaranteeing the minimum sufficient context required for correctness without overwhelming the system or the model.
// Hallucinations Under Missing Context
LLMs rarely abstain from generating responses even when relevant information is missing. Instead, they respond confidently with generic or fabricated answers. In a support setting, this behavior is unacceptable. It directly impacts trust, correctness, and compliance.
A close study of these constraints made one thing clear: passing more context was not the solution. We needed a smarter architecture.
# The Answer: A Hybrid Architecture
Our work consistently pointed toward a hybrid approach combining retrieval augmented generation (RAG) with fine-tuned language models. The key insight was that fine-tuning and retrieval solve different problems. Fine-tuning teaches the model how to answer, and retrieval supplies what to answer. We found that forcing one method to do both leads to inefficiency, instability, or high costs. We therefore designed a system that allows both components to operate from their respective strengths.
# Using RAG for Precision Through Retrieval
We made a deliberate effort to avoid flooding the model with large volumes of raw documents. Instead, we built a searchable, curated knowledge base from our internal Q&A pairs, product manuals, technical documentation, and policy and configuration reference materials. At inference time, the retriever selects only the most relevant content chunks and inserts them into the prompt. This ensures answers are grounded in verified, real data.
We found that this approach significantly decreased hallucination rates, improved factual accuracy, and increased response speed by keeping context windows small and query-specific. However, relying solely on RAG was insufficient. Even when retrieval accuracy was very high, the outputs exhibited large variability in tone, structure, formatting, and the level of procedural detail included. These outputs indicated that while factually accurate, the responses were neither consistently structured nor reliably formatted.
In one chatbot use case specifically designed to improve convergence in inquiries, evaluation showed that even though the small language model had access to nearly 100% correct context, output correctness was only about 70%. The model was unable to extract meaning from long-context inputs and failed to maintain the conversational tone needed to guide users toward deeper technical discussions or follow-up meetings.
This revealed a fundamental limitation of retrieval: RAG provides information, but it cannot instruct a model on how to reason about or communicate within a specific domain.
# Fine-Tuning Qwen: Teaching the Model How to Answer
To improve consistency, tone, and reasoning, we fine-tuned the Qwen model on roughly 1,000 expert Q&A pairs carefully selected to align with the model’s target domain. The goal was not to teach the model facts. We wanted it to learn how to apply domain-specific language, maintain the company’s voice and communication style, follow a consistent response format, reason through answers procedurally, and handle the edge cases that arise within support workflows.
Fine-tuning adjusts how a model behaves, not what it knows — and that distinction matters. Fine-tuning the entire model can lead to catastrophic forgetting and excessive compute costs. To limit these risks, we used low-rank adaptation (LoRA) adapters. LoRA adapters allow fine-tuning of only a small set of adapter matrices while preserving the majority of the base model’s general knowledge. They also reduce the GPU memory required for fine-tuning and produce performance that is nearly equivalent to full model fine-tuning.
The results were clear. The model became substantially more consistent and nuanced. For stable, procedural questions, it produced correct answers many times without requiring any retrieval. However, as expected, the model struggled with questions about new features, updated policies, and long-tail factual queries.
In the same chatbot example, fine-tuning improved tone alignment to approximately 90% but decreased factual accuracy to approximately 50%. The lesson was reinforced once again: fine-tuning does not replace retrieval.
# Why Neither RAG Nor Fine-Tuning Alone Was Enough
These experiments gave us clarity on the trade-offs involved:
- RAG-only systems offer superior factual grounding and recency, but suffer from poor tone consistency and higher latency.
- Fine-tuned-only systems offer superior voice and structural consistency, but fail when knowledge changes or long-tail facts are needed.
Choosing only one approach meant accepting the weaknesses of the other. Combining a fine-tuned model with RAG produced better results than either method alone. Tone accuracy improved to roughly 75% — better than RAG, which had no reliable tone control, though slightly below the fine-tuned model’s 90%. Factual correctness improved to approximately 73%, exceeding both the fine-tuned model alone (approximately 50%) and RAG alone (approximately 70%). The fine-tuned model’s understanding of domain context and expected output format enabled it to better utilize and apply the retrieved context than a base model could.
Alakh Sharma is a Data Scientist at Talentica Software, a global product development company that helps startups build their products. Alakh is an Indian Institute of Science, Bangalore alumnus. He helps businesses gain a competitive edge with the adoption of reinforcement learning, machine learning, and natural language processing. You can find Alakh on LinkedIn.
