A support chatbot that hallucinates prices or invents policies is worse than no chatbot. Production RAG for Arabic support docs needs clean ingestion, hybrid retrieval with reranking, strict prompts with citations, and weekly evaluation on real Saudi questions.
This guide covers ingestion, embeddings, prompting, evaluation, and operations for Arabic help-center chatbots.
1. Ingestion that preserves structure
Start with help articles as markdown with title, slug, updated date, and section headings intact. Strip navigation, footers, and duplicate CTAs. Keep tables for pricing and fees, since they answer 30 percent of tickets.
Split by heading first, then by size to 400 to 600 tokens with 10 percent overlap. Store metadata of doc id, title, slug, section, lang, and version. Embed normalized Arabic text for retrieval but cite original text for display. This handles alef variants and tatweel without showing normalized text to users.
Version embeddings by content hash. When an editor updates a Fatoora fee, re-embed only changed chunks. Keep an audit of which chunk versions answered which ticket for compliance.
2. Embeddings and vector store
For Arabic docs, multilingual-e5-large with a reranker gives best results in our tests. Store vectors in Postgres with pgvector alongside your app data to avoid another service. Use HNSW index with cosine distance, plus full-text GIN index for hybrid search.
Hybrid retrieval is essential for Arabic. Vector finds semantic matches for dialect queries like وش سعر الاشتراك, full-text finds exact SKUs and invoice numbers. Fuse with reciprocal rank, retrieve top 8, then rerank to top 3 with bge-reranker or Cohere rerank multilingual.
Cache embeddings for frequent queries and precompute for top 100 FAQs. This cuts cost 50 percent and p95 latency under 1.2 seconds.
3. Prompting with guardrails
System prompt must state Saudi support agent, answer in Saudi-friendly MSA, no Egyptian dialect words, prices in SAR only, no invented policies, and cite sources with links. Provide three few-shots of ideal answers from your best agents, including one refusal case.
User message should include conversation history summarized to 300 tokens plus retrieved chunks with titles. Ask for concise answer plus sources list. Refuse when top score is low with a template in Arabic that routes to human with ticket prefilled.
Never allow price or legal advice without citation. If chunks conflict, answer with the newest version date and flag for review. Log every prompt, chunks, and scores for weekly review.
4. Evaluation that prevents regressions
Keep 100 real questions from tickets covering dialect, prices, returns, Fatoora, and shipping to Jazan. Score faithfulness by human review, citation accuracy automatically, and dialect leakage via word list. Run weekly after each content update.
Track bad-answer rate, escalation rate, and resolution without human. Target under 5 percent hallucination, over 70 percent auto-resolution for FAQs. Review failures every Monday with support leads and fix source docs, not just prompts. Most RAG failures are bad docs, not bad models.
5. UX, cost, and operations
Stream answers with citations clickable to the exact section. Show confidence as high or needs review, not percentages that confuse. Offer WhatsApp handoff with transcript. For voice notes, transcribe with Whisper then run same RAG.
Cost control is cache plus rerank fewer plus summarize threads. Small 7B models handle classification and summarization, large models handle final answer. Host small locally for PDPL-sensitive logs.
Bottom line is RAG is 20 percent model and 80 percent data pipeline plus evaluation. Clean chunks, hybrid plus rerank, cited prompts, and weekly Saudi evals beat any model swap.
