Comms Arc 3: AI-Drafted Replies for SuperAdmin Review
What & Why
With Arc 2's Conversations inbox live, the SuperAdmin reads every reply in the app — but typing each response by hand is the bulk of the time. This task adds an AI assistant that, for any inbound message, generates a draft reply pre-loaded into the composer using the sender's context (their family/athletes/teams/events), the recent outbound history we sent them, and a curated knowledge base of operational answers admins can edit. The SuperAdmin always reviews and clicks Send — no auto-send, no scheduled AI replies. Per the user's direction, the SuperAdmin remains the single approval point for every outbound reply, and the AI's job is to make that approval fast.
Done looks like
- Inside any conversation thread (from Arc 2), a "Draft a reply" button sits next to the composer. Clicking it calls the AI, which streams a draft reply into the composer field within ~5 seconds.
- The draft is pre-filled — the SuperAdmin can edit any character, can discard and start over, can ignore the AI entirely and type from scratch. Nothing is sent without an explicit Send click.
- Alongside the draft, a small "Sources used" panel shows the knowledge-base entries the AI drew from (with titles linked to the entry, so the admin can verify or edit the underlying entry if the answer was wrong).
- A "Knowledge base" tab is added to the Communication Hub. SuperAdmin can create/edit/delete FAQ-style entries: title, body (markdown), tags, and an optional "applies to" filter (e.g. "registration", "billing", "schedule"). These entries are the AI's primary reference material.
- The AI is given context for each draft: the inbound message text, the sender's linked person record (family + registered athletes + current teams + upcoming events) when recognized, the last 3 outbound messages we sent them (subject + body excerpts pulled from ), and the top-K most relevant knowledge-base entries (selected by simple keyword match in v1, no embeddings).
- The AI is constrained by a system prompt to: stay in the academy's voice, never invent facts (especially names/dates/dollar amounts/policies), explicitly say "I don't have that information — let me check and get back to you" when the knowledge base doesn't cover the question, never promise refunds or schedule changes, and always sign off as the SuperAdmin or the academy by name.
- A simple usage telemetry counter on the Knowledge Base tab shows: drafts generated this week, drafts sent unmodified, drafts edited then sent, drafts discarded. Lets the SuperAdmin see whether the assistant is helping and which knowledge entries are getting used.
- Per-SuperAdmin rate limit (~30 draft generations per hour) prevents accidental cost runaways. Limit overage shows a clear toast.
Out of scope
- Auto-sending any reply. Every send remains a manual click. (Per user.)
- Embedding-based retrieval over the knowledge base. v1 uses keyword/tag match — fast, debuggable, free. Embeddings are a follow-up if recall becomes the bottleneck.
- Multi-language drafts. v1 is English. The model can handle other languages if the inbound is in another language, but no UI surface for forced translation.
- Letting non-SuperAdmin roles see drafts or use the Knowledge Base editor.
- Bulk-replying to multiple threads with one click.
- Auto-classifying threads by topic. The composer always offers a draft regardless; the admin decides whether to use it.
Steps
1.
Schema — Add a table (title, body, tags array, applies-to optional, created/updated timestamps, author user id) and an table (one row per draft generation: thread id, draft text, sources cited as ids, generated-at, terminal-state which is one of ///, sent-email-log-id when sent). Use existing Drizzle conventions. No primary-key type changes. 2.
Knowledge base CRUD — SuperAdmin-only POST/GET/PATCH/DELETE for entries. List endpoint supports tag filtering. UI is a simple table with inline edit modal; no fancy editor — markdown textarea with a live preview is sufficient. 3.
Context builder — A server-side helper that, given a thread id, returns the structured context the AI needs: the inbound message, sender person record + relations (family, athletes, teams, events), last 3 outbound messages, top 5 knowledge-base entries by keyword match against the inbound subject + body. Capped at a generous but bounded total token estimate so a single chatty thread can't blow the prompt size. 4.
AI draft endpoint — SuperAdmin-only POST that takes a thread id, builds the context via the helper, calls the configured AI provider with a versioned system prompt that encodes the constraints listed in "Done looks like", returns the draft text plus the list of source ids cited. Records an row in state. Uses the existing rate-limit pattern (advisory lock + count) for the 30/hour cap. 5.
Composer integration — The Conversations thread view gets a "Draft a reply" button. Clicking it streams the response into the composer (or a non-streamed swap if streaming adds disproportionate complexity for v1; both acceptable). The "Sources used" panel renders alongside. 6.
Outcome tracking — When the SuperAdmin clicks Send from a draft-prefilled composer, the existing reply-send endpoint marks the latest pending draft for that thread as if the body matches the original draft byte-for-byte, otherwise, and links the sent id. Discarding (clearing the composer or generating a new draft over an old one) marks the prior draft as . 7.
Telemetry view — A small panel on the Knowledge Base tab shows the four counts for the trailing 7 days, plus a top-5 list of most-cited knowledge entries. 8.
Tests — Vitest for: rate limit, context builder picks the right knowledge entries by tag and keyword, system prompt is included verbatim in the AI call, outcome tracking correctly classifies vs , knowledge base CRUD enforces SuperAdmin. RTL test for the composer integration showing draft text appearing, sources rendering, and Send marking the draft outcome correctly. 9.
Docs — Update with an "AI-drafted replies" subsection: where the system prompt lives, how to add or edit knowledge base entries, the constraint that drafts always require human Send, and a note on the rate limit + which env var holds the AI API key.
Architectural constraints:
- The AI never sends anything. The endpoint returns text only; the existing reply-send endpoint from Arc 2 is the only outbound path. This is enforced by the endpoint having no Resend client access at all.
- The system prompt lives in a versioned constant in the server code so it's reviewable in source. Do not load it from a database row that an admin could edit without engineering review.
- Use the existing integration that's already installed; do not introduce a second AI SDK or provider.
- Knowledge base entries are markdown rendered server-side into a sanitized form when shown to admins. No raw HTML execution path.
- This task depends on Arc 2 being merged — there's no thread to draft a reply for until the Conversations inbox exists.
Relevant files
- - - - -
No comments:
Post a Comment