# Plan — Native Questionnaire Engine, Part 1 (core engine + sequential builder)

> Status: **DESIGNED, not built.** Saved 2026-06-06 for a future session.
> This replaces the broken n8n conversational bot with a platform-native,
> sequential questionnaire engine. See memory `bot-not-replying-parse-node`
> (the n8n "Parse Incoming Message" node misclassifies every inbound and the
> bot replies to nobody) — the operator's decision is to build native.

## Scope of THIS plan

The questionnaire "system" is **four independent sub-projects**; this plan is
**Part 1 only** (the foundation). The other three get their own design+build
cycles later and plug into this:

1. **Part 1 (this) — core engine + sequential builder.**
2. Part 2 — intake routing / cohort classification (investor / jobseeker /
   distributor; menu / keyword / AI).
3. Part 3 — outcomes / integrations (tag, notify agent, create LeadRat lead,
   email CV — needs email-attachment support added to `EmailService`).
4. Part 4 — cohort visibility (filter contacts by type + questionnaire status).

**Future vision (informs the architecture):** Part 1 is the first building
block of a future **chatbot that connects questionnaires and conversation
flows**. So questionnaires are entities any trigger/flow can invoke, with a
clean `start(contact, questionnaire)` entry and a completion event that can
later hand off elsewhere.

## Decisions (locked during brainstorming, 2026-06-06)

- **Triggers:** inbound **keyword** (a new `FlowRule` action) **and** **QR /
  campaign** action. (Manual-enroll and default-intake deferred.) Both chosen
  triggers are inbound-driven, so the 24h window is open when Q1 is sent — no
  template needed for the first question.
- **Question types (all four):** `free_text`, `single_choice` (numbered text
  menu), `validated_text` (email / number / phone / date, re-prompt on bad
  input), and `interactive` (native WhatsApp reply-buttons / list).
  `single_choice` and `interactive` share `options`; they differ only in render
  mode (text menu vs native interactive). `MessageService.send` already
  supports the interactive message type.
- **Answers:** always recorded in the session's `responses` (full submission
  history). A question can **optionally** map to a contact attribute
  (`mapToAttribute`) → `contact.attributes[key]` (reuses
  `ContactFieldDefinition`; feeds Part 4 filters).
- **Exclusivity:** an active session takes **exclusive control** of the
  contact's replies — captures the answer and **suppresses** other keyword
  FlowRules + the n8n flow-dispatch for that contact until complete/abandoned.
  **Opt-out (STOP) is always honored** and ends the session.
- **Abandonment:** inactivity **nudge at ~24h** (one reminder template, since
  the window may be closed) → **abandoned at ~48h** (no completion outcome).
  Both thresholds configurable per questionnaire. `nudgeTemplateName` optional —
  if unset, skip the nudge and just abandon at the deadline.
- **Completion outcome (Part 1):** apply a configured **tag** (`completionTag`).
  This is the seam Part 3 expands into the full outcome set.
- **Architecture:** questionnaires get **their own tables**; triggers **reuse
  the existing `FlowRule` system** (a `start_questionnaire` action). One place
  operators build triggers (Automation); composable for the future chatbot.
- **Builder UI:** a **new "Questionnaires" nav area** with a **linear** editor
  (add / reorder ↑↓ / edit-inline / delete) — deliberately NOT the node-graph
  Automation the operator dislikes.

## Existing infrastructure this builds on (verified)

- **Inbound hook:** `BasicFlowEngine.dispatch({ waId, text, type })`
  (`apps/api/src/flows/basic-flow-engine.ts`) runs synchronously after the
  message is persisted, BEFORE the n8n `flow-dispatch` queue job. Wired via the
  shared `onInboundMessage` hook in
  `packages/shared/src/meta/envelope-processor.ts`.
- **FlowRule** model + `FlowRulesService` + `BasicFlowEngine` trigger/action
  pattern (trigger: `any_inbound`|`keyword`; actions: `send_template`,
  `apply_tag`). QR campaigns already auto-generate a keyword FlowRule.
- **Send:** `MessageService.send(msg, opts)` — free-form **text** allowed while
  the 24h window is open (checked against `Conversation.windowExpiresAt`);
  **template** allowed anytime; **interactive** message type supported.
- **Contact:** `attributes` JSON, `tags[]`; `TagsService.applyToContact`,
  `ContactsService.patch({attributes})`, `ContactFieldDefinition` for typed
  custom fields. `Conversation` tracks the 24h window (refreshed each inbound).
- **No per-contact session/state exists today** — Part 1 adds it.
- **Worker repeat-job pattern:** `reminder-tick` (scheduled BullMQ job →
  internal API endpoint) is the template to mirror for the inactivity sweep.

## Section 1 — Data model (additive migration)

A questionnaire **definition** keeps its questions as an ordered JSON array
(matches the `FlowRule` JSON-config convention; trivial reorder/edit). Per-
contact runtime is a real table (queryable + sweepable).

```prisma
model Questionnaire {
  id                String   @id @default(uuid())
  name              String
  description       String?
  isActive          Boolean  @default(true) @map("is_active")
  language          String   @default("en")
  questions         Json     @default("[]")   // ordered Question[] (below)
  completionTag     String?  @map("completion_tag")
  nudgeTemplateName String?  @map("nudge_template_name")
  nudgeAfterHours   Int      @default(24) @map("nudge_after_hours")
  abandonAfterHours Int      @default(48) @map("abandon_after_hours")
  createdAt         DateTime @default(now()) @map("created_at")
  updatedAt         DateTime @updatedAt @map("updated_at")
  sessions          QuestionnaireSession[]
  @@map("questionnaires")
}

// Question — JSON shape (stable uuid `id` per entry):
// { id, prompt,
//   type: "free_text" | "single_choice" | "validated_text" | "interactive",
//   options?: { id, label }[],                 // single_choice + interactive
//   validation?: "email"|"number"|"phone"|"date", // validated_text
//   mapToAttribute?: string,                    // optional contact-attribute key
//   required?: boolean }                        // default true

enum QSessionStatus { active completed abandoned }

model QuestionnaireSession {
  id              String        @id @default(uuid())
  questionnaireId String        @map("questionnaire_id")
  questionnaire   Questionnaire @relation(fields: [questionnaireId], references: [id])
  contactId       String        @map("contact_id")
  contact         Contact       @relation(fields: [contactId], references: [id])
  waId            String        @map("wa_id")
  status          QSessionStatus @default(active)
  currentIndex    Int           @default(0) @map("current_index")
  responses       Json          @default("[]")  // [{questionId,prompt,type,answer,optionId?,answeredAt}]
  lastActivityAt  DateTime      @default(now()) @map("last_activity_at")
  nudgedAt        DateTime?     @map("nudged_at")
  startedAt       DateTime      @default(now()) @map("started_at")
  completedAt     DateTime?     @map("completed_at")
  createdAt       DateTime      @default(now()) @map("created_at")
  updatedAt       DateTime      @updatedAt @map("updated_at")
  @@index([contactId, status])
  @@map("questionnaire_sessions")
}
```

- **At most one active session per contact:** add a **partial unique index**
  `CREATE UNIQUE INDEX ... ON questionnaire_sessions (contact_id) WHERE status='active';`
  in the migration (Prisma can't express partial-unique in schema; add raw SQL,
  same approach as other hand-written migrations).
- Add the `sessions` back-relation on `Contact`.

## Section 2 — Runtime

- **Trigger:** add a `FlowRule` action `{ kind: "start_questionnaire",
  questionnaireId }` to the shared `FlowAction` schema + `BasicFlowEngine.execute`.
  Add optional `startQuestionnaireId` to QR campaigns. Both →
  `QuestionnaireEngine.start(contact, questionnaireId)`.
- **Dispatch order** (`BasicFlowEngine.dispatch`): (1) QR scan note (existing) →
  (2) look up an **active** `QuestionnaireSession` for `waId`. If found →
  `QuestionnaireEngine.handleReply(session, incoming)` and return **handled**.
  If none → existing FlowRule matching (a matched `start_questionnaire` →
  `start(...)`, also returns handled).
- **`onInboundMessage` returns `{ handled: boolean }`;** `processEnvelope` skips
  the n8n `flow-dispatch` enqueue when `handled`. Opt-out keyword mid-session:
  end the session + record opt-out (consent path still runs).
- **`QuestionnaireEngine`** (`apps/api/src/questionnaires/questionnaire-engine.service.ts`):
  - `start(contact, questionnaireId)` — create session (currentIndex 0), send Q1.
  - `handleReply(session, incoming)` — validate the reply for the current
    question; **invalid** → re-prompt with a hint (no advance); **valid** →
    record the response (+ optional `mapToAttribute` write), advance
    `currentIndex`; more questions → send next; else → `complete`.
  - `complete(session)` — apply `completionTag` (if set), status=completed,
    `completedAt`.
  - `sweepInactive()` — nudge / abandon (below).
  - `sendQuestion(session, question)` — render per type via `MessageService.send`
    (window open after inbound): free_text/validated → plain text; single_choice
    → numbered text menu; interactive → WA interactive message. Reply parsing
    maps a number / option-keyword / button-id back to an option.
- **Inactivity sweep:** a worker repeat job `questionnaire-sweep` (mirror
  `reminder-tick`) → internal endpoint → `sweepInactive`: for `active` sessions
  where `now - lastActivityAt > nudgeAfterHours` and `nudgedAt` is null and a
  `nudgeTemplateName` exists → send that template, set `nudgedAt`; where
  `now - lastActivityAt > abandonAfterHours` → status=abandoned.

## Section 3 — Builder UI + API

- **API** (`apps/api/src/questionnaires/`): `GET/POST/PATCH/DELETE
  /api/questionnaires` (Zod-validated, same shape as `flow-rules`); read-only
  `GET /api/questionnaires/:id/sessions` for recent submissions.
- **Builder UI** (`apps/web`): a new **Questionnaires** nav area. List of
  questionnaires + a **linear editor**: ordered question list with add / reorder
  (↑↓) / edit-inline / delete; per-question type + options + validation +
  `mapToAttribute`; questionnaire settings (completion tag, nudge template,
  timings). The existing `BasicFlowDialog` gains a `start_questionnaire` action
  option (pick a questionnaire). A minimal read-only **submissions list** per
  questionnaire (contact + answers + status) so answers are visible now (richer
  filtering is Part 4).

## Section 4 — Testing + rollout

- **shared (TDD):** pure helpers — per-type answer validation, menu/interactive
  reply parsing, advance/next-index logic, render-prompt builders.
- **api:** `QuestionnaireEngine` (start / handleReply happy + invalid + complete
  + abandon) with mocked Prisma + `MessageService`; the `start_questionnaire`
  FlowRule action; the `handled` signal suppressing dispatch.
- **worker:** `questionnaire-sweep` processor (nudge vs abandon).
- **web:** the linear editor (add/reorder/edit/delete) + the FlowRule action option.
- **Rollout:** additive Prisma migration (`migrate deploy` against the live DB
  with sourced `.env`, then `generate`); rebuild `@whatapp/shared` if its src
  changed; build + `pm2 restart channels-api channels-worker` + `/health`
  check (unit tests don't catch cross-module DI breaks — see memory
  `deploy-verify-and-di`); web ships on the next `publish-web.sh`.

## Build order (suggested)

1. Migration + Prisma models (Section 1).
2. shared pure helpers + tests (validation, parsing, advance).
3. `QuestionnaireEngine` + module + CRUD controller/service + tests.
4. FlowRule `start_questionnaire` action + `onInboundMessage` handled-signal +
   `BasicFlowEngine` active-session interception + tests.
5. Worker `questionnaire-sweep` + internal endpoint + tests.
6. Web: Questionnaires nav + linear editor + FlowRule action option + submissions list.
7. Verify (lint/typecheck/per-package tests/build), deploy api+worker, publish web.