Quiz System - XPs, Classification, and Analytics

Launch a high-retention practice system that is simple to use, fair to learners, and useful to creators—while keeping taxonomy light, analytics strong, and recommendations reliable.

Product Principles

XP ≠ Skill

XP rewards engagement; skill is inferred separately

Simple Beats Smart

4 domains × 3 Blooms; no difficulty-based XP

Spec-First Generation

Classification decided upstream; post-gen validation is guardrail

Reliability Over Theory

κ & AC1 ≥ 0.75 before wide rollout

Accessibility & Fairness

No penalties for help-seeking or slower devices

Ship, Measure, Iterate

Add complexity only when data demands it

System Architecture

Learner → Quiz Player → XP Engine → XP Ledger
                ↘︎ events       ↘︎ dashboards

Creator → Builder/AI → Classification → Questions Store
                          ↘︎ Item Health
                          ↘︎ Coverage Analytics

Telemetry → Analytics Layer → Recommender
              (Mastery/Health/Spacing)

Admin/QA → Reliability Suite → Classifier Tuning

High-Level System Overview

XP & Gamification

XP Rules

XP is immediate and predictable, focused on engagement rather than difficulty

Base Rewards

Correct Answer: +10 XP
Incorrect Answer: +2 XP (participation reward)

Bonus System

Perfect Score

+10 XP for 100% accuracy

Speed Bonus

+10 XP if accuracy ≥80% and faster than median

Review Bonus

+5 XP for solution review + reflection

Streak Bonus

+5 XP per day, max 7 days (+35 XP cap)

Daily soft cap: After 300 XP/day, payouts at 50% rate to prevent grinding

Gamification Features

Seasons & Leagues

Monthly seasons with XP-based leagues (Bronze/Silver/Gold). Leaderboards reset each season.

Cell-Aligned Badges

Earn badges like "RISK–Analyze Adept" for 30 verified corrects in that cell. Tiers at 10/30/60.

Session Goals

Auto-set daily goals (e.g., "Earn 60 XP today") with progress tracking.

Anti-Exploit & Accessibility

No-Speed Mode: Swaps Speed Bonus for Review Bonus to accommodate learners who need more time or have device limitations

Classification Model

4×3 Matrix (12 Cells)

Domains

Domain	Focus	Example Applications
ANALYSIS	Reading/interpreting market data	Pattern recognition, signal validation
STRATEGY	Choosing approach and playbook	Plan selection, setup comparison
RISK	Position sizing and capital management	Stop placement, size calculation
EXECUTION	Order mechanics and timing policy	Order type selection, venue choice

Bloom's Levels

Apply

One rule on one input

Analyze

Compare/synthesize ≥2 inputs

Evaluate

Pick best option with justification

Classification Precedence

Precedence order: EXECUTION > RISK > STRATEGY > ANALYSIS

Hybrid Classifier

Rules Layer

Keyword markers + precedence resolution

LLM Challenge

Request domain & Bloom with "why-not" reasoning for other domains

Decision Logic

Rules == LLM → high confidence; else apply precedence or review

Reliability Target

κ & AC1 ≥ 0.75 on balanced 12-cell set

Content Creation

Doc-Driven AI Pipeline

{
  "domain": "RISK",
  "bloom": "Analyze",
  "constraints": ["policy-level execution only"],
  "grounding": ["doc://ch12#para3", "doc://ch12#fig2"]
}

AI Content Generation Flow

Manual Builder Features

Coverage-aware prompts ("Low on EXECUTION–Analyze")
Creator chooses domain + bloom or accepts auto-suggestion
Parametric templates offered (not required)
Same validator ensures consistency

Analytics Layer

Item Health Metrics

Metric	Description	Use Case
Success Rate	Correct answers per item	Basic performance
Discrimination	Δ success between top/bottom quartiles	Item quality
Abandon Rate	Quits/timeouts per item	Difficulty indicator
Time Z-score	Deviation from median	Ambiguity detection

Health status progression: new → stable → needs_edit → quarantined

Mastery Tracking

Beta model per cell: α = 1 + correct, β = 1 + incorrect Mastery = α/(α+β)

Spacing & Review

If days_since_last_correct > 7 and mastery < 0.70 → inject review
Light-touch spacing without heavy scheduler

Dashboard Types

Educators

Coverage analytics, item health, impact metrics

Learners

Mastery by cell, streaks, recommended focus

Platform

Retention metrics, reliability scores, inventory health

Parametric Templates

Why Parametric Templates?

Prevent Gaming

Varied surface prevents pattern matching

Stable Labels

Classification consistency with creativity

Better Transfer

Spans different market regimes

Creator Freedom

Express nuance within quality guardrails

Template Structure

Each template includes:

Inputs/Slots: {ASSET}, {TIMEFRAME}, {ENTRY_RULE}
Constraints: Value ranges, mutually exclusive combos
Answer Rule: How to compute correct choice
Distractors: Principled wrong answers
Validation: Classification guardrails

Example Template: Position Size Calculation

Prompt: Account {EQUITY}. Risk {RISK_PCT}% per trade. 
        Entry {ENTRY}, stop {STOP}. What size?

Answer Rule: size = (EQUITY × RISK_PCT) / |ENTRY–STOP|

Distractors:
- Swap TP for stop
- Percent-of-equity share count
- Decimal errors

Implementation Timeline

Week 1

XP engine with base rewards, bonuses, streaks, session goals

Week 2

4-domain/3-Bloom classifier, precedence rules, builder UI

Weeks 3-4

Performance tracking, item health, dashboards v0

Month 2

LLM validator, confidence scoring, reduce manual review to ~20%

Month 3

Beta mastery, recommender v1, spaced review, badges & leagues

Month 6

Consider info_complexity if analytics justify (analytics-only)

Success Metrics

Primary Metrics

κ & AC1 ≥ 0.75 (balanced 12-cell set)
D7 retention ≥ 40%; D30 ≥ 20%

Secondary Metrics

+8–12% improvement in cell-level mastery within 7 days
Creator efficiency: < 90s median to confirm labels
Content health: < 5% traffic to needs_edit/quarantined items

Alert Thresholds

Speed bonus payout > 40% → tighten normative threshold
Any domain < 15% inventory for 2 weeks → trigger creator challenges

Risk Mitigation

Risk	Mitigation
Speed farming	Accuracy gate + normative pace + daily cap
Label drift	Precedence rule + balanced validation
Supply imbalance	Coverage-aware prompts + creator challenges
Accessibility	No-Speed Mode with Review Bonus
Data sparsity	Minimum attempts for quarantine

Future Enhancements

Phase A (Months 1-2)

Creator impact tracking
Basic personalization
Friendly competitions

Phase B (Months 3-6)

Creator payments based on learning impact
Study groups
Content difficulty ranking (analytics-only)

Phase C (Months 6-12)

Advanced predictions
Seasonal narratives
Mentorship programs

All future features would be for analytics only (no XP impact) and added carefully to avoid complexity bloat