# Analytics Product Requirements Document

Comprehensive PRD for the quiz platform analytics section focusing on completion rates, question types, and skill mastery tracking

<Info>
This document outlines the requirements for the analytics section of the quiz platform, providing detailed insights into learner performance tracking and analysis.
</Info>

## Overview

The analytics section of the quiz platform will provide quiz creators, educators, and learners with detailed insights into learner performance. The primary focus is on tracking **completion rates** for quiz questions, with an emphasis on analyzing performance by **question type** and **skill mastery**.

<CardGroup cols={3}>
  <Card title="Track Performance" icon="chart-line">
    Monitor completion rates across different question types
  </Card>
  <Card title="Skill Analysis" icon="brain">
    Analyze learner mastery of specific skills
  </Card>
  <Card title="Data-Driven Insights" icon="lightbulb">
    Enable targeted improvements in quiz design
  </Card>
</CardGroup>

## Objectives

The analytics section aims to achieve the following key objectives:

<Steps>
  <Step title="Question Type Performance">
    Provide a clear, actionable view of how well learners are performing on different question types (Multiple Choice, True/False, Chart Selection)
  </Step>
  <Step title="Skill Mastery Insights">
    Offer insights into learner mastery of specific skills (Pattern Recognition, Risk Management) across all question types
  </Step>
  <Step title="Difficulty Assessment">
    Allow quiz creators and educators to assess performance at varying levels of difficulty (Easy, Medium, Hard)
  </Step>
  <Step title="Easy Interpretation">
    Ensure that the analytics are easy to interpret, even when questions are tagged with multiple skills
  </Step>
  <Step title="Data-Driven Decisions">
    Support data-driven decisions for improving quiz content and learner outcomes
  </Step>
</Steps>

## Key Requirements

### Dual-Track Analytics

<CardGroup cols={2}>
  <Card title="Performance by Question Type" icon="list-check" color="#4F46E5">
    Track the percentage of correct answers for each question type. This metric is independent of skills being tested, focusing purely on question format.
  </Card>
  <Card title="Performance by Skill" icon="graduation-cap" color="#10B981">
    Track the percentage of correct answers for questions tagged with each specific skill. Questions with multiple skills contribute to all associated skill metrics.
  </Card>
</CardGroup>

### Metrics to Track

For both question types and skills, the following metrics must be tracked:

<Note>
All metrics are calculated as: `(Number of correct answers / Number of questions attempted) × 100`
</Note>

- **Number of questions attempted**
- **Number of correct answers**  
- **Percentage of correct answers**

### Handling Multiple Skills per Question

<Warning>
When a question is tagged with multiple skills, a correct or incorrect answer should be counted toward the performance metrics of **all** associated skills.
</Warning>

This ensures that the analytics accurately reflect the multi-dimensional nature of the questions without requiring creators to prioritize or weight skills.

### Difficulty Levels

Questions are categorized based on a scoring system that considers:

<Steps>
  <Step title="Question Type">
    Different question types have inherent difficulty levels (True/False is generally easier than Chart Selection)
  </Step>
  <Step title="Number of Hints">
    More hints reduce the overall difficulty of a question
  </Step>
  <Step title="Topic Complexity">
    Based on the sub-skill's progression level: Basic, Intermediate, or Advanced
  </Step>
</Steps>

The analytics section should allow users to **filter performance data by difficulty level**.

### Sub-Skill Progression

Sub-skills are organized into three progression levels:

<CardGroup cols={3}>
  <Card title="Basic" icon="seedling" color="#60A5FA">
    Foundational concepts like Technical Pattern Foundations
  </Card>
  <Card title="Intermediate" icon="tree" color="#FBBF24">
    Build on basics like Market Structure Analysis
  </Card>
  <Card title="Advanced" icon="rocket" color="#F87171">
    Complex applications like Cross-Asset Analysis
  </Card>
</CardGroup>

### Small Sample Size Handling

<Tip>
To avoid misleading metrics due to small sample sizes:
- Display a warning ("Low sample size") when questions attempted < 5
- Optionally disable or gray out metrics for categories with insufficient data
</Tip>

### Overall Performance Metrics

The analytics section should provide an overall performance summary:

- Total questions attempted
- Total correct answers
- Overall percentage of correct answers

### User Interface and Visualization

The analytics dashboard should be intuitive and visually engaging:

<CardGroup cols={3}>
  <Card title="Bar Charts" icon="chart-bar">
    Performance by question type and skill
  </Card>
  <Card title="Line Graphs" icon="chart-line">
    Performance trends over time
  </Card>
  <Card title="Heatmaps" icon="table-cells">
    Distribution of difficulty levels attempted
  </Card>
</CardGroup>

### Filtering and Customization

Users should be able to filter analytics data by:

- **Difficulty Level** (Easy, Medium, Hard)
- **Sub-Skill Progression** (Basic, Intermediate, Advanced)
- **Specific Skills** or **Question Types**

## Implementation Details

### Data Collection

The platform must log each learner's interaction with quiz questions, capturing:

<Check>
Question type, skills tagged, learner's answer, and difficulty level
</Check>

### Difficulty Scoring System

The difficulty of each question is calculated using a points-based system:

#### Question Type Points

| Question Type | Points |
|--------------|--------|
| True/False | 1 |
| Multiple Choice (4 options) | 2 |
| Multiple Choice (>4 options) | 3 |
| Chart Drawing Selection | 4 |
| Chart Drawing Completion | 5 |

#### Hint Adjustment

| Number of Hints | Point Adjustment |
|----------------|------------------|
| 0 hints | +0 points |
| 1 hint | -1 point |
| 2 hints | -2 points |
| 3+ hints | -3 points |

#### Topic Complexity Points

| Sub-Skill Level | Points |
|----------------|--------|
| Basic | +1 |
| Intermediate | +2 |
| Advanced | +3 |

#### Difficulty Categorization

<Note>
- **Easy**: Score ≤ 3
- **Medium**: 4 ≤ Score ≤ 6  
- **Hard**: Score ≥ 7
</Note>

### Analytics Calculation

<Steps>
  <Step title="Performance by Question Type">
    For each question type, sum the number of questions attempted and correct answers, then calculate the percentage
  </Step>
  <Step title="Performance by Skill">
    For each skill, sum the questions tagged with that skill that were attempted and correctly answered, then calculate the percentage
  </Step>
  <Step title="Overall Performance">
    Sum all questions attempted and correctly answered across the quiz, then calculate the overall percentage
  </Step>
</Steps>

### Dashboard Layout

<Frame caption="Analytics Dashboard Structure">
```
┌─────────────────────────────────────┐
│   Overall Performance Summary       │
│   • Total questions: XX             │
│   • Correct answers: XX             │
│   • Overall percentage: XX%         │
├─────────────────────────────────────┤
│   Performance by Question Type      │
│   [Bar Chart Visualization]         │
├─────────────────────────────────────┤
│   Performance by Skill              │
│   [Bar Chart Visualization]         │
├─────────────────────────────────────┤
│   Filters                           │
│   • Difficulty Level                │
│   • Sub-skill Progression           │
│   • Question Types / Skills         │
└─────────────────────────────────────┘
```
</Frame>

## Example Scenario

Consider a learner who completes 5 episodes:

<Steps>
  <Step title="Episode 1">
    Multiple Choice, Skills: Pattern Recognition + Risk Management, **Correct**
  </Step>
  <Step title="Episode 2">
    True/False, Skill: Risk Management, **Incorrect**
  </Step>
  <Step title="Episode 3">
    Chart Selection, Skills: Pattern Recognition + Decision Making, **Correct**
  </Step>
  <Step title="Episode 4">
    Multiple Choice, Skill: Decision Making, **Correct**
  </Step>
  <Step title="Episode 5">
    Chart Selection, Skills: Risk Management + Decision Making, **Incorrect**
  </Step>
</Steps>

### Results Analysis

#### Performance by Question Type

| Question Type | Attempted | Correct | Percentage |
|--------------|-----------|---------|------------|
| Multiple Choice | 2 | 2 | 100% |
| True/False | 1 | 0 | 0% <Icon icon="exclamation-triangle" size={16} color="#F59E0B" /> |
| Chart Selection | 2 | 1 | 50% |

<Warning>
True/False shows a "Low sample size" warning due to only 1 question attempted
</Warning>

#### Performance by Skill

| Skill | Tagged Questions | Correct | Percentage |
|-------|-----------------|---------|------------|
| Pattern Recognition | 2 | 2 | 100% |
| Risk Management | 4 | 1 | 25% |
| Decision Making | 3 | 2 | 66.7% |

<Tip>
This example demonstrates how the analytics provide clear insights into both question-type performance and skill mastery, with appropriate warnings for small sample sizes.
</Tip>