Analytics Product Requirements Document

You can also view the article in Markdown

This document outlines the requirements for the analytics section of the quiz platform, providing detailed insights into learner performance tracking and analysis.

Overview

The analytics section of the quiz platform will provide quiz creators, educators, and learners with detailed insights into learner performance. The primary focus is on tracking completion rates for quiz questions, with an emphasis on analyzing performance by question type and skill mastery.

Track Performance

Monitor completion rates across different question types

Skill Analysis

Analyze learner mastery of specific skills

Data-Driven Insights

Enable targeted improvements in quiz design

Objectives

The analytics section aims to achieve the following key objectives:

1

Question Type Performance

Provide a clear, actionable view of how well learners are performing on different question types (Multiple Choice, True/False, Chart Selection)

2

Skill Mastery Insights

Offer insights into learner mastery of specific skills (Pattern Recognition, Risk Management) across all question types

3

Difficulty Assessment

Allow quiz creators and educators to assess performance at varying levels of difficulty (Easy, Medium, Hard)

4

Easy Interpretation

Ensure that the analytics are easy to interpret, even when questions are tagged with multiple skills

5

Data-Driven Decisions

Support data-driven decisions for improving quiz content and learner outcomes

Key Requirements

Dual-Track Analytics

Performance by Question Type

Track the percentage of correct answers for each question type. This metric is independent of skills being tested, focusing purely on question format.

Performance by Skill

Track the percentage of correct answers for questions tagged with each specific skill. Questions with multiple skills contribute to all associated skill metrics.

Metrics to Track

For both question types and skills, the following metrics must be tracked:

All metrics are calculated as: (Number of correct answers / Number of questions attempted) × 100

  • Number of questions attempted
  • Number of correct answers
  • Percentage of correct answers

Handling Multiple Skills per Question

When a question is tagged with multiple skills, a correct or incorrect answer should be counted toward the performance metrics of all associated skills.

This ensures that the analytics accurately reflect the multi-dimensional nature of the questions without requiring creators to prioritize or weight skills.

Difficulty Levels

Questions are categorized based on a scoring system that considers:

1

Question Type

Different question types have inherent difficulty levels (True/False is generally easier than Chart Selection)

2

Number of Hints

More hints reduce the overall difficulty of a question

3

Topic Complexity

Based on the sub-skill's progression level: Basic, Intermediate, or Advanced

The analytics section should allow users to filter performance data by difficulty level.

Sub-Skill Progression

Sub-skills are organized into three progression levels:

Basic

Foundational concepts like Technical Pattern Foundations

Intermediate

Build on basics like Market Structure Analysis

Advanced

Complex applications like Cross-Asset Analysis

Small Sample Size Handling

To avoid misleading metrics due to small sample sizes:

  • Display a warning ("Low sample size") when questions attempted < 5
  • Optionally disable or gray out metrics for categories with insufficient data

Overall Performance Metrics

The analytics section should provide an overall performance summary:

  • Total questions attempted
  • Total correct answers
  • Overall percentage of correct answers

User Interface and Visualization

The analytics dashboard should be intuitive and visually engaging:

Bar Charts

Performance by question type and skill

Line Graphs

Performance trends over time

Heatmaps

Distribution of difficulty levels attempted

Filtering and Customization

Users should be able to filter analytics data by:

  • Difficulty Level (Easy, Medium, Hard)
  • Sub-Skill Progression (Basic, Intermediate, Advanced)
  • Specific Skills or Question Types

Implementation Details

Data Collection

The platform must log each learner's interaction with quiz questions, capturing:

Question type, skills tagged, learner's answer, and difficulty level

Difficulty Scoring System

The difficulty of each question is calculated using a points-based system:

Question Type Points

Question TypePoints
True/False1
Multiple Choice (4 options)2
Multiple Choice (>4 options)3
Chart Drawing Selection4
Chart Drawing Completion5

Hint Adjustment

Number of HintsPoint Adjustment
0 hints+0 points
1 hint-1 point
2 hints-2 points
3+ hints-3 points

Topic Complexity Points

Sub-Skill LevelPoints
Basic+1
Intermediate+2
Advanced+3

Difficulty Categorization

  • Easy: Score ≤ 3
  • Medium: 4 ≤ Score ≤ 6
  • Hard: Score ≥ 7

Analytics Calculation

1

Performance by Question Type

For each question type, sum the number of questions attempted and correct answers, then calculate the percentage

2

Performance by Skill

For each skill, sum the questions tagged with that skill that were attempted and correctly answered, then calculate the percentage

3

Overall Performance

Sum all questions attempted and correctly answered across the quiz, then calculate the overall percentage

Dashboard Layout

┌─────────────────────────────────────┐
│   Overall Performance Summary       │
│   • Total questions: XX             │
│   • Correct answers: XX             │
│   • Overall percentage: XX%         │
├─────────────────────────────────────┤
│   Performance by Question Type      │
│   [Bar Chart Visualization]         │
├─────────────────────────────────────┤
│   Performance by Skill              │
│   [Bar Chart Visualization]         │
├─────────────────────────────────────┤
│   Filters                           │
│   • Difficulty Level                │
│   • Sub-skill Progression           │
│   • Question Types / Skills         │
└─────────────────────────────────────┘

Analytics Dashboard Structure

Example Scenario

Consider a learner who completes 5 episodes:

1

Episode 1

Multiple Choice, Skills: Pattern Recognition + Risk Management, Correct

2

Episode 2

True/False, Skill: Risk Management, Incorrect

3

Episode 3

Chart Selection, Skills: Pattern Recognition + Decision Making, Correct

4

Episode 4

Multiple Choice, Skill: Decision Making, Correct

5

Episode 5

Chart Selection, Skills: Risk Management + Decision Making, Incorrect

Results Analysis

Performance by Question Type

Question TypeAttemptedCorrectPercentage
Multiple Choice22100%
True/False100%
Chart Selection2150%

True/False shows a "Low sample size" warning due to only 1 question attempted

Performance by Skill

SkillTagged QuestionsCorrectPercentage
Pattern Recognition22100%
Risk Management4125%
Decision Making3266.7%

This example demonstrates how the analytics provide clear insights into both question-type performance and skill mastery, with appropriate warnings for small sample sizes.