PR: Add Spec Tests Workflow - Comprehensive Model Readiness Testing

Overview

This PR introduces the Spec Tests workflow, a comprehensive testing framework designed to validate model readiness and performance across different parameter configurations. The spec_tests workflow provides both targeted single-parameter testing and exhaustive multi-parameter matrix testing to ensure models meet performance specifications before deployment.

What is the Spec Tests Workflow?

The Spec Tests workflow is a model validation system that:

• Tests model performance across various input/output token configurations
• Validates concurrency handling at different load levels
• Measures performance metrics like TTFT (Time to First Token) and throughput
• Compares against performance targets defined in model configurations
• Generates comprehensive test reports for model readiness assessment

Key Features

🎯 Two Testing Modes:

• Single Mode: Test specific parameter combinations with explicit control
• Multiple Mode: Comprehensive cross-product matrix testing across all valid parameter combinations

🔧 Flexible Parameter Control:

• Input/output token lengths
• Concurrency levels (batch sizes)
• Number of prompts (user simulation)
• Context length limits
• Custom parameter combinations via workflow-args

📊 Performance Validation:

• Automatic extraction of performance targets from model configs
• Comparison against theoretical and reference benchmarks
• Support for different target levels (theoretical, reference, etc.)

⏱️ Endurance Testing:

• 24-hour continuous testing mode for stability validation
• Stress testing with high concurrency and prompt volumes

How It Works

Architecture

spec_tests/
├── run_spec_tests.py           # Main entry point
├── spec_tests_config.py        # Parameter space generation
├── spec_tests_benchmarking_script.py  # Benchmarking engine
└── run/
    ├── spec_tests.py           # Core workflow orchestrator
    ├── spec_test_tasks.py      # Parameter generation logic
    ├── spec_test_prompt.py     # Prompt generation
    ├── spec_test_run.py        # Test execution
    └── spec_tests_env_vars.py  # Environment configuration

Parameter Generation Logic

Single Mode:

• Uses explicit parameters from command line arguments
• Calculates input/output sizes based on max_context_length
• Defaults: max_context_length=8192, max_concurrent=1, num_prompts=1

Multiple Mode:

• Generates cross-product of all valid parameter combinations
• Uses model-specific constraints (max_context, max_concurrency)
• Includes validated combinations from performance reference data
• Context sizes: max, ~50%, ~10% of model's max_context
• Output sizes: 128, 64 tokens (fixed)
• Concurrency: 1, 2, ~50% of max, validated values (max concurrency temporarily disabled)
• Prompt patterns: single (1), load-matched (=concurrency) (5x concurrency temporarily disabled)

Usage Examples

Basic Usage

Single Parameter Test

Test a specific configuration with explicit parameters:

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --run-mode single --docker-server

Custom Context Length

Test with a specific context length:

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --run-mode single --max-context-length 4096 --docker-server

Load Testing with Concurrency

Test with specific concurrency and prompt count:

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --run-mode single --max-context-length 2048 --workflow-args "max_concurrent=16 num_prompts=64" --docker-server

Comprehensive Matrix Testing

Run all valid parameter combinations:

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --run-mode multiple --docker-server

Advanced Usage

High-Load Stress Testing

python run.py --model "Qwen2.5-72B-Instruct" --workflow spec_tests --device t3k --run-mode single --max-context-length 8192 --workflow-args "max_concurrent=32 num_prompts=160" --docker-server

Custom Input/Output Sizes

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --run-mode single --workflow-args "input_size=1024 output_size=512 max_concurrent=8 num_prompts=24" --docker-server

Endurance Testing (24 hours)

python run.py --model "Llama-3.1-8B-Instruct" --workflow spec_tests --device n300 --endurance-mode --docker-server

Model-Specific Examples

Large Model Testing (70B+)

# Qwen2.5-72B on T3K with reduced context for faster testing
python run.py --model "Qwen2.5-72B-Instruct" --workflow spec_tests --device t3k --run-mode single --max-context-length 2048 --workflow-args "max_concurrent=16 num_prompts=32" --docker-server

# Full parameter matrix for comprehensive validation
python run.py --model "Qwen2.5-72B-Instruct" --workflow spec_tests --device t3k --run-mode multiple --docker-server

Parameter Reference

Core Parameters

Workflow Args (via --workflow-args)

Usage Patterns

Single Test: max_concurrent=1 num_prompts=1
Load Test: max_concurrent=16 num_prompts=16
Stress Test: max_concurrent=32 num_prompts=160
Burst Test: max_concurrent=8 num_prompts=100

Output and Reports

Test Execution Output

=== Parameter Space Information ===
Model ID: id_tt-transformers_Llama-3.1-8B-Instruct_n300
Device: n300
Max Context Limit: 128000
Max Concurrency Limit: 32
Max Context Length: 4096
Validated Combinations: 2
Total Spec Test Parameters: 1
Performance Target Levels: ['theoretical']
===================================

Generated Test Combinations (Multiple Mode)

The workflow automatically generates test combinations based on:

• Context Sizes: [128000, 64000, 12800] (max, 50%, 10%)
• Output Sizes: [128, 64] (fixed)
• Input Sizes: Calculated as context_size - output_size
• Concurrency: [1, 2, 16] (baseline, small, mid-scale - max concurrency temporarily disabled)
• Prompt Counts: [1, match_concurrency] (baseline, load - 5x concurrency temporarily disabled)

Performance Metrics

Each test measures:

• TTFT (Time to First Token): Latency to first response token
• Throughput: Tokens per second
• Request Rate: Requests per second
• Error Rates: Failed requests percentage
• Resource Utilization: Memory, compute usage

Integration with Model Configurations

The spec_tests workflow automatically integrates with model configurations to:

1. Extract Performance Targets: Reads theoretical and reference performance targets
2. Validate Parameter Bounds: Ensures tests stay within model constraints
3. Use Validated Combinations: Incorporates pre-tested parameter sets
4. Apply Device Constraints: Respects device-specific limitations

Example Model Config Integration

# From model_performance_reference.json
"Llama-3.1-8B": {
    "n300": [
        {
            "isl": 128,
            "osl": 128, 
            "max_concurrency": 1,
            "num_prompts": 8,
            "targets": {
                "theoretical": {
                    "ttft_ms": 16,
                    "tput_user": 66
                }
            }
        }
    ]
}

Benefits

For Development Teams

• Early Detection: Catch performance regressions before deployment
• Parameter Optimization: Find optimal configurations for different use cases
• Load Planning: Understand capacity limits and scaling characteristics

For QA/Testing

• Comprehensive Coverage: Test all valid parameter combinations automatically
• Reproducible Results: Consistent test parameters and environments
• Performance Baselines: Establish and track performance targets

For DevOps/Infrastructure

• Capacity Planning: Understand resource requirements at scale
• SLA Validation: Verify models meet performance commitments
• Deployment Readiness: Comprehensive pre-deployment validation

Technical Implementation

Key Components

1. SpecTestParamSpace: Generates parameter combinations based on model constraints
2. SpecTestTask: Orchestrates parameter generation for single/multiple modes
3. SpecTestPrompt: Creates test prompts with specified token characteristics
4. SpecTestRun: Executes benchmarks and collects performance metrics
5. SpecTestsEnvVars: Manages environment configuration and secrets

Workflow Integration

• Integrates with existing run.py CLI interface
• Uses standard workflow patterns (docker-server, logging, etc.)
• Supports all workflow arguments and overrides
• Compatible with existing model configuration system

This spec_tests workflow provides a robust foundation for model validation and performance testing, ensuring models meet quality and performance standards before deployment to production environments.