fenrir/tests/TESTING_GUIDE.md

Fenrir Testing Guide

Complete guide to running and writing tests for the Fenrir screen reader.

Quick Start

1. Install Test Dependencies

# Install test requirements
pip install -r tests/requirements.txt

# Or install individually
pip install pytest pytest-cov pytest-mock pytest-timeout

2. Run Tests

# Run all tests (unit + integration)
pytest tests/

# Run only unit tests (fastest)
pytest tests/unit/ -v

# Run only integration tests
pytest tests/integration/ -v

# Run with coverage report
pytest tests/ --cov=src/fenrirscreenreader --cov-report=html
# Then open htmlcov/index.html in a browser

# Run specific test file
pytest tests/unit/test_settings_validation.py -v

# Run specific test class
pytest tests/unit/test_settings_validation.py::TestSpeechSettingsValidation -v

# Run specific test
pytest tests/unit/test_settings_validation.py::TestSpeechSettingsValidation::test_speech_rate_valid_range -v

3. Useful Test Options

# Stop on first failure
pytest tests/ -x

# Show test output (print statements, logging)
pytest tests/ -s

# Run tests in parallel (faster, requires: pip install pytest-xdist)
pytest tests/ -n auto

# Show slowest 10 tests
pytest tests/ --durations=10

# Run only tests matching a keyword
pytest tests/ -k "remote"

# Run tests with specific markers
pytest tests/ -m unit          # Only unit tests
pytest tests/ -m integration   # Only integration tests
pytest tests/ -m "not slow"    # Skip slow tests

Test Structure

tests/
├── README.md                       # Test overview and strategy
├── TESTING_GUIDE.md               # This file - detailed usage guide
├── requirements.txt               # Test dependencies
├── conftest.py                    # Shared fixtures and pytest config
├── unit/                          # Unit tests (fast, isolated)
│   ├── __init__.py
│   ├── test_settings_validation.py  # Settings validation tests
│   ├── test_cursor_utils.py         # Cursor calculation tests
│   └── test_text_utils.py           # Text processing tests
├── integration/                   # Integration tests (require mocking)
│   ├── __init__.py
│   ├── test_remote_control.py       # Remote control functionality
│   ├── test_command_manager.py      # Command loading/execution
│   └── test_event_manager.py        # Event queue processing
└── drivers/                       # Driver tests (require root)
    ├── __init__.py
    ├── test_vcsa_driver.py          # TTY screen reading
    └── test_evdev_driver.py         # Keyboard input capture

Writing New Tests

Unit Test Example

"""tests/unit/test_my_feature.py"""
import pytest

@pytest.mark.unit
def test_speech_rate_calculation():
    """Test that speech rate is calculated correctly."""
    rate = calculate_speech_rate(0.5)
    assert 0.0 <= rate <= 1.0
    assert rate == 0.5

Integration Test Example

"""tests/integration/test_my_integration.py"""
import pytest

@pytest.mark.integration
def test_remote_command_execution(mock_environment):
    """Test remote command execution flow."""
    manager = RemoteManager()
    manager.initialize(mock_environment)

    result = manager.handle_command_execution_with_response("say test")

    assert result["success"] is True
    mock_environment["runtime"]["OutputManager"].speak_text.assert_called_once()

Using Fixtures

Common fixtures are defined in conftest.py:

def test_with_mock_environment(mock_environment):
    """Use the shared mock environment fixture."""
    # mock_environment provides mocked runtime managers
    assert "runtime" in mock_environment
    assert "DebugManager" in mock_environment["runtime"]

def test_with_temp_config(temp_config_file):
    """Use a temporary config file."""
    # temp_config_file is a Path object to a valid test config
    assert temp_config_file.exists()
    content = temp_config_file.read_text()
    assert "[speech]" in content

Test Markers

Tests can be marked to categorize them:

@pytest.mark.unit           # Fast, isolated unit test
@pytest.mark.integration    # Integration test with mocking
@pytest.mark.driver         # Requires root access (skipped by default)
@pytest.mark.slow           # Takes > 1 second
@pytest.mark.remote         # Tests remote control functionality
@pytest.mark.settings       # Tests settings/configuration
@pytest.mark.commands       # Tests command system
@pytest.mark.vmenu          # Tests VMenu system

Run tests by marker:

pytest tests/ -m unit              # Only unit tests
pytest tests/ -m "unit or integration"  # Unit and integration
pytest tests/ -m "not slow"        # Skip slow tests

Code Coverage

View Coverage Report

# Generate HTML coverage report
pytest tests/ --cov=src/fenrirscreenreader --cov-report=html

# Open report in browser
firefox htmlcov/index.html    # Or your preferred browser

# Terminal coverage report
pytest tests/ --cov=src/fenrirscreenreader --cov-report=term-missing

Coverage Goals

• Unit Tests: 80%+ coverage on utility functions and validation logic
• Integration Tests: 60%+ coverage on core managers
• Overall: 70%+ coverage on non-driver code

Driver code is excluded from coverage as it requires hardware interaction.

Testing Best Practices

1. Test One Thing

# Good - tests one specific behavior
def test_speech_rate_rejects_negative():
    with pytest.raises(ValueError):
        validate_rate(-1.0)

# Bad - tests multiple unrelated things
def test_speech_settings():
    validate_rate(0.5)      # Rate validation
    validate_pitch(1.0)     # Pitch validation
    validate_volume(0.8)    # Volume validation

2. Use Descriptive Names

# Good - clear what's being tested
def test_speech_rate_rejects_values_above_three():
    ...

# Bad - unclear purpose
def test_rate():
    ...

3. Arrange-Act-Assert Pattern

def test_remote_command_parsing():
    # Arrange - set up test data
    manager = RemoteManager()
    command = "say Hello World"

    # Act - execute the code being tested
    result = manager.parse_command(command)

    # Assert - verify the result
    assert result["action"] == "say"
    assert result["text"] == "Hello World"

4. Mock External Dependencies

def test_clipboard_export(mock_environment, tmp_path):
    """Test clipboard export without real file operations."""
    # Use mock environment instead of real Fenrir runtime
    manager = RemoteManager()
    manager.initialize(mock_environment)

    # Use temporary path instead of /tmp
    clipboard_path = tmp_path / "clipboard"
    mock_environment["runtime"]["SettingsManager"].get_setting = Mock(
        return_value=str(clipboard_path)
    )

    manager.export_clipboard()

    assert clipboard_path.exists()

5. Test Error Paths

def test_remote_command_handles_invalid_input():
    """Test that invalid commands are handled gracefully."""
    manager = RemoteManager()

    # Test with various invalid inputs
    result1 = manager.handle_command_execution_with_response("")
    result2 = manager.handle_command_execution_with_response("invalid")
    result3 = manager.handle_command_execution_with_response("command unknown")

    # All should return error results, not crash
    assert all(not r["success"] for r in [result1, result2, result3])

Debugging Tests

Run with More Verbosity

# Show test names and outcomes
pytest tests/ -v

# Show test names, outcomes, and print statements
pytest tests/ -v -s

# Show local variables on failure
pytest tests/ --showlocals

# Show full diff on assertion failures
pytest tests/ -vv

Use pytest.set_trace() for Debugging

def test_complex_logic():
    result = complex_function()
    pytest.set_trace()  # Drop into debugger here
    assert result == expected

Run Single Test Repeatedly

# Useful for debugging flaky tests
pytest tests/unit/test_my_test.py::test_specific_test --count=100

Continuous Integration

GitHub Actions Example

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r tests/requirements.txt
      - name: Run tests
        run: pytest tests/ --cov=src/fenrirscreenreader --cov-report=xml
      - name: Upload coverage
        uses: codecov/codecov-action@v3

Common Issues

ImportError: No module named 'fenrirscreenreader'

Solution: Make sure you're running pytest from the project root, or set PYTHONPATH:

export PYTHONPATH="${PYTHONPATH}:$(pwd)/src"
pytest tests/

Tests hang or timeout

Solution: Use the timeout decorator or pytest-timeout:

pytest tests/ --timeout=30  # Global 30s timeout

Or mark specific tests:

@pytest.mark.timeout(5)
def test_that_might_hang():
    ...

Mocks not working as expected

Solution: Check that you're patching the right location:

# Good - patch where it's used
@patch('fenrirscreenreader.core.remoteManager.OutputManager')

# Bad - patch where it's defined
@patch('fenrirscreenreader.core.outputManager.OutputManager')

Advanced Topics

Parametrized Tests

Test multiple inputs with one test:

@pytest.mark.parametrize("rate,expected", [
    (0.0, True),
    (1.5, True),
    (3.0, True),
    (-1.0, False),
    (10.0, False),
])
def test_rate_validation(rate, expected):
    try:
        validate_rate(rate)
        assert expected is True
    except ValueError:
        assert expected is False

Test Fixtures with Cleanup

@pytest.fixture
def temp_fenrir_instance():
    """Start a test Fenrir instance."""
    fenrir = FenrirTestInstance()
    fenrir.start()

    yield fenrir  # Test runs here

    # Cleanup after test
    fenrir.stop()
    fenrir.cleanup()

Testing Async Code

@pytest.mark.asyncio
async def test_async_speech():
    result = await async_speak("test")
    assert result.success

Getting Help

• Pytest Documentation: https://docs.pytest.org/
• Fenrir Issues: https://github.com/chrys87/fenrir/issues
• Test Coverage: Run with --cov-report=html and inspect htmlcov/index.html

Contributing Tests

When contributing tests:

1. Follow naming conventions: test_*.py for files, test_* for functions
2. Add docstrings: Explain what each test verifies
3. Use appropriate markers: @pytest.mark.unit, @pytest.mark.integration, etc.
4. Keep tests fast: Unit tests should complete in <100ms
5. Test edge cases: Empty strings, None, negative numbers, etc.
6. Update this guide: If you add new test patterns or fixtures

Happy testing! 🧪