Tests added, see the documentation in the tests directory for details. Improved the socket code.
This commit is contained in:
Storm Dragon
430
tests/TESTING_GUIDE.md
Normal file
430
tests/TESTING_GUIDE.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# Fenrir Testing Guide
|
||||
|
||||
Complete guide to running and writing tests for the Fenrir screen reader.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Install Test Dependencies
|
||||
|
||||
```bash
|
||||
# Install test requirements
|
||||
pip install -r tests/requirements.txt
|
||||
|
||||
# Or install individually
|
||||
pip install pytest pytest-cov pytest-mock pytest-timeout
|
||||
```
|
||||
|
||||
### 2. Run Tests
|
||||
|
||||
```bash
|
||||
# Run all tests (unit + integration)
|
||||
pytest tests/
|
||||
|
||||
# Run only unit tests (fastest)
|
||||
pytest tests/unit/ -v
|
||||
|
||||
# Run only integration tests
|
||||
pytest tests/integration/ -v
|
||||
|
||||
# Run with coverage report
|
||||
pytest tests/ --cov=src/fenrirscreenreader --cov-report=html
|
||||
# Then open htmlcov/index.html in a browser
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/unit/test_settings_validation.py -v
|
||||
|
||||
# Run specific test class
|
||||
pytest tests/unit/test_settings_validation.py::TestSpeechSettingsValidation -v
|
||||
|
||||
# Run specific test
|
||||
pytest tests/unit/test_settings_validation.py::TestSpeechSettingsValidation::test_speech_rate_valid_range -v
|
||||
```
|
||||
|
||||
### 3. Useful Test Options
|
||||
|
||||
```bash
|
||||
# Stop on first failure
|
||||
pytest tests/ -x
|
||||
|
||||
# Show test output (print statements, logging)
|
||||
pytest tests/ -s
|
||||
|
||||
# Run tests in parallel (faster, requires: pip install pytest-xdist)
|
||||
pytest tests/ -n auto
|
||||
|
||||
# Show slowest 10 tests
|
||||
pytest tests/ --durations=10
|
||||
|
||||
# Run only tests matching a keyword
|
||||
pytest tests/ -k "remote"
|
||||
|
||||
# Run tests with specific markers
|
||||
pytest tests/ -m unit # Only unit tests
|
||||
pytest tests/ -m integration # Only integration tests
|
||||
pytest tests/ -m "not slow" # Skip slow tests
|
||||
```
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── README.md # Test overview and strategy
|
||||
├── TESTING_GUIDE.md # This file - detailed usage guide
|
||||
├── requirements.txt # Test dependencies
|
||||
├── conftest.py # Shared fixtures and pytest config
|
||||
├── unit/ # Unit tests (fast, isolated)
|
||||
│ ├── __init__.py
|
||||
│ ├── test_settings_validation.py # Settings validation tests
|
||||
│ ├── test_cursor_utils.py # Cursor calculation tests
|
||||
│ └── test_text_utils.py # Text processing tests
|
||||
├── integration/ # Integration tests (require mocking)
|
||||
│ ├── __init__.py
|
||||
│ ├── test_remote_control.py # Remote control functionality
|
||||
│ ├── test_command_manager.py # Command loading/execution
|
||||
│ └── test_event_manager.py # Event queue processing
|
||||
└── drivers/ # Driver tests (require root)
|
||||
├── __init__.py
|
||||
├── test_vcsa_driver.py # TTY screen reading
|
||||
└── test_evdev_driver.py # Keyboard input capture
|
||||
```
|
||||
|
||||
## Writing New Tests
|
||||
|
||||
### Unit Test Example
|
||||
|
||||
```python
|
||||
"""tests/unit/test_my_feature.py"""
|
||||
import pytest
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speech_rate_calculation():
|
||||
"""Test that speech rate is calculated correctly."""
|
||||
rate = calculate_speech_rate(0.5)
|
||||
assert 0.0 <= rate <= 1.0
|
||||
assert rate == 0.5
|
||||
```
|
||||
|
||||
### Integration Test Example
|
||||
|
||||
```python
|
||||
"""tests/integration/test_my_integration.py"""
|
||||
import pytest
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_remote_command_execution(mock_environment):
|
||||
"""Test remote command execution flow."""
|
||||
manager = RemoteManager()
|
||||
manager.initialize(mock_environment)
|
||||
|
||||
result = manager.handle_command_execution_with_response("say test")
|
||||
|
||||
assert result["success"] is True
|
||||
mock_environment["runtime"]["OutputManager"].speak_text.assert_called_once()
|
||||
```
|
||||
|
||||
### Using Fixtures
|
||||
|
||||
Common fixtures are defined in `conftest.py`:
|
||||
|
||||
```python
|
||||
def test_with_mock_environment(mock_environment):
|
||||
"""Use the shared mock environment fixture."""
|
||||
# mock_environment provides mocked runtime managers
|
||||
assert "runtime" in mock_environment
|
||||
assert "DebugManager" in mock_environment["runtime"]
|
||||
|
||||
def test_with_temp_config(temp_config_file):
|
||||
"""Use a temporary config file."""
|
||||
# temp_config_file is a Path object to a valid test config
|
||||
assert temp_config_file.exists()
|
||||
content = temp_config_file.read_text()
|
||||
assert "[speech]" in content
|
||||
```
|
||||
|
||||
## Test Markers
|
||||
|
||||
Tests can be marked to categorize them:
|
||||
|
||||
```python
|
||||
@pytest.mark.unit # Fast, isolated unit test
|
||||
@pytest.mark.integration # Integration test with mocking
|
||||
@pytest.mark.driver # Requires root access (skipped by default)
|
||||
@pytest.mark.slow # Takes > 1 second
|
||||
@pytest.mark.remote # Tests remote control functionality
|
||||
@pytest.mark.settings # Tests settings/configuration
|
||||
@pytest.mark.commands # Tests command system
|
||||
@pytest.mark.vmenu # Tests VMenu system
|
||||
```
|
||||
|
||||
Run tests by marker:
|
||||
```bash
|
||||
pytest tests/ -m unit # Only unit tests
|
||||
pytest tests/ -m "unit or integration" # Unit and integration
|
||||
pytest tests/ -m "not slow" # Skip slow tests
|
||||
```
|
||||
|
||||
## Code Coverage
|
||||
|
||||
### View Coverage Report
|
||||
|
||||
```bash
|
||||
# Generate HTML coverage report
|
||||
pytest tests/ --cov=src/fenrirscreenreader --cov-report=html
|
||||
|
||||
# Open report in browser
|
||||
firefox htmlcov/index.html # Or your preferred browser
|
||||
|
||||
# Terminal coverage report
|
||||
pytest tests/ --cov=src/fenrirscreenreader --cov-report=term-missing
|
||||
```
|
||||
|
||||
### Coverage Goals
|
||||
|
||||
- **Unit Tests**: 80%+ coverage on utility functions and validation logic
|
||||
- **Integration Tests**: 60%+ coverage on core managers
|
||||
- **Overall**: 70%+ coverage on non-driver code
|
||||
|
||||
Driver code is excluded from coverage as it requires hardware interaction.
|
||||
|
||||
## Testing Best Practices
|
||||
|
||||
### 1. Test One Thing
|
||||
|
||||
```python
|
||||
# Good - tests one specific behavior
|
||||
def test_speech_rate_rejects_negative():
|
||||
with pytest.raises(ValueError):
|
||||
validate_rate(-1.0)
|
||||
|
||||
# Bad - tests multiple unrelated things
|
||||
def test_speech_settings():
|
||||
validate_rate(0.5) # Rate validation
|
||||
validate_pitch(1.0) # Pitch validation
|
||||
validate_volume(0.8) # Volume validation
|
||||
```
|
||||
|
||||
### 2. Use Descriptive Names
|
||||
|
||||
```python
|
||||
# Good - clear what's being tested
|
||||
def test_speech_rate_rejects_values_above_three():
|
||||
...
|
||||
|
||||
# Bad - unclear purpose
|
||||
def test_rate():
|
||||
...
|
||||
```
|
||||
|
||||
### 3. Arrange-Act-Assert Pattern
|
||||
|
||||
```python
|
||||
def test_remote_command_parsing():
|
||||
# Arrange - set up test data
|
||||
manager = RemoteManager()
|
||||
command = "say Hello World"
|
||||
|
||||
# Act - execute the code being tested
|
||||
result = manager.parse_command(command)
|
||||
|
||||
# Assert - verify the result
|
||||
assert result["action"] == "say"
|
||||
assert result["text"] == "Hello World"
|
||||
```
|
||||
|
||||
### 4. Mock External Dependencies
|
||||
|
||||
```python
|
||||
def test_clipboard_export(mock_environment, tmp_path):
|
||||
"""Test clipboard export without real file operations."""
|
||||
# Use mock environment instead of real Fenrir runtime
|
||||
manager = RemoteManager()
|
||||
manager.initialize(mock_environment)
|
||||
|
||||
# Use temporary path instead of /tmp
|
||||
clipboard_path = tmp_path / "clipboard"
|
||||
mock_environment["runtime"]["SettingsManager"].get_setting = Mock(
|
||||
return_value=str(clipboard_path)
|
||||
)
|
||||
|
||||
manager.export_clipboard()
|
||||
|
||||
assert clipboard_path.exists()
|
||||
```
|
||||
|
||||
### 5. Test Error Paths
|
||||
|
||||
```python
|
||||
def test_remote_command_handles_invalid_input():
|
||||
"""Test that invalid commands are handled gracefully."""
|
||||
manager = RemoteManager()
|
||||
|
||||
# Test with various invalid inputs
|
||||
result1 = manager.handle_command_execution_with_response("")
|
||||
result2 = manager.handle_command_execution_with_response("invalid")
|
||||
result3 = manager.handle_command_execution_with_response("command unknown")
|
||||
|
||||
# All should return error results, not crash
|
||||
assert all(not r["success"] for r in [result1, result2, result3])
|
||||
```
|
||||
|
||||
## Debugging Tests
|
||||
|
||||
### Run with More Verbosity
|
||||
|
||||
```bash
|
||||
# Show test names and outcomes
|
||||
pytest tests/ -v
|
||||
|
||||
# Show test names, outcomes, and print statements
|
||||
pytest tests/ -v -s
|
||||
|
||||
# Show local variables on failure
|
||||
pytest tests/ --showlocals
|
||||
|
||||
# Show full diff on assertion failures
|
||||
pytest tests/ -vv
|
||||
```
|
||||
|
||||
### Use pytest.set_trace() for Debugging
|
||||
|
||||
```python
|
||||
def test_complex_logic():
|
||||
result = complex_function()
|
||||
pytest.set_trace() # Drop into debugger here
|
||||
assert result == expected
|
||||
```
|
||||
|
||||
### Run Single Test Repeatedly
|
||||
|
||||
```bash
|
||||
# Useful for debugging flaky tests
|
||||
pytest tests/unit/test_my_test.py::test_specific_test --count=100
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -r requirements.txt
|
||||
pip install -r tests/requirements.txt
|
||||
- name: Run tests
|
||||
run: pytest tests/ --cov=src/fenrirscreenreader --cov-report=xml
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### ImportError: No module named 'fenrirscreenreader'
|
||||
|
||||
**Solution**: Make sure you're running pytest from the project root, or set PYTHONPATH:
|
||||
```bash
|
||||
export PYTHONPATH="${PYTHONPATH}:$(pwd)/src"
|
||||
pytest tests/
|
||||
```
|
||||
|
||||
### Tests hang or timeout
|
||||
|
||||
**Solution**: Use the timeout decorator or pytest-timeout:
|
||||
```bash
|
||||
pytest tests/ --timeout=30 # Global 30s timeout
|
||||
```
|
||||
|
||||
Or mark specific tests:
|
||||
```python
|
||||
@pytest.mark.timeout(5)
|
||||
def test_that_might_hang():
|
||||
...
|
||||
```
|
||||
|
||||
### Mocks not working as expected
|
||||
|
||||
**Solution**: Check that you're patching the right location:
|
||||
```python
|
||||
# Good - patch where it's used
|
||||
@patch('fenrirscreenreader.core.remoteManager.OutputManager')
|
||||
|
||||
# Bad - patch where it's defined
|
||||
@patch('fenrirscreenreader.core.outputManager.OutputManager')
|
||||
```
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
### Parametrized Tests
|
||||
|
||||
Test multiple inputs with one test:
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize("rate,expected", [
|
||||
(0.0, True),
|
||||
(1.5, True),
|
||||
(3.0, True),
|
||||
(-1.0, False),
|
||||
(10.0, False),
|
||||
])
|
||||
def test_rate_validation(rate, expected):
|
||||
try:
|
||||
validate_rate(rate)
|
||||
assert expected is True
|
||||
except ValueError:
|
||||
assert expected is False
|
||||
```
|
||||
|
||||
### Test Fixtures with Cleanup
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def temp_fenrir_instance():
|
||||
"""Start a test Fenrir instance."""
|
||||
fenrir = FenrirTestInstance()
|
||||
fenrir.start()
|
||||
|
||||
yield fenrir # Test runs here
|
||||
|
||||
# Cleanup after test
|
||||
fenrir.stop()
|
||||
fenrir.cleanup()
|
||||
```
|
||||
|
||||
### Testing Async Code
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_speech():
|
||||
result = await async_speak("test")
|
||||
assert result.success
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
- **Pytest Documentation**: https://docs.pytest.org/
|
||||
- **Fenrir Issues**: https://github.com/chrys87/fenrir/issues
|
||||
- **Test Coverage**: Run with `--cov-report=html` and inspect `htmlcov/index.html`
|
||||
|
||||
## Contributing Tests
|
||||
|
||||
When contributing tests:
|
||||
|
||||
1. **Follow naming conventions**: `test_*.py` for files, `test_*` for functions
|
||||
2. **Add docstrings**: Explain what each test verifies
|
||||
3. **Use appropriate markers**: `@pytest.mark.unit`, `@pytest.mark.integration`, etc.
|
||||
4. **Keep tests fast**: Unit tests should complete in <100ms
|
||||
5. **Test edge cases**: Empty strings, None, negative numbers, etc.
|
||||
6. **Update this guide**: If you add new test patterns or fixtures
|
||||
|
||||
Happy testing! 🧪
|
||||
Reference in New Issue
Block a user