Testing Guide¶
Overview¶
Comprehensive testing guide for the Open Resource Broker, including unit tests, integration tests, and Docker containerization testing.
Test Categories¶
Unit Tests¶
Fast, isolated tests for individual components:
Integration Tests¶
Test component interactions:
Docker Tests¶
Test Docker containerization:
End-to-End Tests¶
Full workflow testing:
Test Structure¶
tests/
+--- unit/ # Unit tests
| +--- domain/ # Domain layer tests
| +--- application/ # Application layer tests
| +--- infrastructure/ # Infrastructure tests
+--- integration/ # Integration tests
| +--- api/ # API integration tests
| +--- aws/ # AWS integration tests
+--- docker/ # Docker tests
| +--- test_dockerfile.py
| +--- test_container_integration.py
| +--- test_docker_compose.py
+--- e2e/ # End-to-end tests
+--- performance/ # Performance tests
+--- security/ # Security tests
Running Tests¶
All Tests¶
# Run all tests
pytest
# Run with coverage
pytest --cov=src --cov-report=html --cov-report=term-missing
Specific Test Categories¶
# Unit tests only
pytest -m unit
# Integration tests
pytest -m integration
# Docker tests
pytest -m docker
# AWS tests (requires AWS credentials)
pytest -m aws
# Security tests
pytest -m security
# Performance tests
pytest -m performance
Test Configuration¶
The project uses pytest.ini for configuration:
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
markers =
unit: Unit tests - fast, isolated tests
integration: Integration tests - test component interactions
docker: Docker containerization tests
aws: Tests that interact with AWS services (mocked)
security: Security-related tests
performance: Performance and load tests
Docker Testing¶
Dockerfile Testing¶
Tests Docker build process and container structure:
def test_dockerfile_structure(self, dockerfile_path):
"""Test Dockerfile structure and best practices."""
content = dockerfile_path.read_text()
# Check for multi-stage build
assert "FROM python:3.11-slim as builder" in content
assert "FROM python:3.11-slim as production" in content
# Check for security best practices
assert "RUN groupadd -r orb && useradd -r -g orb" in content
assert "USER orb" in content
Container Integration Testing¶
Tests container functionality and configuration:
def test_container_environment_variables(self, built_image):
"""Test container responds to environment variables."""
result = subprocess.run([
"docker", "run", "--rm",
"-e", "HF_SERVER_ENABLED=true",
"-e", "HF_AUTH_ENABLED=false",
built_image, "version"
], capture_output=True, text=True, timeout=30)
assert result.returncode == 0
Docker Compose Testing¶
Tests Docker Compose configurations:
def test_docker_compose_dev_service_configuration(self, project_root):
"""Test development Docker Compose service configuration."""
with open(project_root / "docker-compose.yml") as f:
compose_config = yaml.safe_load(f)
assert "orb-api" in compose_config["services"]
assert "build" in compose_config["services"]["orb-api"]
Test Utilities¶
Fixtures¶
Common test fixtures in tests/conftest.py:
@pytest.fixture
def mock_aws_credentials():
"""Mock AWS credentials for testing."""
with mock.patch.dict(os.environ, {
'AWS_ACCESS_KEY_ID': 'testing',
'AWS_SECRET_ACCESS_KEY': 'testing',
'AWS_SECURITY_TOKEN': 'testing',
'AWS_SESSION_TOKEN': 'testing',
}):
yield
@pytest.fixture
def mock_ec2():
"""Mock EC2 service for testing."""
with mock_aws():
yield boto3.client('ec2', region_name='us-east-1')
Test Data Builders¶
Factory pattern for test data:
class TemplateBuilder:
def __init__(self):
self.template_data = {
"templateId": "test-template",
"templateName": "Test Template",
"provider_api": "aws",
"imageId": "ami-12345678",
"instanceType": "t3.micro"
}
def with_id(self, template_id: str):
self.template_data["templateId"] = template_id
return self
def build(self) -> Template:
return Template(**self.template_data)
Continuous Integration¶
GitHub Actions¶
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.11, 3.12, 3.13]
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run tests
run: pytest --cov=src --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
Docker Testing in CI¶
docker-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Run Docker tests
run: |
pip install pytest
pytest -m docker -v
Test Best Practices¶
Writing Good Tests¶
- Arrange, Act, Assert: Structure tests clearly
- Descriptive Names: Test names should describe what they test
- Single Responsibility: One assertion per test when possible
- Independent Tests: Tests should not depend on each other
- Mock External Dependencies: Use mocks for AWS, databases, etc.
Example Test¶
def test_request_machines_creates_valid_request():
# Arrange
template = TemplateBuilder().with_id("test-template").build()
machine_count = 2
# Act
request = create_machine_request(template, machine_count)
# Assert
assert request.template_id == "test-template"
assert request.machine_count == 2
assert request.status == RequestStatus.PENDING
Performance Testing¶
Load Testing¶
@pytest.mark.performance
def test_api_performance():
"""Test API performance under load."""
import time
import concurrent.futures
def make_request():
response = client.get("/health")
return response.status_code == 200
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(make_request) for _ in range(100)]
results = [future.result() for future in futures]
total_time = time.time() - start_time
assert all(results)
assert total_time < 10 # 100 requests in under 10 seconds
Security Testing¶
Authentication Testing¶
@pytest.mark.security
def test_authentication_required():
"""Test that protected endpoints require authentication."""
response = client.get("/api/v1/templates")
assert response.status_code == 401
def test_invalid_token_rejected():
"""Test that invalid tokens are rejected."""
headers = {"Authorization": "Bearer invalid-token"}
response = client.get("/api/v1/templates", headers=headers)
assert response.status_code == 401
For more testing examples, see the test files in the tests/ directory.