3. Standards: Levels of Test for Vendor Layer

Level of Test for the Vendor Layer

This document defines the testing requirements for the Vendor Layer, categorized into levels to facilitate phased delivery and vendor participation. These levels collectively form the Vendor Testing Suite (VTS).

Level 1: Component Function Testing (Isolated Component Environment, Pre-Commit) – Vendor Testing

• Purpose: API Function testing of individual components + requirements documentation.
• Scope: Thorough testing of each function and parameter.
• Execution: Performed by component engineers during initial pre-commit interface development or when modifying existing components, before commit.

Level 2: Component Unit Testing (No upper stack layers, Pre-Commit) - Vendor Test

• Purpose: Focused testing of individual modules in a component, aligned with requirements documentation. Scope: Independent component testing.
• Scope: Verifies the functionality of the component.
• Execution: Performed by component engineers pre-commit during initial development or when implementing changes.

Level 3: – Component External Stimulus Testing (No upper stack layers, Pre-Commit) - Vendor Test

• Purpose: Pre-commit testing using external stimuli to validate component responses and adherence to requirements.
• Scope:
  • May trigger both Level 1 (Component Function) and Level 2 (Component Unit) tests for deeper verification.
  • Includes specific tests designed to interact with external devices and evaluate component behaviour.
  • Infrastructure: May require specialized setup, either on engineers' desks or in dedicated racks, and runs on vendor-provided equipment.

Level 4:

Level 4 encompasses a comprehensive range of system-level tests focused on validating the performance, stability, and integration capabilities of the vendor layer within the ecosystem.

L4 - System Performance Profiling

• Purpose: To monitor and analyse system resource usage during key operations.
• Scope:
  • CPU Load: Measure CPU utilization during activities like video playback, app launch, and channel switching. Continuous monitoring via a C application with logging capabilities.
  • Memory Usage: Track memory consumption patterns to identify potential leaks or areas for optimization.
  • Stress Testing: Push the device to its limits to identify bottlenecks and stability issues.
  • Power Consumption: Testing: Measure power usage under different conditions.
  • Tools/Methods: Utilize profiling tools like perf, valgrind, or other suitable frameworks for performance data collection.
  • Metrics and Reporting: Collect and analyse performance metrics (CPU usage, memory allocation, frame rates) and generate reports to pinpoint bottlenecks and areas for improvement.

L4 - Ad hoc Analysis

• Purpose: To conduct deep-dive investigations into specific issues or performance bottlenecks.
• Scope:
  • Triggered by: Field issues, complex bugs, performance bottlenecks, or customer escalations.
  • Methodology: Employ in-depth analysis using debugging tools (e.g., gdb), specialized logs, and code inspection.
  • Collaboration: Work closely with development teams to identify root causes and implement solutions.

L4 - System Interface Testing

• Purpose: To validate interactions with external interfaces and devices.
• Scope:
  • Key Interfaces:
    • Bluetooth (Bluez): Verify connectivity, pairing, and data transfer.
    • WiFi (WPA-supplicant): Test connection stability, roaming, and security.
    • OpenGLES, OpenGL, Graphics, Vulcan: Validate graphics rendering, performance, and compliance. (Video Only)
    • Kernel Headers, SysFS: Verify kernel interface compatibility and system information access.
  • Testing Approach: Utilize unit tests, integration tests, and functional testing to ensure proper communication and data exchange.

L4 - QA / Layer Smoke Testing

• Purpose: To guard against regressions and ensure basic functionality before release.
• Scope:
  • Release Checks: Prevent new L4 releases from breaking functionality in higher layers (MW and APP).
  • Smoke Tests: Execute essential tests of the upper layer to quickly assess core functions.
  • Application-Level Checks: Test video playback (if applicable) (e.g., Netflix) and full-stack scenarios to meet release criteria.
  • Test Specification: Reference the existing specification document for detailed test cases and expected results.

L4 - Deep Dive Testing

• Purpose: To perform comprehensive functional and performance validation.
• **Scope: (Video) **
  • Audio Quality: Test audio playback across various output interfaces, measuring fidelity, latency, and synchronization.
  • Video Quality: Evaluate video playback across supported codecs, assessing smoothness, artefacts, and colour accuracy.
  • A/V Synchronization: Verify audio-video sync across different content and codecs.
  • Graphics Performance: Conduct end-to-end testing, measuring frame rates, rendering quality, and resource usage.
  • Mixing A/V + Display Scaling: Test scenarios like picture-in-picture to ensure correct scaling and display of multiple video planes.
  • Trick Modes: Evaluate seeking accuracy and responsiveness, and test video plane scaling during trick modes.