Institute for Telecommunication Sciences / Research / 5G / IORS / 2025.05 IORS

IORS Virtual Meeting, May 2025

Experiences in lab peer review and procedure harmonization for inter-lab and tool-independent reproducibility in O-RAN testing

This virtual session explored how and when to use peer review to ensure high-quality test results and build trust in the Open RAN testing certification and badging process.

Logistics

  • Moderators: Led by i14y Lab and VIAVI / VALOR, represented by:
    • Andreas Gladisch (Deutsche Telekom and i14y Lab)
    • Carsten Rossenhövel (European Advanced Networking Test Center (EANTC) and i14y Lab)
    • Ian C. Wong (VIVAI Solutions and VALOR)
  • Virtual Meeting on May 19, 2025
  • Video recording available temporarily to registered attendees only

Agenda

  • Introduction NTIA / IORS (NTIA)
  • Goal of repeatable testing (i14y Lab | DT)
  • Results VALOR (VALOR)
  • Results i14y (i14y Lab)
  • Learnings and recommendation
  • Discussion

Event Summary

The IORS virtual session on May 19, 2025, was led by representatives from the Germany i14y Open RAN Lab and the U.S. VIAVI Automated Lab-as-a-Service for Open RAN (VALOR). Over 100 participants attended the virtual session. Prior to the session, i14y and VALOR had collaboratively selected, executed, and compared identical test cases in each of their labs. The goal of the collaboration was to assess the state of test plans and methodologies to ensure repeatable, consistent testing so that operators can confidently deploy Open RAN networks.  

During the session, Ian Wong (VIAVI/VALOR), Andreas Gladisch (Deutsche Telekom), and Carsten Rossenhövel (EANTC/i14y Lab) provided a comparative analysis of test results, outcomes, and recommendations.

The i14y and VALOR labs tested six O-RAN ALLIANCE Test and Integration Focus Group (TIFG) end-to-end (E2E) Test Specification version 6.00 functional and performance test cases, such as user equipment (UE) registration, UE deregistration, and downlink/uplink peak throughput. The labs tested a commonly agreed upon combination of Open Radio Unit (O-RU)/Open Distributed Unit (O-DU)/Open Central Unit (O-CU) vendors, aligning their test beds but intentionally not sharing low-level configurations. The results—such as registration/deregistration and throughput—were recorded and compared using the coefficient variation method.

The lab-to-lab comparisons showed variations within an acceptable range, but some required closer examination. Analyses of the results led to the following recommendations:

  •  It is critical to specify all components of the test setup. For example, the 5G Core can impact the system under test (SUT) performance and results.
  • Specific equipment configuration parameters must be specified. For example, to compare system performance, the number of UEs must be identical.
  • Test plans must specify the same technology. Variations in technology can produce very different results.

The following recommendations enable faster iteration and deployment of tested and validated 5G Open RAN components in real-world scenarios:

  • Establish repeatable and consistent testing methodologies through standardized, reproducible test procedures. 
  • Benchmark performance across environments by comparing results from independent lab setups.
  • Create a knowledge-sharing framework to share insights and benefit stakeholders.
  • Support global standardization with O-RAN Alliance, TIP, and 3GPP to drive repeatable and consistent testing.

The IORS virtual session included a lively discussion with attendees, with many detailed technical questions. The number of attendees and their involvement showed the relevance of the topic. That was also reflected in the interest to continue this work expressed by i14y Lab, VALOR, and representatives of other labs in the audience.