Graph Neural Networking Challenge 2023
Creating a Network Digital Twin with Real Network Data
Evaluation: Closed
Registration: Closed
ITU AI/ML in 5G Challenge
ITU invites you to participate in the ITU Artificial Intelligence/Machine Learning in 5G Challenge, a competition which is scheduled to run from now until the end of the year. Participation in the Challenge is free of charge and open to all interested parties from countries that are members of ITU.
Detailed information about this competition can be found on the Challenge website.
BNN-UPC is glad to announce the “Graph Neural Networking challenge 2023”, which is organized as part of the “ITU AI/ML in 5G Challenge.”
Final ranking
| RANK | TEAM | MAPE (%) |
| 1st | m0b1us | 20.001436 |
| 2nd | BUPT_CMCC | 22.680460 |
| 3rd | LKN | 35.389392 |
| 4th | Yanbiyu | 43.393626 |
| 5th | CAU-Networks | 43.439188 |
Overview
In recent years, the networking community has produced robust Graph Neural Networks (GNN) that can accurately mimic complex network environments. Modern GNN architectures enable building lightweight and accurate Network Digital Twins that can operate in real-time. However, and as a consequence of the lack of real-world data, current ML-based models have been mainly developed and trained using simulated data. This has strongly limited our understanding on how existing models perform in a real network.
This edition of the Graph Neural Networking challenge focuses on developing for the first time a GNN-based Network Digital Twin using a dataset from a real network. This includes modeling the input traffic as well as the relationship between traffic, topology, and configuration (e.g. routing) to produce accurate estimates of the network performance (e.g. delay).
Problem statement
The goal of this challenge is to create a Network Digital Twin based on neural networks which can accurately estimate QoS performance metrics given the network state and the input traffic. More in detail, solutions must predict the resulting per-flow mean delay given: (i) a network topology (L2 and L3), (ii) a set of input flow packet traces, and (iii) a routing configuration. The following figure presents a schematic representation.
Figure 1: Schematic representation of RouteNet.
As a baseline we provide an open-source implementation of RouteNet: RouteNet-Fermi. This model is the outcome of the three previous editions of the challenge and it has been developed and trained based on simulated data. More importantly for this challenge, it does not support arbitrary packet traces as input.
RouteNet-Fermi assumes that the input traffic is generated from a stationary stochastic process and uses the moments of such distributions (e.g. mean, variance, etc) as features describing the traffic. In addition, the simulation environment represents an ideal network scenario and RouteNet-Fermi has not been tested with real networks.
The objective of this challenge is to build a neural network that provides accurate estimates of the mean per-flow delay with a dataset generated from a real network. As a novelty, the dataset contains the input per-flow packet traces. We encourage participants to focus on how to summarize the per-flow packet traces and the non-ideal behavior of real networks. The proposed solution can be based on RouteNet-Fermi, but we also encourage participants to develop their own from-scratch ideas.
Dataset
For this challenge, we provide two datasets generated with a real network testbed. Each dataset (Fig. 2) contains samples generated in several topologies and includes hundreds of network configurations (routing) and traffic flows. In the first dataset traffic is generated using Contstant-Bit Rate (CBR) while in the second traffic is generated using Multi-Burst Rate (MBR). In both cases packet size is randomly selected per flow and constant for all packets within the flow. The range of the packet size is between 100 and 1450 bytes. More information about this can be found later on.
Figure 2: Structure of the Training Dataset
Each sample is labeled with average per-flow performance measurements (mean per-flow and per-packet delay) resulting from running the traffic through the network. In particular, this challenge focuses on the prediction of the mean per-flow delay. This is the average delay of all the packets of each flow at the egress point of the network.
To rank solutions we will post a test dataset. This dataset will contain samples obtained from the same distribution as the training datasets, including a mix of CBR and MBR samples. This dataset will not contain any information related to the mean per-flow delay.
Information about the testbed and dataset can be found here:
Resources
All the main resources can be found in the following GitHub repository:
This repository includes a README file, which includes the instructions on how to download the dataset and use it to train the baseline models. We also included a guide on how you can modify the baseline model’s code in order to create your own submissions.
Below there are other useful resources for the challenge:
Kickoff webinar
Evaluation
The objective is to test the properties of the proposed solutions. To this end, before the end of the challenge, we will provide a test dataset.
This dataset will contain samples with similar distributions to the samples included in the provided dataset. Participants must label this dataset using their neural network models and send the results, the estimated mean per-flow delay for all flows, in CSV format. For the evaluation, we will use the Mean Absolute Percentage Error (MAPE) score computed over the delay predictions produced by candidate solutions:
Solutions with lower MAPE score will be the winners.
During the evaluation, participants will be able to submit automatically their solutions in our evaluation platform and see a provisional ranking with all the teams in real-time.
Note: The test dataset will not include any data related to flow performance measurements (i.e., flow delay, jitter, loss). It means that these features cannot be used as input for the proposed solutions. Users of the dataset API can assume that NO data under the performance_matrix can be used as input for the model, as well as the delay information in pkt_info.
Baseline
As a baseline, we provide RouteNet-Fermi [link], a Graph Neural Network (GNN) architecture recently proposed to estimate end-to-end performance metrics in networks. RouteNet-Fermi is the outcome of three editions of the challenge, and it has an unprecedented ability to make accurate performance predictions even in network scenarios larger than those seen during the training phase. In addition, RouteNet-Fermi supports a wide range of queue scheduling policies and several stochastic traffic models. Notably, it does not support packet traces as input.
We provide an open-source implementation of RouteNet-Fermi used as the baseline, as well as a tutorial on how to execute the code and modify fundamental characteristics of the model [link]. Participants are encouraged to modify the model or design their own neural network architecture from scratch.
You can find at this [link] a deck of slides introducing the challenge and some tips from the organizers to have a good start.
Final submissions and awards
After the score-based evaluation phase (see “Evaluation” section), a ranking of all the teams will be posted on this website.
The top-5 teams must send to the organizers: (i) the code of the neural network solution proposed, (ii) the neural network model already trained, and (iii) a brief document describing the proposed solution and how to reproduce it (1-3 pages). This will be revised to check that the solutions comply with all the rules of the challenge.
Top teams will be invited to present their solutions at the 2nd GNNet workshop, co-located with ACM CoNEXT (December 2023) held in Paris. Please, find more details about this workshop below:
The winning teams of the Graph Neural Networking challenge will receive the following cash prizes:
- 1st prize: 2000 EUR
- Runner-up: 500 EUR
Provided that the Organizing Team validates the solutions and the Judges Panel from the ITU AI/ML in 5G Challenge determines that the solution satisfies the judging criteria.
Also, top-3 teams will advance to the Grand Challenge Finale of the ITU AI/ML in 5G Challenge.
The organizers encourage the publication of papers describing the best solutions. Always respecting the desire of those participants that may want to keep their solutions private.
Rules
IMPORTANT NOTE: It is mandatory filling the registration form to officially participate in this challenge.
Please, note you should register additionally in the portal of the ITU AI/ML in 5G challenge (“Login” and “Create Team”): https://challenge.aiforgood.itu.int/match/matchitem/82
All participants must satisfy the following rules:
To participate in this challenge, the following rules must be satisfied:
- The proposed solution cannot use network simulation tools or solutions derived from them.
- Solutions must be trained only with samples included in the training dataset we provide. Data augmentation techniques are allowed as long as they exclusively use data from the training dataset. However, it is not allowed to use additional data obtained from other sources (e.g., network testbeds or simulators) or public datasets.
- During the evaluation process, each team can submit a maximum of 5 solutions (TXT files) per day. In total, teams can make up to 20 evaluation submissions. In case of receiving more submissions, only the first submissions up to these limits will be considered.
- The challenge is open to all participants except members of the challenge organizing team, members of the challenge advisory board, and/or individuals belonging to or associated with the research group “Barcelona Neural Networking Center-UPC”.
- Any dispute will be resolved by the Organizing Committee supported by at least one member of the ITU AI/ML in 5G organization.
Note: In the challenge, you may use any existing neural network architecture (e.g., the RouteNet-Fermi implementation we provide). However, it has to be trained from scratch and it must be clearly cited in the description of the solution. In the case of RouteNet-Fermi, it should be cited as it is in [link].
In case of any personal questions about the rules, please contact the organizers via email at gnnetchallenge bnn.upc.edu .
Contact and updates
Subscribe to [Challenge2023 mailing list] for announcements and questions/comments.
Please, note that it is needed to subscribe to the mailing list before sending any email. Otherwise, emails are dropped by default.
Timeline
- Challenge duration: June-Nov 2023
- Open registration: June 7th-Sep 30th
- Release of tools and validation dataset:
June 30thJuly 3rd - Score-based evaluation phase: Oct 2nd-Oct 17th 2023
- Provisional ranking of all the teams: Oct 18th 2023
- Top-5 teams submit the dataset, code and documentation: Nov 1st 2023
- Final ranking and official announcement of top-3 teams: Nov 2023
- Best solutions pitch in a 3-day event end of to determine the finalists: 28 – 30 November 2023
- GNNet Challenge award ceremony and presentation: Dec 8th 2023 in Paris
- ITU Award ceremony and presentations: Dec 13th, 2023
Please note that these dates are tentative and may change slightly over the course of the challenge. Stay tuned for further updates via the mailing list and also on this website.
Organizing Committee
Miquel Ferriol
BNN-UPC
Albert López
BNN-UPC
Carlos Güemes
BNN-UPC
Prof. Albert Cabellos
BNN-UPC
Prof. Pere Barlet-Ros
BNN-UPC
Scientific Advisory Board
krzysztof Rusek
AGH University of Science and Technology
Fabien Geyer
Technical University of Munich
José Suárez-Varela
Telefónica Research
Barcelona Neural Networking Center (BNN-UPC)
Universitat Politècnica de Catalunya
ITU AI/ML in 5G Challenge
ITU invites you to participate in the ITU Artificial Intelligence/Machine Learning in 5G Challenge, a competition which is scheduled to run from now until the end of the year. Participation in the Challenge is free of charge and open to all interested parties from countries that are members of ITU.
Detailed information about this competition can be found on the Challenge website.
Want to explore the application of GNNs to networks?
Check out IGNNITION
Framework for fast prototyping of GNNs for communication networks
Developed by networking enthusiasts for scientists and engineers of the field.
Build a custom GNN tailored to your networking problem.
Cite this competition
Plain text (IEEE format):
J. Suárez-Varela, et al., “The Graph Neural Networking challenge: a world-wide competition for education in AI/ML for networks,” ACM SIGCOMM Computer Communication Review, vol. 51, no. 3, pp. 9–16, 2021.
BibTEX:
@article{suarez2021graph,
title={The graph neural networking challenge: a worldwide competition for education in AI/ML for networks},
author={Su{\'a}rez-Varela, Jos{\'e} and others},
journal={ACM SIGCOMM Computer Communication Review},
volume={51},
number={3},
pages={9--16},
year={2021}
} References
[1] J. Suárez-Varela, P. Almasan, M. Ferriol-Galmés, K. Rusek, F. Geyer, X. Cheng, X. Shi, S. Xiao, F. Scarselli, A. Cabellos-Aparicio, P. Barlet-Ros, “Graph Neural Networks for Communication Networks: Context, Use Cases and Opportunities”, arXiv:2112.14792, 2021. [paper]
[2] K. Rusek, J. Suárez-Varela, A. Mestres, P. Barlet-Ros, A. Cabellos-Aparicio, “Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN,” In Proceedings of the ACM Symposium on SDN Research (SOSR), pp. 140-151, 2019. [ACM SOSR 2019] [arXiv]