Graph Neural Networking
Challenge 2020
ITU Artificial Intelligence/Machine Learning in 5G Challenge
ITU invites you to participate in the ITU Artificial Intelligence/Machine Learning in 5G Challenge, a competition which is scheduled to run from now until the end of the year. Participation in the Challenge is free of charge and open to all interested parties from countries that are members of ITU.
Detailed information about it can be found on the Challenge website, which includes the document “ITU AI/ML 5G Challenge: Participation Guidelines”.
BNN-UPC is glad to announce below the “Graph Neural Networking challenge 2020”, which is organized as part of the “ITU Artificial Intelligence/Machine Learning in 5G Challenge.”
This event has finished. Check for new editions at this [link]
Final ranking
| RANK | TEAM | MAPE (%) |
| 1 | Steredeg | 1.53 |
| 2 | Salzburg Research | 1.95 |
| 3 | Gradient Ascent | 5.42 |
| 4 | (Disqualified team) | |
| 5 | GLOBAL | 53.44 |
| 6 | CTL-KAUST | 72.81 |
| 7 | HPE CMS France | 74.77 |
| 8 | WAZD | 95.91 |
| 9 | IITH Team 1 | 183.79 |
| 10 | Tokio | 338.26 |
Overview
Network modeling is essential to build optimization tools for networking. For instance, an accurate network model enables to predict the resulting performance (e.g., delay, jitter, loss) and helps to find the configuration that maximizes the network performance according to a target policy (e.g., minimize the end-to-end delay).
Nowadays, network models are either based on packet-level simulators or analytical models (e.g., queuing theory). The former are very costly computationally, while the latter are fast but not accurate. In this context, Machine Learning (ML) arises as a promising solution to build accurate network models able to operate in real time.
Recently, Graph Neural Networks (GNN) have shown a strong potential to be integrated into commercial products for network control and management. Early works using GNN have demonstrated an unprecedented capability to learn from different network characteristics that are fundamentally represented as graphs, such as the topology, the routing configuration, or the traffic that flows along a series of nodes in the network. In contrast to previous ML-based solutions, GNN enables to produce accurate predictions even in networks unseen during the training phase. Nowadays, GNN is a hot topic in the ML field and, as such, we are witnessing significant efforts to leverage its potential in many different fields (e.g., chemistry, physics, social networks). In the networking field, the application of GNN is gaining increasing attention and, as it becomes more mature, is expected to have a major impact in the networking industry.
Problem statement
The goal of this challenge is to create a solution based on neural networks that estimates performance metrics given a network snapshot. More in detail, this solution (Fig. 1) must predict the resulting per-source-destination mean per-packet delay given: (i) a network topology, (ii) a network configuration (routing, queue scheduling), and (iii) a source-destination traffic matrix.
Figure 1: Schematic representation of the neural network-based solution requested
Baseline
As a baseline, we provide RouteNet [1], a Graph Neural Network (GNN) architecture recently proposed to estimate per-source-destination performance metrics (e.g., delay, jitter, loss) in networks. Thanks to its GNN architecture, RouteNet revealed an unprecedented ability to make acccurate performance predictions even in network scenarios unseen during the training phase, including other network topologies, routing configurations, and traffic matrices (Fig. 2).
Figure 2: Schematic representation of RouteNet
In this challenge, we extend the problem to modeling network performance in the presence of different queue scheduling policies at network nodes. So far, RouteNet does not include support for modeling the impact of multi-queue scheduling policies on network performance and, consequently, it produces poor estimates of network scenarios that include such policies.
We provide an open source implementation of RouteNet including a tutorial on how to use and modify fundamental characteristics of the model.
Participants are encouraged to update RouteNet or submit their own neural network architecture.
Dataset
For this challenge, we provide a dataset generated with the OMNet++ network simulator, which is a discrete event packet-level network simulator. The dataset (Fig. 3) contains samples simulated in several topologies and includes hundreds of network configurations (routing, queue scheduling) and traffic matrices. Each sample is labeled with per-flow performance measurements (mean per-packet delay, jitter and loss) resulting from the simulation. In particular, this challenge focuses only on the prediction of mean per-packet delay for each flow.
In this challenge, we extend the problem to modeling network performance in the presence of different queue scheduling policies at network nodes. So far, RouteNet does not include support for modeling the impact of multi-queue scheduling policies on network performance and, consequently, it produces poor estimates of network scenarios that include such policies. We provide an open source implementation of RouteNet including a tutorial on how to use and modify fundamental characteristics of the model. Participants are encouraged to update RouteNet or submit their own neural network architecture.
Figure 3: Schematic representation of the datasets provided for the challenge
Three different datasets will be provided for this challenge: (i) training, (ii) validation, (iii) test.
The training and validation datasets include the per-flow performance measurements (output labels). These datasets will be released at the beginning of the challenge. The test dataset will not include any per-flow performance measurements, and it will be used at the evaluation phase to test the accuracy of the proposed solutions. See tentative release dates at the “Timeline” section.
The training dataset contains samples simulated in two different network topologies, while the validation and test datasets include samples simulated in a different topology.
You can download the datasets and see a detailed description at the link below:.
In order to easily read and process the data from our datasets, we provide a Python API. We strongly recommend use this API in order to abstract participants from the complex structure of our datasets.
Evaluation
Before the end of the challenge, we will provide a test dataset. This dataset will contain samples with similar distribution to the samples present in the validation dataset. Participants must label this dataset with their neural network models and send the results in CSV format. For the evaluation, we will use the Mean Absolute Percentage Error (MAPE) score computed over all the source-destination delay predictions produced by the candidate solutions:
Solutions with lower MAPE score will be the winners.
Note: The test datset will not include any data of performance measurements (i.e., per-flow delay, jitter and loss). It means that any of these features can be used as input of the proposed solutions. For those using the dataset API, they can assume that NO data under the performance_matrix data structure can be used as input for the model.
Rules
It is mandatory filling the registration form at https://challenge2020.bnn.upc.edu/registration to officially participate in this challenge.
To participate in this challenge, the following rules must be satisfied:
- You can participate in teams of up to 4 members (i.e., 1-4 members). All the team members should be announced at the beginning (in the registration form) and will be considered to have an equal contribution.
- The proposed solutions must be fundamentally based on neural network models.
- The proposed solution cannot use network simulation tools.
- Solutions must be trained only with samples included in the training dataset we provide. It is not allowed to use additional data obtained from other datasets or synthetically generated.
- During the evaluation process, each team can submit a maximum of 5 solutions (CSV files) per day. In total, teams can make up to 20 evaluation submissions. In case of receiving more submissions, only the first submissions up to these limits will be considered.
- The challenge is open to all participants except members of the organizing team and its associated research group “Barcelona Neural Networking Center-UPC”.
Important notice: In the challenge, you may use any existing neural network architecture (e.g., the RouteNet implementation we provide). However, it has to be trained from scratch and it must be clearly cited in the solution description. In the case of RouteNet, it should be cited as it is in [1].
In case of any personal questions about the rules, please contact the organizers via email to gnnetchallenge <at> bnn.upc.edu .
Final submissions and winners
After the score-based evaluation phase (see “Evaluation” section), a ranking of all the teams will be published.
Then, top 5 teams must send to the organizers: (i) the code of the neural network solution proposed, (ii) the neural network model already trained, and (iii) a brief document describing the proposed solution and how to reproduce it (1-3 pages). This will be revised to check that the solutions comply with all the rules of the challenge.
Top 3 solutions of this challenge will have access to the global round of the ITU AI/ML in 5G Challenge, which will award the best solutions among all the challenges proposed under this initiative:
- 1st prize: 5,000 CHF (approx. $5,510 /4,640 €)
- 2nd prize: 3,000 CHF (approx. $3,306 / 2,784€)
- 3rd prize: 2,000 CHF (approx. $2,204 / 1,856€)
- 3 Runners up will receive 1,000 CHF (approx. $1,102 / 928€) each.
There will be other amounts to be announced for the winners of each problem statement.See more details at the (ITU AI/ML challenge website).
Moreover, top 3 teams will be recognized in this website and certificates of appreciation will be generated for them.
Lastly, members of the winning team will have the opportunity to present their solutions at the BigDAMA workshop, which will be tentatively co-located with ACM CoNEXT (December 1-4, 2020).
Unfortunatelly, the BigDAMA workshop will not be celebrated with ACM CoNEXT this year. We are looking for alternatives to publish top solutions in other venues. We will keep you updated.
The organizers are open (and encourage) to publish co-authored papers describing the top solutions proposed. Always respecting the desire of those participants that want to maintain their solutions private.
Contact and updates
Subscribe to [Challenge-KDN mailing list] for announcements and questions/comments.
Please, bear in mind that it is needed to subscribe to the mailing list before sending any email.
Resources
We summarize below the main resources provided for this challenge:
- Summary slides (Link: https://www.itu.int/en/ITU-T/AI/challenge/2020/Documents/Presentation_gnnet_challenge.pdf)
- Registration form (Link: https://challenge2020.bnn.upc.edu/registration)
- Training/validation datasets (Link: https://challenge.bnn.upc.edu/dataset)
- API to easily read data from the datasets provided (Link: https://github.com/knowledgedefinednetworking/datanetAPI/tree/challenge2020 )
- Baseline: RouteNet implementation in TensorFlow 2.1 including a brief tutorial (Link: https://github.com/knowledgedefinednetworking/RouteNet-challenge)
- Mailing list for questions and comments related to the challenge (Link to subscribe: https://mail.knowledgedefinednetworking.org/cgi-bin/mailman/listinfo/challenge-kdn).
Timeline
Challenge duration: May 22nd-Nov 20thRegistration open to participants: Deadline Aug 21st (extended)Training and validation datasets release: May 22ndTest dataset release: Sep 30thScore-based evaluation phase: Oct 1st-Oct 15thProvisional ranking of all the teams: Oct 16thTop 5 solutions submit their code and documentation: Oct 31stWinners (top 3) official announcement: Nov 20thAwards and presentation: mid-December
Organizers
BNN-UPC
BNN-UPC
BNN-UPC
BNN-UPC
BNN-UPC
BNN-UPC
AGH
Barcelona Neural Networking Center (BNN-UPC)
Universitat Politècnica de Catalunya
ITU Artificial Intelligence/Machine Learning in 5G Challenge
ITU invites you to participate in the ITU Artificial Intelligence/Machine Learning in 5G Challenge, a competition which is scheduled to run from now until the end of the year. Participation in the Challenge is free of charge and open to all interested parties from countries that are members of ITU.
Detailed information about it can be found on the Challenge website, which includes the document “ITU AI/ML 5G Challenge: Participation Guidelines”.
References
[1] K. Rusek, J. Suárez-Varela, A. Mestres, P. Barlet-Ros, A. Cabellos-Aparicio, “Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN,” In Proceedings of the ACM Symposium on SDN Research (SOSR), pp. 140-151, 2019. [ACM SOSR 2019] [arXiv]