Share us



Announcement:

- The 2021 IEEE Autonomous Driving AI Test Challenge is now an online only event. It will take place from 23rd August to 26th August
- Please check for all training recordings and slides
- Click to See Deliverables
- IMPORTANT:
     We would also like to remind you that the submissions are accepted through GitHub (NOT via website).
     We have sent submission instructions to all registered teams via emails. If you registered but have not received the instructions,
     please let us know ASAP by sending emails to avtest2021@gmail.com
- Deadline: We are happy to announce that we extend the submission deadline for Deliverable#1 till the 7th of May, 11:59pm PST.


news

Latest News:

Mar. 24 2021:
We are going to host the first Q&A / demo sessions on March 26 / March 27 and April 9/10. Please click the details

Oct. 26 2020:
Please check the the new Announcement on participating the online challenges. For further details about the online challenges, please take a look at the online venue page.
Online Venue


How it might look



Sponsors





2021 IEEE Autonomous Driving AI Test Challenge

Sponsored by IEEE AI Test Conference 2021

Introduction:

Testing autonomous driving vehicles refers to a quality assurance process in which diverse validation activities are performed based on well-defined quality assurance standards and assessment criteria. In a quality validation process, unmanned autonomous driving vehicles are validated at different levels (component, integration, and system) based on the pre-defined quality requirements to assure system quality in algorithms, functions, components, behaviors, connectivity, sensing, performance, intelligence, and decision-making in diverse contexts and conditions. To reduce the cost and increase the testing efficiency in the autonomous driving industry, simulation testing receives increasing attention and effort in recent years. A high-fidelity simulation software usually contains the mathematical representations of the environment, the dynamics of autonomous vehicles and surrounding vehicles, the sensors models, etc., at different levels, and is needed to facilitate the testing and development of autonomous driving systems. In order to perform efficient simulation testing, techniques for optimizing and accelerating testing processes are in great demand.

This challenge is set up as a platform to address this demand, and advocate the importance and need of quality validation and automation for autonomous driverless cars. This platform provides a global competition opportunity for international student teams and professional teams to develop diverse simulation testing techniques and approaches in test scenario generation and automation.

Simulation environment:

In this challenge, LGSVL simulator will be used to support simulation testing and execution. LGSVL simulator is a Unity-based autonomous vehicle simulator developed by LG Electronics America R&D Center. The LGSVL simulator can generate various realistic 3D environments by adjusting environmental parameters including maps, weather, traffic, and pedestrians. It can also simulate different sensor outputs, including camera, Lidar, radar, ultrasonic, etc., and lots of virtual sensors to generate ground truth data (e.g. depth, semantic/instance segmentation, 2D/3D bounding box, etc.). Users can generate and test arbitrary edge case scenarios and simulate billions of miles. LGSVL simulator is fully integrated with the open-source platforms Baidu Apollo and Autoware. In this challenge, teams will use LGSVL integrated with Baidu Apollo to generate and evaluate test cases.

The details about LGSVL could be found in the following URLs:
https://www.lgsvlsimulator.com/
https://www.lgsvlsimulator.com/docs/

AV Simulation Test Challenge:

The challenge consists of two phases:

1. Training and team selection. During this phase, training materials including tutorial videos, documents, and online Q&A sessions will be delivered to the teams so that the teams will get familiar with LGSVL simulator.
Available training materials:
> Tutorial: End-to-end video tutorial of LGSVL simulator with Apollo driving
> Getting started with LGSVL simulator
> Getting started with Baidu Apollo
> Online course on Software Testing for Complex Intelligent Systems and Autonomous Vehicles, video list

Two Q&A / demo sessions details
(1) March 26, 2021 from 5pm - 6:30pm PST
     March 27, 2021 from 9am - 10:30am PST)
(2) April 9, 2021 at 5pm - 6:30pm PST
     April 10, 2021 at 9am - 10:30am PST
Please check for all the Zoom links and recordings

2. Competition. During this phase, teams use the traffic accident/crash database to create scenarios, automation scripts, and UI and submit them for evaluation.

Challenge Tasks for Each Team:

1. Form a team and make your challenge registration via specified challenge platform. Each team can have up to 6 members.
2. Attend challenge training sessions
3. Work and submit the first phase deliverable. Each team needs to work on deliverable #1 and submit the following artifacts for each scenario to the team’s submission folder in the given challenge submission link.
-- Scenarios and associated scripts
-- Simulation execution report (Note that “Test Results” are not available in LGSVL Simulator 2020.06 which was initially recommended for deliverable #1, so this report is not required for deliverable #1. Teams may instead choose to include any interesting console output generated by the scenario script during the execution of each scenario. If using SVL Simulator 2021.1, a report may be generated by enabling the Create Test Report option, printing the resulting Test Results screen as a PDF and including that report.)
-- Scenario demo videos (e.g. screen capture video; submit link to YouTube, Google Drive, etc.)
-- Selected scenario trip/route and map information (e.g. which maps were used; include descriptions and/or sketches of routes for ego and NPC trajectories)
-- A (written) simulation test report, which documents the generated test scenario scripting methods, and problem findings.
4. Work and submit the second phase deliverable. Each team needs to work on deliverable #2 and submit your challenge project artifacts (same as before, listed above) for each scenario to the team’s submission folder in the given challenge submission link.
5. Final challenge demo and evaluation for selected final teams (for the selected final teams).
6. Final paper submission and presentation for selected final teams (for the selected final teams).

Click to See Deliverables

Challenge Evaluations:

The results will be evaluated based on
1) Test simulation automation in test simulation modeling, auto-script generation, auto-result validation, auto-coverage analysis, and auto-report
2) Models and methodology
3) Simulation demos based on specified routes, scenarios, and rules
4) The number of test simulation scenarios

The submissions will be evaluated based on the following criteria:

AV Test Simulation Generation – Evaluation Criteria

This evaluation criteria listed below provides detailed assessment rubrics for AV test simulation scenario generation and script generation from different teams.



AV Test Simulation Demo – Evaluation Criteria

This evaluation criteria listed below provides detailed assessment rubrics for challenge demos given from different teams.



Challenge Organization Platform

Github

Getting Started with LGSVL:

The instructions on how to get started with LGSVL and Apollo including the system requirements can be found at https://www.lgsvlsimulator.com/docs/getting-started/ and https://github.com/ApolloAuto/apollo. Each team must have local machines running LGSVL Simulator and Apollo.

LGSVL cloud will be used to run scenarios by judges for verification. LGSVL cloud can run LGSVL Simulator, Apollo ad stack, and scenario/Python scripts. However, users can only view the visualization of recorded data and can see an analysis report of all the simulations after simulations are finished.

Cloud support:

Amazon AWS cloud environment

Challenge Awards: ($10K)

- First place team ($5K)
- Second place team ($3K)
- 2 Third place teams ($2K, each with 1K)







      Register for challenge

Dates

Important Dates

Registration deadline :
March 15 2021

Training, preparation starts:
March 2021

First phase challenge submission:
May 7 2021

Notification of selected teams :
May 15 2021

Second phase challenge submission:
July 15 2021

Paper submission:
July 15 2021

AI Test Conference presentations
August 23 2021

Announcement of winners:
August 2021

Contacts

Contacts:

Jerry Gao
Jerry.Gao@sjsu.edu

Wencen Wu
wencen.wu@sjsu.edu

Oum-El-Kheir Aktouf
oum-el-kheir.aktouf@lcis.grenoble-inp.fr


About Website:
Everette Li
everetteli12@gmail.com

Zizhen Huang
zizhen.huang@sjsu.edu