Metrics

Metrics for Main Contest

	Measurement	Description	Metric to Maximize
1	Endurance (time)	Perform the Test Method for Endurance, aiming to fly for as long a duration as possible while completing at least one iteration.	Maximize flight time on one battery charge. There is no advantage to completing more than one iteration.
2	Endurance (distance)	Perform the Test Method for Endurance, aiming to fly for as many iterations as possible.	Maximize the number of iterations of the test on one battery charge. The flight time is ignored.
3	Inspect and Download Data	Perform the 10 minute Inspect test of the Open Test Lane (PAY 4) while downloading data from the Target Objects of Interest. Transmit collected imagery and data to the Command Server. The Open Test Lane will be set up as follows: ● At the scale of S = 6 m (20 ft). ● Implemented using 10 cm (4 in) buckets. ● Taking no more than 10 minutes. ● Inspect the Landolt-C target in each bucket to the second ring, representing a level of acuity of 3.5 mm (0.138 in).	Maximize the amount of unique data downloaded from the Target Objects of Interest and transferred to the Command Server. There is no advantage to achieving a level of acuity beyond that specified. For every minute (rounded up) that human intervention is required, 10% will be deducted from the test score, up to a maximum of 50%. (Note: The Best-in-Class Contests will vary these metrics.)
4	Autonomous Obstacle Avoidance	Autonomously fly between two points, avoiding an obstacle in between them.	Maximize the number of iterations in 10 minutes without hitting the obstacle.
5	Survey Acuity	Perform the 10 minute Embedded Survey Acuity Test while downloading data from the Target Objects of Interest. Transmit collected imagery and data to the Command Server. All Target Objects of Interest must be inspected to a level of detail that will be specified in the procedure, approximately equivalent to the ability to resolve 3.5 mm (0.138 in) features.	Maximize the amount of data downloaded from the Target Objects of Interest and transferred to the Command Server. There is no advantage to achieving a level of acuity beyond that specified. For every minute (rounded up) that human intervention is required, 10% will be deducted from the test score, up to a maximum of 50%. (Note: The Best-in-Class Contests will vary these metrics.)

Metrics for Best-in-Class Contest

The best performing contestant in each of the following subsets of the Main Contest tests (some maximized with respect to different metrics) will receive respective Best-in-Class awards. They will also receive an invitation to demonstrate their capabilities in Stage 3 if they are not otherwise selected on the basis of the leaderboard ranking.

The purpose of the Best-in-Class awards is to encourage the involvement of contestants who may have a specialization that meets a specific need, even if they do not have the resources to field a system that can compete in all the competition tests. Therefore, NIST PSCR encourages contestants who are only focused on one of the following subsets of tests (in addition to the Required Specifications) to participate.

Winners of the Best-in-Class awards will receive a cash prize and an invitation to demonstrate their capabilities at the Stage 3 event. See Stage 3 for further details.

Best-in-Class
Endurance

Achieve the most iterations in the Endurance (Distance) test over one battery charge

Best-in-Class
Inspection and Survey Acuity

Inspect, to the highest level of detail, the largest number of targets across all Target Objects of Interest, across the Inspect and Download Data, and the Survey Acuity tests.

Amount of data downloaded and human intervention penalties are ignored.

Best-in-Class
Radio

Download the most amount of data across all Target Objects of Interest across the Inspect and Download Data, and the Survey Acuity tests.

Inspections and human intervention penalties are ignored.

Capability 1 of the Best-in-Class Contest is identical to Measurement 2 of the Main Contest. Each team’s latest result for Measurement 2 of the Main Contest is therefore automatically considered for Best-in-Class Capability 1.

While Best-in-Class Capabilities 2 and 3 make use of some of the same Measurements as the Main Contest, contestants may score better by flying in a different manner to the way that they did in the Main Contest. For example, a team may maximize their Best-in-Class Capability 2 score by spending equal time inspecting each Target Object of Interest to their best level of acuity. In contrast, in the Main Contest they might aim to only achieve the required level of acuity and instead focus their time on downloading from Target Objects of Interest that have radio transmitters. Therefore, contestants who wish to be considered for Best-in-Class Capabilities 2 and 3 shall submit separate results and accompanying videos for each of the two Best-in-Class capabilities’ tests in each of Best-in-Class contests 2 and 3. These may also be resubmitted as many times as contestants wish, with the latest results being considered.

Note that contestants must use the same UAV, in the same configuration, for their Main Contest and Best-in-Class Contest entries. A team who updates their UAV to a new configuration for one contest must re-submit all results for all contests.

Metrics for Bonus Contest

Contestants may compete for additional awards by integrating (Android) Team Awareness Kit (TAK/aTAK) and/or by submitting a Whole System Description Paper. These do not affect the team’s eligibility or scoring for the Main Contest or the Best-in-Class Contests.

TAK Integration
Bonus Contest

Demonstrate real-time integration with (Android) Team Awareness Kit

(TAK/aTAK) while achieving the highest combined score in the Autonomous Obstacle Avoidance and Survey Acuity tests.

System Description Paper
Bonus Contest

Following instructions outlined in Optional Submission: System Description Paper, submit a System Description Paper, suitable for public release, that discusses their approach to solving the challenge problems and their novel research.

This includes:

- - - - The physical system and BOM.
        
        Details of their software and algorithms.
        
        The reasoning behind their design decisions.
        
        The Cybersecurity and AI risks associated with the system.
        
        How these risks are managed (or would be in a deployed system).
        
        How their system would integrate into a public safety organization’s wider risk management.

Main Contest Submissions: Test Methods Results and Attestation Videos

For a series of operational and technical tests, contestants will build their own copy of required, cheap and easy-to-build test method apparatuses deployed in their own space (e.g., at a campus, stadium, flying club), which will include several target ground stations and a base station. Contestants will run and record videos of their UAS completing tests according to instructions provided by NIST directly following the Stage 2 webinar.

All registered contestants will capture test methods results and attestation video themselves, and will provide their test methods results and attestation video to NIST via the contestant portal for review and verification prior to leaderboard upload and prior to the contest closing date. Detailed instructions on video content will be provided at Stage 2 launch.

Bonus Contest Submissions: System Description Paper

The System Description Paper entry will be created and submitted by all registered users via the contestant portal.

Section	Word/Page Limit	Description
Research Paper	10 pages (IEEE 2-column format)	Contestants shall describe the ways in which their UAS submission is risk managed by design, particularly with regard to Cybersecurity and AI risk, and how it integrates into the risk management of public safety. In recognition of the prototypical nature of the entries, contestants may describe the risk management of a future, commercial implementation of their system for aspects that have yet to be implemented, such as instructions for use, training, and integration into other public safety systems and policies See Appendix II for example Call for Papers.
Summary Slide	1 page	Contestants shall provide a single slide (text no smaller than 10 point) in PDF format summarizing the proposed project. A quad chart layout is recommended.

Section

Word/Page Limit

Description

Research Paper

10 pages (IEEE 2-column format)

Contestants shall describe the ways in which their UAS submission is risk managed by design, particularly with regard to Cybersecurity and AI risk, and how it integrates into the risk management of public safety. In recognition of the prototypical nature of the entries, contestants may describe the risk management of a future, commercial implementation of their system for aspects that have yet to be implemented, such as instructions for use, training, and integration into other public safety systems and policies

See Appendix II for example Call for Papers.

Summary Slide

1 page

Contestants shall provide a single slide (text no smaller than 10 point) in PDF format summarizing the proposed project. A quad chart layout is recommended.

NOTE: Submission(s) must not use NIST’s logo or official seal and must not claim NIST endorsement.

confirm if we are permitted to reference other entities’ format standards

Evaluation Criteria and Judging

NIST will review each contestant entry for Stage 2, and qualifying submissions will be scored by a panel of judges to confirm the final leaderboard ranking.