Evaluation Criteria and Judging

The panel of SMEs and judges will review each contestant entry in Stage 1. The panel makes an independent assessment of each Components Paper based on the scoring criteria outlined below. As these papers may be released publicly, do not include sensitive materials in the paper (e.g., personally identifiable information, social security numbers, business-sensitive information, tax id numbers).

Criterion 1: Technical Alignment (50%)

The extent to which the proposed research aligns with the needs and open problems within the stated domain; the responsiveness to the public safety scenarios; the likelihood that successful implementation of the proposed solution will have a significant real-world impact; and the feasibility of implementation of the technology on a UAS, at a reasonable cost, in the immediate term.

Criterion 2: Risk Management Understanding (20%)

The extent to which the contestants demonstrate an understanding of the benefits and risks of implementations of their research, and how it fits into the risk management of public safety end users, particularly with regard to Cybersecurity and Artificial Intelligence (AI) risks.

Criterion 3: Presentation and Clarity (30%)

The quality of presentation and clarity, including the appropriate use of language, ease with which the contributions of the research can be understood, and the likelihood that an appropriately knowledgeable researcher can recreate, verify, and build on this work.

Contestants whose entry meets competition eligibility requirements and whose submission meets Stage 1 submission requirements will be included in the Stage 1 review. Specific scores will not be released publicly or provided to the contestant. Contestants selected by the judges panel will be awarded cash prizes as outlined in Awards and Funding.

NIST will review each contestant entry for Stage 2. Contestants failing to pass Criterion 1 and/or meet the minimum requirements will be disqualified and will be ineligible to compete in this stage. Following the deadline, qualifying submissions will be evaluated on the leaderboard and scored by a panel of judges to confirm the final leaderboard ranking.

Criteria: Measurement of Capabilities – Whole System (also used for Optional Mid-Stage Progress Review)

Criterion 1: Required Specifications (pass/fail)

Review the contestants’ prototype UAS, video, team statement, and bill of materials to ensure that the UAS is compliant with all Required Specifications. Full compliance is required.

Criterion 2: Measured Specifications and Capabilities (Leaderboard Ranking)

This criterion comprises test method(s) results verification and attestation video performance for the following metrics:

• Inspect and Download Data (30%)
• Survey Acuity (30%)
• Autonomous Obstacle Avoidance (20%)
• Endurance (Time) (10%)
• Endurance (Distance) (10%)

Criteria: Best-in-Class

Best-in-Class awards recognize contestants achieving the most outstanding metrics in a specific capability that advances the challenge goals. All contestants in Stage 2 who pass Criterion 1: Required Specifications are eligible for the following Best-In-Class awards:

• Best-in-Class Endurance

• Best-in-Class Inspection and Survey Acuity

• Best-in-Class Radio

From the Main Contest, up to the top 10 Stage 2 winners selected by the judge panel will be awarded cash prizes as outlined in Awards and Funding and receive invitations to Stage 3: Live Evaluation. From the Best-in-Class Contests, the best performing team for each capability will receive respective Best-in-Class awards. NIST reserves the right to award non-cash prize invitations to contestants who are not in the top 10 Stage 2 winners or Best-in-Class Winners, but who pass Criterion 1 and receive a score on Criterion 2 above 30% of the total.

NIST will review each contestant entry in the Live Final contest. A submission that fails to meet the requirements specified in Required Specifications will be disqualified and will be ineligible to compete in this contest. Submissions that pass the UAS Safety Review will be evaluated and scored by a panel of SMEs and judges. Submissions will be judged according to the criteria below:

Criterion 1: Required Specifications (pass/fail)

Examine the contestants’ UAS and contest deliverables to ensure that the UAS is compliant with all Required Specifications. Full compliance is required.

Criterion 2: Performance in Testing and Scenario Stations  (Max 100/100 points)

Contestants have the performance of their systems measured in each testing station. In one station, the Test Method station, they will replicate the measurements they performed in Stage 2. In the scenario stations, they will measure their performance in operational scenarios. Teams may repeat stations; a team’s highest measurement in each station will be counted. All stations are normalized to the highest measurement in that station across all teams to compute each team’s score in each station. All stations are weighted equally to yield the total score for each team. 

Prize Award Evaluation: First through Third Place

For the First through Third Place awards, contestants will be evaluated using a point system. Contestants who pass Criterion 1: Required Specifications and have the highest aggregate score across the remaining criterion will be ranked; the highest three scoring contestants will receive place awards.

Prize Award Evaluation: Best-in-Class Categories

Special category awards recognize contestants with specific requirements for outstanding achievement or contribution to advancing the challenge goals. All contestants in Stage 3 who pass Criterion 1: Required Specifications are eligible for Special Category awards. Contestants will be evaluated at-large, and up to five $10,000 prize awards are anticipated to be awarded: