Whole System

Measurement of Capabilities

In this stage, all eligible contestants can enter. Each contestant will need to purchase, build, or adapt an existing UAS capable of flight, sensing, and with radio hardware necessary to satisfy the Required Specifications.

Contestants will build specified test method apparatuses to run/fly, record attestation videos, and measure the performance of their system. They will submit their measurements and corresponding attestation videos to a contest leaderboard for review by judges and other contestants. Contestants may update their test methods results and accompanying attestation video at any time prior to the closing date. Contestants may wish to submit early and have their test methods results and attestation video assessed by other contestants. Contestants may also submit by a mid-stage deadline and request a preliminary review from competition organizers.

Up to 10 contestants will be selected from the top 10 as ranked on the leaderboard, after verification of their measurements, to receive cash prize awards and an invitation to participate in Stage 3: Live Final.

Introduction

NIST PSCR invites all eligible contestants, including but not limited to all Stage 1 participants, to demonstrate their technical ability to solve the overall challenge problem. They will do this by measuring different aspects of their solution’s performance through self-administered, video-attested test methods at their home location or location of their choosing. Each test measures one metric. Contestants will submit the attestation videos and self-evaluated measurements of these metrics to a leaderboard. The leaderboard will be accessible to all registered challenge contestants and will remain open for unlimited resubmissions until the stage deadline. For each metric, the best measurement of the team’s final submission will be used as the final score for each criterion. The top 10 contestants with the highest combined scores across all metrics will be awarded cash prizes and invitations to participate in Stage 3: Live Final. Additionally, up to 3 contestants may win Best-in-Class Awards.  Other contestants may win a non-cash prize invitation to Stage 3.

Mid-Stage Submission for Progress Review Awards (optional)

An optional mid-stage submission deadline will give contestants the option to submit test method measurements and attestation videos in order to compete for mid-stage cash prize awards and receive feedback from the competition organizers before the final submission. Contestants failing to pass Criterion 1 and/or meet the minimum requirements will be ineligible to compete in this Mid-Stage Progress Review.  Competition organizers will provide feedback and judges will award cash prizes for up to 20 submissions as noted in Awards and Funding based on passing Criterion 1: Required Specifications and ranking in performance demonstrated through submission materials. Following the mid-stage submission deadline, contestants may continue to submit additional test measurements and attestation videos until the final submission deadline.

Performance Measurement

Contestants will be provided with detailed instructions to construct their own test method apparatuses, along with procedures to follow for measuring their UAS’ performance and recording videos that attest to these measurements. Contestants are free to run the tests, measure their own performance, and submit their results and videos as many times as they wish. The team’s last submission will be counted for their final score. The test methods are designed so that the properly recorded video can be used to prove that the test was correctly constructed and run, and that the reported test method(s) measurements are accurate.

UAV Configuration

As the test methods represent a decomposition of the tasks performed during a mission, all of the test results that a team wishes to count towards their Stage 2 score must be achieved with the same UAV configuration. If the UAV configuration changes, the team shall report that this has occurred, and all previous results will be invalidated. Organizations with several different UAV configurations are invited to enter the challenge with a new team, one submission per unique UAV configuration, where there are no overlapping team members.

A change of any part in the UAV that is not a like-for-like replacement is considered a configuration change. Examples of changes that are considered configuration changes include:

• Changing or adjusting landing gear (unless the adjustment is done purely by the UAV’s own actuation).
• Replacing the battery with a larger or smaller capacity unit.
• Replacing the propellers with ones that are different in size, pitch, or other specification.
• Changing the camera for one that has a different specification.
• Changing the antenna or radio with one that has a different physical or electrical specification.

Examples of changes that are not considered configuration changes include:

• Changing a battery for one that is of identical specification (disregard age differences).
• Replacing broken propellers with new ones of the same specification, as long as all tests are started with propellers that are new or have minimal wear.
• Changing the software or firmware on components of the UAV (as this could conceivably be done over the air without physical contact with the UAV).

Contestants should consult with the competition organizers if there is any doubt that a repair or other change to the UAV constitutes a new configuration.

This restriction only covers the UAV (the flying vehicle itself). Parts of the UAS (e.g., the ground control station) that remain in the physical vicinity of the team members during the mission may be changed between tests.

Leaderboard Evaluation and Informal Peer Review

All measurements submitted in Stage 2 will be publicly viewable by all other contestants at the time of upload. Competition organizers reserve the right to release all measurements and videos to the general public at the end of the competition. Contestants will ensure that no sensitive information is present in the videos.

When the Stage 2 submission period opens, contestants are allowed to submit their test method(s) measurements and attestation video early for peer review by other contestants via the leaderboard. Once the leaderboard is open to submissions, eligible contestants can view and provide feedback on contestants’ videos through the competition portal.

Contestants may continually submit test method(s) measurements and attestation videos up until the Stage 2 submission deadline to correct non-compliant videos or if they achieve better measurements in later testing. Note that any changes to the UAV configuration, such as improvements to the UAV itself, will require contestants to resubmit all test measurements and videos with the new configuration. The final leaderboard ranking will be evaluated by the competition organizers to confirm the ranking. Any videos of tests found to be non-compliant will be discarded so teams are highly encouraged to submit early and make use of peer review.