NIST AI Risk Management Framework: what it asks of a training dataset list

Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities. A maintained inventory identifies the AI systems and models in use, and the resource given to maintaining it is proportionate to the risk the inventoried systems carry.

Policies and procedures are in place that address AI risks associated with third-party entities, including risks of infringement of a third party’s intellectual property or other rights. Third-party AI risk is addressed by policy covering data, models, software and services obtained externally, including the rights position on training data and model outputs.

AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied and documented. Third-party data, model, software and hardware dependencies are monitored on an ongoing basis, not assessed once at procurement, and the controls applied are recorded.

Pre-trained models which are used for development are monitored as part of AI system regular monitoring and maintenance. Pre-trained and transfer-learned models are treated as a monitored component in their own right, since their provenance and behaviour are not controlled by the deploying organisation.

Scientific integrity and TEVV considerations are identified and documented, including those related to experimental design, data collection and selection (e.g., availability, representativeness, suitability), system trustworthiness, and construct validation. The evaluation design is documented as a scientific claim: what was measured, on what data, and whether the measure validly stands for the property claimed.

Approaches for mapping AI technology and legal risks of its components – including the use of third-party data or software – are in place, followed, and documented, as are risks of infringement of a third-party’s intellectual property or other rights. There is a followed approach for mapping the technology and legal risk carried by each component, including data and software obtained from third parties and the rights position attached to them.

Internal risk controls for components of the AI system including third-party AI technologies are identified and documented. For each component carrying risk, the internal control applied to it is identified and written down, so the risk mapping produces controls rather than a list.

Test sets, metrics, and details about the tools used during test, evaluation, validation, and verification (TEVV) are documented. The TEVV record is complete enough for the evaluation to be repeated: which test sets, which metrics, which tools and which versions.

Privacy risk of the AI system – as identified in the MAP function – is examined and documented. Privacy examination covers what the AI system makes possible, inference and re-identification from training data and outputs, not only the lawfulness of the input data.

Fairness and bias – as identified in the MAP function – is evaluated and results are documented. Fairness evaluation states which fairness definition was applied and why, evaluates against it, and records the results including where the definition itself is contested.