Using an AI system in production
Assigning responsibilities and documenting the processing.
Allowing humans to retain control
While AI systems can automate tasks and can sometimes be more efficient than a human operator, a human must be called upon when necessary to prevent misuse and ensure certain errors do not become systematic. Measures taken to achieve this must be identified, effective and sustainable.
The following questions may help the data controller to implement the framework allowing compliance with these conditions.
Is supervision by a human operator planned?
Have mechanisms been anticipated for the operator to manually modify a decision made by the AI system (e.g.: modify the profile given to the user of a video sharing platform) or to stop its operation (e.g.: cut off access to an online chatbot whose automatic learning is believed to have led to a system drift)?
Which mechanisms?
What type of intervention is planned?
Human-in-the-loop (human intervention capacity in each decision-making cycle of the system)
Human-on-the-loop (human intervention capability during the system design cycle and monitoring of system operation)
Human-in-command (ability to supervise the overall activity of the AI system and decide when and how to use the AI system in a given situation)
Are these mechanisms subject to clearly formulated protocols known to everyone?
Are they integrated naturally in the processing?
Have the necessary human and material resources been planned?
Are the individuals responsible for this task clearly identified?
Are they easily able to exercise control over the AI system?
Have they received sufficient training?
Is the information provided to them during supervision sufficient?
Have the tools for operator control been sufficiently tested?
Has the identification of cases where human intervention is necessary been put in place (e.g. through the calculation of a confidence indicator)?
How has the risk of automation bias (the tendency of an operator to place too much trust in an automated process) been taken into account?
Have mechanisms been implemented to compensate for this bias?
Which mechanisms?
Implementing transparency to ensure trust
In order to build trust between the data subjects and the AI system data controller, there must be maximum transparency, both in explaining how the system itself works and in explaining individual decisions. Logging to facilitate explicability should not, however, come at the expense of the privacy of operators or data subjects.
Are the elements (data used for inference, confidence indicators, system version, etc.) used for decision-making by the AI system logged?
Can the logged elements be used to explain, retrospectively, a particular decision made by the AI system?
Is this information kept for a justified period of time?
Is it limited strictly to the categories necessary to explain the decision?
Are the operations led by the AI system explained to individuals required to interact with it?
Is the explanation made sufficiently clear and understandable, for example by using specific scenarios and explaining the limitations, assumptions and extreme cases of the system? (e.g. an industrial robot assisting a human in their task will be able to drill but not screw)
Are technical and methodological tools used to enable explainability of the system?
Is the code open source?
Ensuring the quality of processing
In addition to the transparency and supervision measures put in place, technical measures must also ensure that the quality of the AI system's output is maintained throughout its lifetime.
Is there an automatic analysis of the logs to alert the user and/or the individual in the event of failure, abnormal operation or an attack?
Is quality control and matching of the data collected in the real environment with the training and validation data maintained while using the AI system?
Is the quality of the output of the AI system monitored during the life cycle of the AI system?
Specific risks
Some risks specific to certain AI systems require special attention. The examples below help to identify some of the issues.
In the case of continuous learning, is the quality of the data used for training monitored throughout the life cycle of the system?
In the case of federated learning, are measures taken to compensate for the fact that the data is not independent and distributed identically? (e.g. federated learning of a speech recognition model on voice assistants)
If an AI system is used to collect personal data through data mining, is there a check to ensure that only the data of the subjects concerned is collected?
Is there a check to ensure the integrity and veracity of the data (e.g. by checking the authenticity of the sources, and the quality of the data retrieval)?
In the case of a content recommendation algorithm, could the suggestions influence a person's opinions (political, for example)?
Could they go against the interest of certain persons (legal or natural)?
No information is collected by the CNIL.
Would you like to contribute?
Write to ia[@]cnil.fr
