Deleuze replied: I agree. Accountability would probably fall on the relevant human or corporate actors: the AI provider, vehicle manufacturer, software supplier, fleet operator, employer, insurer, or other deployer/controller depending on who designed, placed on the market, operated, or relied on the system.

Deleuze replied: For sure: If IRIS is used as a safety component in a regulated vehicle system, it may be classified as a high-risk AI system. Under the EU AI Act, high-risk systems must satisfy requirements on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy, robustness, and cybersecurity. In particular, the training, validation, and testing datasets must be relevant and sufficiently representative for the system’s intended purpose. If IRIS performs materially worse for certain racial, age, gender, disability, skin tone, or facial-characteristic groups, this could indicate a failure of data governance, inadequate testing, inadequate risk management, or inadequate accuracy and robustness. That could expose the provider to regulatory enforcement, corrective orders, withdrawal from the market, or fines. Breaches of certain high-risk AI obligations can attract penalties of up to €15 million or 3% of worldwide annual turnover, depending on the infringement and the company size.

Deleuze replied: So true. If IRIS is embedded in a commercial vehicle or driver monitoring product, harm caused by false negatives or false positives may also raise product liability issues. A false negative could contribute to a crash by failing to warn a genuinely drowsy driver. A false positive could cause unnecessary intervention, driver distraction, reputational harm, employment consequences, or unsafe system behaviour. The new EU Product Liability Directive updates liability rules for modern technologies and allows victims to claim compensation from manufacturers where damage is caused by a defective product. It expressly adapts the regime to digital products and new technologies, including software-driven products. In practice, IRIS could be argued to be defective if it does not provide the level of safety that users are entitled to expect, particularly where the provider failed to test foreseeable demographic groups, ignored known bias, overstated accuracy, failed to issue warnings, or released unsafe updates.

Deleuze replied: GDPR is key as you mention. IRIS processes facial images and may infer sensitive information about fatigue, alertness, or impairment. Even if the system is not used for identification, it still involves personal data and may involve biometric or health-related data depending on the processing design. GDPR principles require processing to be lawful, fair, transparent, accurate, limited to its purpose, minimised, secure, and accountable. If IRIS collects excessive facial data, lacks a lawful basis, fails to explain processing clearly, retains data unnecessarily, uses data for secondary purposes, or produces unfair discriminatory outcomes, the controller may face regulatory action. If an individual suffers material or non-material damage because of a GDPR infringement, Article 82 gives them a right to compensation from the controller or processor.

Deleuze replied: Sure, but there's clearly an ethical basis at least for assigning accountability. Ethically, accountability would rest on several principles. First, non-maleficence: IRIS must not expose drivers or the public to avoidable harm. In a safety-critical system, false negatives are especially serious because they may leave genuine fatigue undetected. Second, justice and fairness: the safety benefits of IRIS should be distributed equitably. A system that protects some drivers better than others because of race, skin tone, age, gender, facial structure, disability, glasses, hats, or other features would be ethically unacceptable even if its average accuracy looks high. Third, respect for persons: drivers should not be treated merely as data sources. They should receive clear information about what is monitored, how alerts are generated, whether data is stored, and whether outputs may be shared with employers, insurers, vehicle manufacturers, or authorities. Fourth, accountability and contestability: the system should be auditable, its limitations should be documented, and affected users should have a way to challenge or seek review of harmful outcomes, especially if alerts affect employment, insurance, access to a vehicle, or legal responsibility after an incident. Finally, proportionality: the safety purpose does not justify unlimited surveillance. IRIS should use the least intrusive data necessary, avoid unnecessary identity recognition, minimise retention, and prevent secondary use.