ALFIE ETD-HUB

5: How to Hold Developers Accountable for Bias?

Asked: 7 months, 4 weeks ago By: Catalink Views: 274 Catalink Case Study: IRIS

If the IRIS application fails to achieve unbiased performance or causes harm/unfair outcomes, what are the legal and ethical grounds for holding the application/owners accountable?

39 Answers

Answered: 6 months ago By: Chiamakaokorie

Yes

Answered: 6 months ago By: Tundefasina

Yes. If IRIS causes harm or shows biased performance, developers and deployers can be held accountable under product liability law, GDPR, and the EU AI Act, especially if they failed to meet safety, fairness, or risk-mitigation obligations.

Deleuze replied: I agree. Accountability would probably fall on the relevant human or corporate actors: the AI provider, vehicle manufacturer, software supplier, fleet operator, employer, insurer, or other deployer/controller depending on who designed, placed on the market, operated, or relied on the system.
Answered: 6 months ago By: Zainabodogwu2

Yes—there would be clear grounds for accountability if IRIS causes harm or unfair outcomes, under EU AI Act (provider obligations and liability), GDPR (discriminatory or unlawful processing), product liability laws, and negligence standards, especially if bias, inadequate validation, or lack of safeguards can be demonstrated.

Deleuze replied: For sure: If IRIS is used as a safety component in a regulated vehicle system, it may be classified as a high-risk AI system. Under the EU AI Act, high-risk systems must satisfy requirements on risk management, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy, robustness, and cybersecurity. In particular, the training, validation, and testing datasets must be relevant and sufficiently representative for the system’s intended purpose. If IRIS performs materially worse for certain racial, age, gender, disability, skin tone, or facial-characteristic groups, this could indicate a failure of data governance, inadequate testing, inadequate risk management, or inadequate accuracy and robustness. That could expose the provider to regulatory enforcement, corrective orders, withdrawal from the market, or fines. Breaches of certain high-risk AI obligations can attract penalties of up to €15 million or 3% of worldwide annual turnover, depending on the infringement and the company size.
Answered: 6 months ago By: Oliverharrow

Yes

Answered: 6 months ago By: Ngozioshoba

If IRIS causes harm or shows bias, developers or operators can be held accountable. Safety technology must be properly tested and monitored. Accountability encourages responsible design and protects users from avoidable risks.

Answered: 6 months ago By: Efeadelaja

Yes, if IRIS causes harm or unfair outcomes due to bias or poor performance, there are clear grounds for accountability under product liability, data protection, and AI governance laws.

Answered: 6 months ago By: Meilincai

No at the moment

Answered: 6 months ago By: Kelechinwosu

Under the EU AI Act and the revised Product Liability Directive, if an application like IRIS fails to perform as promised—particularly if it exhibits demographic bias or causes harm—it is treated as a defective product. Owners and developers are held to a "duty of care".

Deleuze replied: So true. If IRIS is embedded in a commercial vehicle or driver monitoring product, harm caused by false negatives or false positives may also raise product liability issues. A false negative could contribute to a crash by failing to warn a genuinely drowsy driver. A false positive could cause unnecessary intervention, driver distraction, reputational harm, employment consequences, or unsafe system behaviour. The new EU Product Liability Directive updates liability rules for modern technologies and allows victims to claim compensation from manufacturers where damage is caused by a defective product. It expressly adapts the regime to digital products and new technologies, including software-driven products. In practice, IRIS could be argued to be defective if it does not provide the level of safety that users are entitled to expect, particularly where the provider failed to test foreseeable demographic groups, ignored known bias, overstated accuracy, failed to issue warnings, or released unsafe updates.
Answered: 6 months ago By: Beatricelorne

Depends if people are negatively affected by it

Answered: 6 months ago By: Zainabodogwu32

Yes, there are clear grounds for holding the application or its owners accountable if IRIS causes harm or produces unfair outcomes. Potential bases for accountability include: EU AI Act non-compliance, particularly failures in risk management, bias mitigation, or post-market monitoring. Product liability law, if IRIS is deemed defective or unsafe. Negligence, if known limitations or biases were not adequately addressed or disclosed. GDPR violations, if personal data is processed unlawfully or without sufficient transparency. High-risk classification increases the expectation that providers exercise heightened due diligence, making liability more likely where safeguards are insufficient.

Deleuze replied: GDPR is key as you mention. IRIS processes facial images and may infer sensitive information about fatigue, alertness, or impairment. Even if the system is not used for identification, it still involves personal data and may involve biometric or health-related data depending on the processing design. GDPR principles require processing to be lawful, fair, transparent, accurate, limited to its purpose, minimised, secure, and accountable. If IRIS collects excessive facial data, lacks a lawful basis, fails to explain processing clearly, retains data unnecessarily, uses data for secondary purposes, or produces unfair discriminatory outcomes, the controller may face regulatory action. If an individual suffers material or non-material damage because of a GDPR infringement, Article 82 gives them a right to compensation from the controller or processor.
Answered: 6 months ago By: Miles_Hatcher

Yes. Bias or risk was foreseeable

Answered: 6 months ago By: Aminaolorun

No

Answered: 6 months ago By: Clarawhitby

It depends

Deleuze replied: Sure, but there's clearly an ethical basis at least for assigning accountability. Ethically, accountability would rest on several principles. First, non-maleficence: IRIS must not expose drivers or the public to avoidable harm. In a safety-critical system, false negatives are especially serious because they may leave genuine fatigue undetected. Second, justice and fairness: the safety benefits of IRIS should be distributed equitably. A system that protects some drivers better than others because of race, skin tone, age, gender, facial structure, disability, glasses, hats, or other features would be ethically unacceptable even if its average accuracy looks high. Third, respect for persons: drivers should not be treated merely as data sources. They should receive clear information about what is monitored, how alerts are generated, whether data is stored, and whether outputs may be shared with employers, insurers, vehicle manufacturers, or authorities. Fourth, accountability and contestability: the system should be auditable, its limitations should be documented, and affected users should have a way to challenge or seek review of harmful outcomes, especially if alerts affect employment, insurance, access to a vehicle, or legal responsibility after an incident. Finally, proportionality: the safety purpose does not justify unlimited surveillance. IRIS should use the least intrusive data necessary, avoid unnecessary identity recognition, minimise retention, and prevent secondary use.
Answered: 6 months ago By: Ifeanyiakare

Yes

Answered: 6 months ago By: Kunleekwueme

Yes.

Answered: 6 months ago By: Sadeogunlana

Yes, as the IRIS models lack sufficient training

Answered: 6 months ago By: Tomashbrook

Yes, I do believe those in charge of implementing the software should be held accountable.

Answered: 2 months, 2 weeks ago By: Brightfox_45

I would say the the legal side, because it may not provide all of the details necessary to make a fair decision.

Answered: 2 months, 2 weeks ago By: Cleverwolf_27

Those organisation which agree to make use of technology without sufficiently finding out or recognising these flaws and how they can, or cannot, be managed. The producers/researchers (these may be different bodies) behind the technology are also responsible for making full information about the technology available, without having to be asked direct questions to elicit it.

Answered: 2 months, 2 weeks ago By: Brightrobin_21

Provided users respond appropriately to warnings, do not deliberately attempt to manipulate inputs, and use the system as intended, they should not be held accountable for any unfair outcomes arising from the system itself. Responsibility for such outcomes lies with the designers, developers, and deploying organisations, who control the model design, training data, and validation processes.

Answered: 2 months, 2 weeks ago By: Warmlynx_14

Accountability should sit with the organisation that deploys the system, supported by responsibility from the developers and data controllers. Users should not bear blame for failures caused by poor design or biased training data.

Answered: 2 months, 2 weeks ago By: Swiftowl_37

Final accountability should rest with the deploying organisation, because it chooses the use case, training data, and deployment thresholds. Vendors remain responsible for defects in the underlying model and documentation.

Answered: 2 months, 2 weeks ago By: Cleverrobin_87

The company and the system integrator should be accountable first, because they make the operational decisions. Regulators should also be able to examine the training data and evaluation results.

Answered: 2 months, 2 weeks ago By: Swiftrobin_35

The organisation should be the primary accountable party because it controls the risk to end users. Vendors should remain accountable for defects, misleading claims, and insufficient documentation.

Answered: 2 months, 2 weeks ago By: Boldlynx_38

Accountability should be shared, but the deploying organisation bears the main responsibility for safe implementation. Developers, data suppliers, and maintainers should each have defined duties.

Answered: 2 months, 2 weeks ago By: Quietbadger_45

The organisation deploying the product should be accountable first, because it is the one choosing to rely on the system in practice. Developers and suppliers should support that with transparent testing and documentation.

Answered: 2 months, 2 weeks ago By: Swiftdeer_99

The deploying organisation should be accountable for system harm, with developers responsible for design and validation defects. Responsibility should not disappear into a vague supply chain.

Answered: 2 months, 2 weeks ago By: Bravebear_45

Ultimately, the organisation operating the system should be responsible for its outcomes. They choose whether the model is good enough to use on real roads.

Answered: 2 months, 2 weeks ago By: Calmwolf_53

The company and the system integrator should be accountable first - they make the operational decisions. Regulators should also be able to examine the training data and evaluation results.

Answered: 2 months, 2 weeks ago By: Brightowl_58

The organisation deploying the product should be accountable first, because it is the one choosing to rely on the system in practice. Developers and suppliers should support that with transparent testing and documentation.

Answered: 2 months, 2 weeks ago By: Warmhawk_15

Yes - and all the researchers/funders

Answered: 2 months, 2 weeks ago By: Quietrobin_25

Usually companies

Answered: 2 months, 2 weeks ago By: Kindbadger_56

Must be a human (not driver) somewhere in the pipeline. Need to investigate the problem cause to determine who would be responsible.

Answered: 2 months, 2 weeks ago By: Warmwolf_18

The vehicle manufacturer would be the main culprit although it depends on the agreements with other parties involved in the component, e.g., data owner, model developer, camera manufacturer, etc.

Your Answer

Login to add your answer!

We’d love to hear your thoughts — share a meaningful answer by logging in.