Responsible AI Developer Toolkit
Algorithmic management systems, often incorporating AI, are increasingly developed by private companies and licensed to workplaces around the world to make managerial decisions. This includes technologies which analyse data collected from CVs or video interviews to make hiring decisions, all the way to software that uses data collected from employee vehicles, computer activity and wearables to monitor productivity. Algorithmic management is regulated in a host of legal domains, from data protection to labour laws to discrimination. These legal frameworks impose both ex ante and ex post obligations on AI developers to design algorithmic management systems which are fair, transparent, and accountable. Certain provisions also require AI developers to engage with users and communities to inform deployment choices, enabling co-creation and participatory technology design.
In our toolkit for responsible AI developers, we provide a summary of some of the legal obligations which apply to developers of algorithmic management systems in the UK and EU. Building upon this analysis, the toolkit offers suggestions for certain design choices responsible AI developers can take to ensure that algorithmic management systems support the productivity and efficiency of algorithmic management, while upholding effective worker protection.
Our toolkit summarises and extracts key information from over five years of research conducted by the iManage team, led by Professor Jeremias Adams-Prassl. Our analysis focuses on the UK and EU contexts, but may be applicable to other jurisdictions with similar legal landscapes or be drawn upon for comparative analysis. The toolkit only offers a snapshot of our research, with the full published papers linked where applicable and a further research list at the bottom of the page.
What laws shape how fair algorithmic management systems should be developed and deployed?
[M]ore fundamentally, how can we foster genuine innovation whilst also protecting fundamental rights at work? — Jeremias Adams-Prassl, Halefom Abraha, Aislinn Kelly-Lyth, Sangh Rakshita, and Michael ‘Six’ Silberman
Building non-discriminatory algorithmic management systems
The potential for algorithms to discriminate is now well-documented, and algorithmic management tools are no exception. Existing work explores 'fair' machine learning; these typically appeal to discrimination law frameworks to delineate metrics and methods to mitigate bias and unfair algorithmic outputs. Yet, many of these metrics do not account for the breadth and scope of EU and UK discrimination laws. Indeed, direct discrimination has a wider legal scope than typically assumed in the fair machine learning literature (Adams-Prassl et al, 2022). This sub-category of discrimination laws are even more important for developers of algorithmic management systems to be aware of since few exceptions can justify direct discrimination—even if unintentional. The summary below outlines various cases whereby an algorithm might be directly discriminatory, even if protected characteristics or deliberate proxies are not used.
Applicable provisions:
- A decision may be inherently discriminatory (most relevant to the literature on 'proxy discrimination') because it was made using a criterion which is inextricably linked to a protected characteristic.
- In Dekker (Case C-177/88, 1990), for example, the Court regarded pregnancy as inextricably linked to sex and found a decision based on pregnancy to be inherently discriminatory on grounds of sex.
- From a design perspective, to produce an inherently discriminatory outcome, a model must (i) contain a criterion (or criteria) which is (or are) inextricably linked to a protected characteristic (proxy capacity) and (ii) apply that criterion (or criteria) with the result that an individual is treated less favourably (proxy use) (Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024)
- Proxy capacity: the extent to which A, a variable that represents a protected characteristic, can be recreated from a proxy variable P (Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024).
- Proxy use: considers the extent to which a proxy P induces predictions Y^ (Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024).
- Importantly, the inextricable link in proxy capacity need not hold for all protected groups (see James v Eastleigh) Measuring potential proxy capacity can be done via statistical association. Depending on the type of data (numerical, ordinal, binary, categorical), different measures could be appropriate (e.g., Pearson correlation, Spearman’s rank correlation, mutual information) (Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024).
- Complex proxy capacities can be measured by considering how well a protected variable can be predicted from a proxy feature set (see e.g., Feldman et al (2015); Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024)
Applicable provisions:
- Subjectively discriminatory mental processes occur when a protected characteristic plays a role (whether consciously or subconsciously) in a decision-maker’s less favourable treatment of a person (see, e.g., Case C-83/14, 2015; Case C-54/07, 2008).
- Case law on subconscious subjective discrimination has been developed intuitively by judges who generally ask on the facts, whether a given (human) decision was ‘on grounds of’ a protected characteristic — this translates uneasily to the machine context (Weerts, Kelly-Lyth, Binns and Adams-Prassl, 2024)
Non-discriminatory design decisions for algorithmic management
The use of algorithmic management in the workplace could indeed make discriminatory criteria more traceable and outcomes more quantifiable by replacing unreliable and opaque human decision-making processes (Adams-Prassl et al, 2022). However, developers of algorithmic management rarely make decision-making processes transparent, posing barriers to individuals challenging decisions and enforcement action. Much of this opacity is due to decisions made by developers and not technical limitations. Indeed, there are design choices that can be made to improve understanding of how certain data or factors contribute to decision outcomes. These include:
- Empirically modelling pipelines and causal relationships can aid the measurement and tracing of the factors that impact downstream unfairness in algorithmic management — e.g., this could be biased labellers of training data, historical exclusion, modelling decisions, or some other process (Binns, Adams-Prassl, Kelly-Lyth, 2023)
- 'Fairness in pipelines' also showcases the causal relationships between variables that account for disparities between protected classes which can expose how compound decision-making processes can result in unfairness (Binns, Adams-Prassl, Kelly-Lyth, 2023)
- Measure proxy capacity via statistical associations (e.g., Pearson correlation, Spearman’s rank correlation, mutual information) – be aware that the inextricable link need not hold for all protected groups for discrimination to occur.
- Complex proxy capacities can be measured by considering how well a protected variable can be predicted from a proxy feature set (see e.g., Feldman et al (2015))
Building transparent algorithmic management systems
Certain legal provisions governing algorithmic management require systems to be transparent. Broadly, this refers to the need for workers to be aware of and notified about the existence of algorithmic management and its data processing (ex ante) and, following a decision, the rules/parameters used to make decisions by an algorithmic management system should also be visible and explained to workers (ex post). Algorithmic transparency is valued in this context to empower workers to understand—and thus, where necessary, challenge—decisions, rebalance the bargaining power in the employment relationship, as well as safeguard against discriminatory/erroneous decision-making.
Applicable provisions:
- Workers must be informed about how their personal data is being processed by algorithmic management; this includes the categories and sources of information processed – regardless of whether this is collected from the worker or other sources (e.g., use of social media data in hiring software) (Article 12–14 UK GDPR; Article 12–14 EU GDPR)
- Workers should be informed about data processing 'in a concise, transparent, intelligible and easily accessible form, using clear and plain language' (Article 5(1) UK GDPR; Article 5(1) EU GDPR)
- If workers are subjected to 'solely automated decisions which have a legal/significant effect', they must be informed about the existence of these algorithmic management technologies, alongside the significance and envisaged consequences of such processing (Article 15(1)(h) UK GDPR; Article 15(1)(h) EU GDPR)
- Rules specifically for platform workers stipulate that digital labour platforms must, by the first working day, provide workers with 'concise, clearly written, transparent, intelligible, and easily accessible' information about the use of algorithmic management systems (Article 9(1)-(3) PWD)
- If emotion recognition or biometrics are used, workers must be informed about the existence of such technologies (Article 50(3) EU AI Act)
- Information about high-risk AI systems (e.g., algorithmic management systems) must be documented in a publicly available database by providers and deployers (Article 71 EU AI Act)
Applicable provisions:
- Digital labour platforms must also declare the 'main parameters' used in algorithmic management systems, together with their importance and how the platform worker's personal data influence the decisions (Article 9(1)(b) PWD)
- If algorithmic management systems are used to make decisions about the refusal of remuneration, termination, or other similarly significant decisions, then digital labour platforms must provide the grounds for these decisions (Article 11(1) PWD)
- Workers who are subjected to 'solely automated decisions which have a legal/significant effect' are also entitled to 'meaningful information about the logic involved in decisions' – workers can invoke these rights at any reasonable interval (Article 15(1)(h) EU GDPR; Article 15(1)(h) UK GDPR).
- The 'right to meaningful information' should not include a complex explanation of the algorithms used or code, but instead, a comprehensible explanation of how the decision was made which the data subject can be expected to understand (see Dun & Bradstreet).
Improve transparency in and over algorithmic management systems
Increasing the ex post transparency of algorithmic management systems can —sometimes— present tensions with the accuracy or complexity of a model, e.g., when neural networks are adopted (Veale, Silberman and Binns, 2023). While the degradation to model performance can sometimes be overstated, in line with legal obligations, developers may anyway be required to make design choices to improve the transparency over algorithmic management decision-making processes. This could be done by:
- Constraining a model to minimise synergistic and non-linear relationships where this does not significantly compromise performance (Veale, Silberman and Binns, 2023).
- Consider whether simpler predictive systems can be used over neural networks and deep learning, which suffer from model complexity and data opacity making it more difficult for developers to adhere to transparency obligations (Veale, Silberman and Binns, 2023).
- Compile a publicly accessible database which contains instructions for use, technical details, and information about the characteristics, capability and limitations of high-risk algorithmic management systems (Kelly-Lyth, 2021)
- Design user-friendly and accessible ways to inform workers about the existence, significance, and consequences of data processing via algorithmic management systems
Building accountable algorithmic management systems
There are many reasons for different types of human oversight of algorithmic decisions, relating to accuracy, flexibility and dignity. Legislators have been eager to implement and refine provisions across labour laws, the AI Act, and data protection with respect to the extent to which automated systems must incorporate human-in-the-loop (a person is an active part of the execution path) vs human-on-the-loop (human supervises the system in real time and retains the authority to intervene, override, or stop it). This has also included rules about which decisions should never have a 'human-out-of-the-loop (no human intervention or oversight).
Applicable provisions:
- Data subjects have the right not to be subject to solely automated decisions (including profiling) which have legal or similarly significant effects on them (e.g., e-recruitment). The prohibition does not apply if the decision is contractural (not available if special category data is processed) or based on consent (Article 22 EU GDPR).
- In cases where the consent or contract provisions are relied upon, data controllers must provide suitable safeguards including – e.g., the right to human intervention and the right for data subjects to express their view or contest a decision (Article 22(3) EU GDPR).
- High-risk AI systems (e.g., algorithmic management) must be designed in ways so that humans can effectively oversee their operation, prevent health and safety risks, and protect fundamental rights (Article 14, EU AI Act)
Applicable provisions:
- Digital labour platforms must 'regularly monitor and evaluate the impact of individual decisions taken or supported by automated monitoring and decision-making systems’
- Where processing personal data is likely to result in a 'high-risk to individuals' rights and freedoms' (Article 35(1) EU GDPR), a Data Processing Impact Assessment (DPIA) should be completed.
- DPIAs should contain, at least: a systematic description of the envisaged processing, an assessment of the necessity and proportionality, an assessment of the risks to the rights and freedoms of data subjects, and the measures envisaged to address the risks (Article 35(7) EU GDPR).
- Algorithmic Management Impact Assessments (AMIA) can also be completed at the design, development, and deployment stages to evaluate impacts on workers’ rights and freedoms and impacts on work conditions and quality (Adams-Prassl et al, 2022; Kelly-Lyth and Thomas, 2023)
Applicable provisions:
- Platform workers are entitled to access a contact person to discuss and clarify the facts, circumstances, and reasons that have led to a decision (Article 11(1) PWD).
- Platform workers may also be entitled to a written statement of the reasons that certain decisions may have been taken, e.g., decisions concerning dismissal, refusal or remuneration (Article 11(1) PWD)
- Digital labour platforms must design in, through the platform's digital infrastructure or similarly effective means, the possibility for persons performing platform work to contact and communicate with each other and through representatives (Article 20, PWD)
Building accountable algorithmic management systems
Accountable algorithmic management systems are not only facilitated through meaningful human involvement, but also require an ecosystem approach so that technologies (and their deployment conditions) are monitored consistently, and communication channels offer meaningful feedback mechanisms. Ways in which these can be designed and maintained effectively include:
- Conducting and updating DPIAs and AMIAs throughout the development and deployment of algorithmic management systems to evaluate and prevent harms presented by technologies deployed in workplaces (Adams-Prassl et al, 2022; Kelly-Lyth and Thomas, 2023)
- Where explanations or reasons for decisions are required to be provided to workers, ensure that these are specifically tailored to individual circumstances (local explanations) so that individuals can meaningfully understand and utilise the information (Veale, Silberman and Binns, 2023)
- If there are designated human contact points, ensure that these individuals are adequately trained to respond to worker's questions about algorithmic management, ensuring that they also have 'local explanations' at their disposal to offer clarity on both the algorithm's behaviour and the broader socio-technical circumstances leading to the decision (Veale, Silberman and Binns, 2023)
- Digital labour systems should design communication channels into the platform infrastructure which are organised, easy to use, and technically reliable (Veale, Silberman and Binns, 2023)
- Communication channels on digital labour platforms should allow users to easily find links posted by workers directing them to other platforms that are run, circulated, and approved by significant numbers of workers (e.g., to facilitate organisation or union efforts) (Veale, Silberman and Binns, 2023)
• • On top of internal checks, AI developers should seek external conformity assessments for high-risk algorithmic management systems which pose a significant risk to workers’ rights and freedoms (Kelly-Lyth, 2021).
Co-building algorithmic management systems
Co-building algorithmic management systems includes directly engaging workers, trade unions, or affected communities in the design, validation, and oversight of these technologies. Rules, particularly those applicable to digital labour platforms and platform workers, stipulate participatory development of algorithmic management systems. Participatory design not only improves the accuracy and effectiveness of the technologies by incorporating feedback and utilising worker's tacit knowledge, but it also builds trust in the technologies with those who use them.
Applicable provisions:
- Where appropriate, Data Protection Impact Assessments (DPIAs) require data controllers to seek the views of workers or their representatives on the intended processing (Article 36 UK GDPR; Article 36 EU GDPR)
- Where a DPIA indicates that data processing is high-risk and the employer cannot sufficiently address these risks, national supervisory authorities should be consulted before the processing (Article 36 UK GDPR; Article 36 EU GDPR)
- Where digital labour platforms are making decisions likely to lead to the introduction of or substantial changes in the use of algorithmic management, platform workers' representatives (or where there are no such representatives, platform workers) should be informed and consulted about these changes (Article 13(2) PWD)
- Digital labour platforms must declare the number of individuals performing platform work and their contractual and employment status, and any general terms and conditions that apply to these individuals to national labour authorities (Article 9(1) PWD). This data must also be shared with worker representatives, and both representatives and authorities can ask for clarifications, to which the digital labour platforms must provide a ‘substantiated reply' (Article 17(5) PWD)
Involve workers and their representatives in the design, development, and deployment of algorithmic management systems
- Undertake and disclose AMIA for algorithmic management systems to workers and their representatives (Adams-Prassl et al, 2022; Kelly-Lyth and Thomas, 2023. As part of this, workers and representatives should have the right to negotiate and conclude collective agreements.
- Disclose information about the number of algorithmically facilitated significant decisions of different types made in the workplace, such as termination of accounts, and the number of human reviews undertaken and their results with worker representatives (Veale, Silberman and Binns, 2023)
- Take care to perform information and consultation duties with appropriate timing, method and content so these engagements are effective and mutually beneficial (Veale, Silberman and Binns, 2023)
- Design information and consultation processes with the diverse laws, customs, and languages of workers in mind – e.g., this may require provision of translation and interpretation services for worker representatives (Veale, Silberman and Binns, 2023).
- Where no worker representative exists, ensure that information and consultation is still effective by setting down rules to reduce the risk of low-quality procedures that may bias outcomes, e.g., by choosing or briefing particular influential workers or experts, through ‘agenda-setting’ power (e.g., choosing or selectively framing options in worker surveys), or through exclusion of groups of workers (Veale, Silberman and Binns, 2023).
Looking for further information?
Find a list of the research the iManage team conducted and drew on to develop the employee toolkit.
Algorithmic management:
The following article by Aislinn Kelly-Lynth and Anna Thomas and provides an overview of the impacts of algorithmic management. This Wired article also provides a shorter, lay summary of algorithmic management and the work of the iManage team.
Data protection law:
In the following article, Regulating algorithmic employment decisions through data protection law, Halefom Abraha examines how the GDPR applies to algorithmic management, from information and consultation to transparency rights.
Discrimination law:
For a detailed account of the application of UK direct discrimination law to algorithmic decision-making, Directly Discriminatory Algorithms by Jeremias Adams-Prassl, Reuben Binns, and Aislinn Kelly-Lyth examines the case law on direct discrimination in the UK (with brief comparisons to the EU and US), showing its utility in the context of biased algorithmic decision-making. For an detailed account specific to the employment context, see Aislinn Kelly-Lyth's Algorithmic discrimination at work which applies direct and indirect discrimination law to algorithms at work, while advocating for the need for improved transparency for these existing protections to have any weight.
Reuben Binns, Jeremias Adams-Prassl and Aislinn Kelly-Lyth's Legal Taxonomies of Machine Bias: Revisting Direct Discrimination builds on the papers above and transposes the findings into technical analysis. It explores how jurisprudence on UK/EU direct discrimination is overlooked in taxonomies of machine bias and algorithmic fairness. Hilde Weerts, Reuben Binns, Aislinn Kelly-Lyth and Jeremias Adams-Prassl's Unlawful Proxy Discrimination: A Framework for ChallengingInherently Discriminatory Algorithms extends this analysis to explore different elements of discrimination law in terms of proxy capacity and proxy use tests to offer technical approaches and metrics that could bedeveloped or applied to identify inherent direct discrimination in algorithmic decision-making.
Platform Workers Directive:
For an analysis of the Platform Workers Directive with respect to algorithmic management, see Fortifying the algorithmic management provisions in the proposed Platform Work Directive by Michael Veale, Michael 'Six' Silberman, and Reuben Binns. Note, this research was conducted while the Platform Workers Directive was being debated, so its analysis is focused on the proposal/draft version of the Directive.
EU AI Act:
For an overview of the applicability of the EU AI Act to algorithmic management, see Aislinn Kelly-Lyth's article in the Comparative Labor Law & Policy Journal.
This site is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 947806).
If you have any questions or feedback about this site or our research, please get in touch by emailing ai.work@law.ox.ac.uk.