Responsible AI Developer Toolkit

Co-building fair, transparent, and accountable algorithmic management systems

Algorithmic management systems, often incorporating AI, are increasingly developed by private companies and licensed to workplaces around the world to make managerial decisions. This includes technologies which analyse data collected from CVs or video interviews to make hiring decisions, all the way to software that uses data collected from employee vehicles, computer activity and wearables to monitor productivity. Algorithmic management is regulated in a host of legal domains, from data protection to labour laws to discrimination. These legal frameworks impose both ex ante and ex post obligations on AI developers to design algorithmic management systems which are fair, transparent, and accountable. Certain provisions also require AI developers to engage with users and communities to inform deployment choices, enabling co-creation and participatory technology design. 

In our toolkit for responsible AI developers, we provide a summary of some of the legal obligations which apply to developers of algorithmic management systems in the UK and EU. Building upon this analysis, the toolkit offers suggestions for certain design choices responsible AI developers can take to ensure that algorithmic management systems support the productivity and efficiency of algorithmic management, while upholding effective worker protection.

Our toolkit summarises and extracts key information from over five years of research conducted by the iManage team, led by Professor Jeremias Adams-Prassl. Our analysis focuses on the UK and EU contexts, but may be applicable to other jurisdictions with similar legal landscapes or be drawn upon for comparative analysis. The toolkit only offers a snapshot of our research, with the full published papers linked where applicable and a further research list at the bottom of the page. 

What laws shape how fair algorithmic management systems should be developed and deployed? 

[M]ore fundamentally, how can we foster genuine innovation whilst also protecting fundamental rights at work? — Jeremias Adams-Prassl, Halefom Abraha, Aislinn Kelly-Lyth, Sangh Rakshita, and Michael ‘Six’ Silberman

Woman working simultaneously as a delivery worker, remote employee, and caring her child, representing the opportunities and risks of work digitalization.
Julieta Longo & Digit / Better Images of AI / CC BY 4.0

Building non-discriminatory algorithmic management systems

The potential for algorithms to discriminate is now well-documented, and algorithmic management tools are no exception. Existing work explores 'fair' machine learning; these typically appeal to discrimination law frameworks to delineate metrics and methods to mitigate bias and unfair algorithmic outputs. Yet, many of these metrics do not account for the breadth and scope of EU and UK discrimination laws. Indeed, direct discrimination has a wider legal scope than typically assumed in the fair machine learning literature (Adams-Prassl et al, 2022). This sub-category of discrimination laws are even more important for developers of algorithmic management systems to be aware of since few exceptions can justify direct discrimination—even if unintentional. The summary below outlines various cases whereby an algorithm might be directly discriminatory, even if protected characteristics or deliberate proxies are not used. 

Building transparent algorithmic management systems

Certain legal provisions governing algorithmic management require systems to be transparent. Broadly, this refers to the need for workers to be aware of and notified about the existence of algorithmic management and its data processing (ex ante) and, following a decision, the rules/parameters used to make decisions by an algorithmic management system should also be visible and explained to workers (ex post). Algorithmic transparency is valued in this context to empower workers to understand—and thus, where necessary, challenge—decisions, rebalance the bargaining power in the employment relationship, as well as safeguard against discriminatory/erroneous decision-making. 

Building accountable algorithmic management systems

There are many reasons for different types of human oversight of algorithmic decisions, relating to accuracy, flexibility and dignity. Legislators have been eager to implement and refine provisions across labour laws, the AI Act, and data protection with respect to the extent to which automated systems must incorporate human-in-the-loop (a person is an active part of the execution path) vs human-on-the-loop (human supervises the system in real time and retains the authority to intervene, override, or stop it). This has also included rules about which decisions should never have a 'human-out-of-the-loop (no human intervention or oversight). 

Co-building algorithmic management systems

Co-building algorithmic management systems includes directly engaging workers, trade unions, or affected communities in the design, validation, and oversight of these technologies. Rules, particularly those applicable to digital labour platforms and platform workers, stipulate participatory development of algorithmic management systems. Participatory design not only improves the accuracy and effectiveness of the technologies by incorporating feedback and utilising worker's tacit knowledge, but it also builds trust in the technologies with those who use them. 

This site is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 947806). 

If you have any questions or feedback about this site or our research, please get in touch by emailing ai.work@law.ox.ac.uk. 

Logos for EU emblem, ERC logo, University of Oxford and Bonavero Institute of Human Rights

On this page

Our algorithmic management toolkit for employees informs individuals about the diverse rights and protections available in legal and regulatory instruments, while pointing to their gaps and limitations.