Home
- What Is The Bias Glossay?
- What Is The Bias Glossary Ultimate Goal?
- What Problem Does The Bias Glossary Solve?
- What Design Principles Underlie The Bias Glossary
- How Does The Bias Glossary Accomplish Its Goals?
- The Bias Glossary initial categories
The Bias Glossary is envisioned as an innovative, collaborative platform designed to systematically document and update biases associated with various healthcare datasets. Unlike existing frameworks that provide static snapshots of data characteristics, the Bias Glossary aims to establish a dynamic, community-driven repository where biases are continually identified, reported, and revised. This living document will serve as a comprehensive reference point for researchers, clinicians, and AI developers, allowing them to understand not only the general attributes of a dataset but also the specific biases it may harbor. By integrating contributions from a diverse community of stakeholders, the Bias Glossary will evolve with the expanding landscape of medical data and emerging insights into biases, ensuring that the information remains current and relevant.
The platform features detailed documentation on each dataset, including its origin, collection process, and any amendments made to its associated biases, thereby providing transparency and traceability. In essence, the Bias Glossary will act as both a repository and a forum, fostering a collaborative environment for sharing knowledge and best practices in addressing dataset biases.
Furthermore, the Bias Glossary will serve as a valuable educational tool for all stakeholders involved in healthcare AI. It will provide insights into the multifaceted nature of bias, ranging from data collection methods to societal and demographic influences—thus promoting a more critical approach to dataset analysis and usage. The collaborative nature of the platform will also encourage a community-driven approach to bias identification and resolution, enhancing collective understanding and fostering a culture of accountability and ethical responsibility in the development of AI technologies.
The ultimate goal of the Bias Glossary is to enhance the integrity and efficacy of AI applications in healthcare by providing a resource that helps mitigate the risk of bias from the outset. By equipping stakeholders with detailed, up-to-date information on dataset biases, the Glossary aids in the development of more accurate and fair AI algorithms. It also supports the broader objective of ethical AI use, aligning with international efforts to ensure that AI systems are safe, secure, and trustworthy. As AI continues to permeate healthcare, the Bias Glossary will be indispensable in promoting a more equitable healthcare system, where decisions supported by AI are as unbiased and inclusive as possible.
The issue of clinical bias is a well-known topic in the medical field, underscored by extensive research that explores its manifestations within a societal framework marked by inequity, prejudice, and discrimination4. These biases, often rooted in unconscious cognitive shortcuts, have tangible and detrimental effects on patient care, leading to disparities in diagnosis, treatment, and outcomes across diverse populations5. For instance, when compared to white Americans, pain management in black Americans is systematically worse, due to false beliefs about biological differences between these two groups6. Moreover, racial and ethnic minority patients are less likely to be screened for diabetic retinopathy, even though they are more likely to have poorer glycemic control7.
Training AI on clinical data derived from a world embedded with such biases risks not merely replicating, but also amplifying and perpetuating them8. Notwithstanding, it could even reconfigure new ones which would remain elusive due to the inherently opaque nature of some AI algorithms9. Already, evidence of bias in AI spans across a variety of applications, from sex-based disparities in algorithms predicting cardiovascular risks10, to ethnic disparities in the detection of skin-related diseases such as melanoma11. Notably, the issue of algorithmic bias extends beyond historically marginalized groups, potentially affecting anyone whose profile deviates from the predominant characteristics of the training datasets, whether in terms of skin color, gender, age, disease characteristics or even the hospital’s zip code9.
Similarly, medical devices, whether or not integrated with AI technology, are susceptible to perpetuating the same types of biases, introducing risks that disproportionately affect marginalized populations. The pulse oximeter, for instance, has been documented to exhibit reduced accuracy in individuals with darker skin tones12, a discrepancy that can lead to poorer clinical outcomes for these populations13. Moreover, measurements made by commonly used medical devices such as thermometers, sphygmomanometer, encephalography, and electrocardiography machines have also been shown to potentially carry biases14.
In light of this, there has been growing awareness on the need for transparency and accountability for AI medical applications. With it, the investigation of bias in AI algorithms and medical devices is a rapidly advancing field, with many stakeholders increasingly cognizant of the potential risks biases pose. Recently, both the European Parliament, through the EU-AI Act15, and the White House, via the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence16, have initiated measures to mitigate bias in AI algorithms. Nevertheless, much of the effort in this domain has been directed towards post hoc analysis—examining models for bias after their development and deployment17. We see this approach as costly, inefficient, and unable to promote systemic change.
To get ahead of this issue, some researchers have put forward commendable efforts aimed at enhancing the understanding of datasets’ collection processes, origins, development intents, recommended uses, and ethical considerations. These initiatives seek to establish standardized means for researchers and developers to quickly access critical information about datasets intended for training medical devices, algorithms, or conducting epidemiological research. Notable among these efforts are Data Cards18, Data Statements19, Datasheet for Datasets20, Model Cards21, AI-Usage Cards22, and the Dataset Nutritional Label17. Each of these proposals contributes with valuable frameworks for documenting various aspects of datasets and models, facilitating a more responsible and informed use of data in AI development.
However, most existing initiatives primarily address general dataset characteristics and usage guidelines without delving deeper into the specific biases that datasets may contain. These existing frameworks, while foundational, are not equipped to dynamically track or update bias-related issues as they evolve or as new evidence comes to light. Consequently, they fall short of providing the nuanced understanding required to preemptively recognize and mitigate biases specific to each dataset. This oversight underscores the critical need for a tool that is specifically built to systematically index, catalog, describe, and provide a potential approach to mitigating biases. To that end, we propose the development of a model of data transparency which we call the “Bias Glossary”.
Similar to open-source projects like Linux and Python, which utilize a variety of software development practices and principles that contribute to their success and widespread adoption, the Bias Glossary also follows several key practices commonly used in these and other open-source projects. These include:
-
Version Ccontrol: the glossary embraces a transparent process to manage changes to our codebase using GitHub—a platform widely recognized for its robust version control and collaborative features. This allows multiple community members, including researchers, clinicians, AI developers, and other stakeholders, to work on the project simultaneously, tracking changes and managing versions efficiently;
-
Code Review and Pull Requests: changes to the code are submitted as pull requests, which are then reviewed by other community members. Pull requests are essentially proposals for revisions or additions to the glossary, which are made publicly available for review. Each submission is rigorously evaluated through a peer review process managed by the project maintainers—carefully selected based on their expertise and dedication to promoting unbiased AI in healthcare. This peer review process ensures that the code meets our project's standards for quality and functionality;
-
Documentation: good documentation is crucial for open-source projects. It not only helps new users understand how to use our tool but also assists new contributors in understanding the codebase and the project’s architecture.
This methodology not only fosters an ongoing, dynamic update process but also ensures that the glossary maintains a high level of academic rigor. The open-source model promotes inclusivity and collective responsibility, essential for addressing the multifaceted nature of biases in healthcare datasets. This approach mirrors the principles of openness, peer review, and community engagement that are hallmarks of both academic rigor and the key practices of open-source projects.
Of note, we do not aim to provide the final structure of the Bias Glossary, nor do we claim that the following categories are exhaustive. Instead, we aim to suggest one potential structure as a starting point to populate the Bias Glossary, consisting of four initial categories, namely: Participants not missing at random, Validity of data points, Data not missing at random, and Miscellaneous.
This category captures bias stemming from absence or underrepresentation of specific patient groups within the dataset, encompassing not only demographic factors but also clinical conditions, socioeconomic statuses, and accessibility variables which may skew research outcomes and subsequent clinical applications. The Bias Glossary under this category aims to illuminate the hidden disparities by documenting the absence of certain groups due to various selection biases or data collection constraints. This awareness is critical as it allows researchers and clinicians to critically evaluate the dataset and its applicability to the general population, ensuring that medical interventions developed from AI models do not inadvertently perpetuate health inequities.
The second category examines the integrity of data collected, focusing on potential biases introduced through the use of various medical devices and data recording methodologies. This category is pivotal as it questions the foundational accuracy of the dataset itself —whether the data points reflect true patient states or are distorted by technological and procedural variances. By cataloging these potential sources of error, the Bias Glossary promotes a more nuanced understanding of the data, which is essential for developing reliable AI models.
This category investigates the uneven data collection practices that may occur across various patient groups due to factors such as race, socioeconomic status, geographical location, and other demographic or contextual influences. It underscores the necessity to meticulously examine and question the consistency and fairness of data collection protocols and their execution among diverse patient populations. This detailed scrutiny is crucial for identifying and understanding the systemic errors and biases that could detrimentally impact clinical research and the training of AI algorithms.
The fourth category encompasses a broad range of biases that do not neatly fit into the other categories but are nonetheless crucial for understanding and using the dataset responsibly. These might include biases related to the geographic location of data collection, time-period specific healthcare practices, or administrative biases in how data are recorded and processed. This section will be populated with examples that highlight less obvious but impactful biases affecting data interpretation and application in AI systems.
Learn more about How-to-Update-a-Bias-Glossary.