Improving AI Performance for People of Color: Diagnosing Melanoma & Other Skin Cancers

Access to experts for skin checks – namely, dermatologists – is a challenge in the US and across the globe. Many people experience undue stress and anxiety when they identify a concerning mole but can’t get an appointment to see a dermatologist for many days, weeks, or in some cases months. Primary care doctors and urgent care clinics serve on the front lines for patients presenting with skin lesions of concern, however they often lack the specialized training and tools of a dermatologist.

Scientists, biotechnology companies, and device makers are developing new tools to aid individuals, primary care doctors, and dermatologists in the timely detection of melanoma and skins cancers. Tools such as apps for smart phones, tele-dermatology, and cataloging of skin images over time to detect changes in moles or lesions (e.g., such as total body photography) are only a few examples of the many tools in development intended to help ease the burden, identify skin lesions of concern, and create a ‘triage’ system for patients that need to get into dermatology practices more quickly versus those that can safely and confidently wait for their scheduled visit. The majority of these tools also utilize machine learning and artificial intelligence (AI) to help distinguish benign versus concerning lesions.

The Challenges of Existing AI-Data Sets for Melanoma Detection

The problem with the current state-of-the-art AI and image-based algorithms is that they have been developed using images of moles from light skinned (white) individuals. As a result, existing AI tools are not sensitive enough in people with darker skin. Dr. Albert Chiou, Clinical Associate Professor of Dermatology at Stanford University, the junior faculty member on the L’Oréal Dermatological Beauty Brands-MRA Team Science Award, and his colleagues are working to fix these shortcomings. 

Dr. Chiou, along with team leaders Drs. Roberto Novoa, Susan Swetter, James Zou, Justin Ko, Roxana Daneshjou and others are funded by the L’Oréal-MRA Team Science Award to refine the AI algorithm they developed and published in 2017 by incorporating images from People of Color. Learn more about People of Color and melanoma.

“Many dermatologists and other melanoma experts have been warning about potential biases that can arise when applying AI algorithms to skin cancer diagnostic tasks, particularly if the AI algorithms were not trained with data reflective of the diversity of skin tones present in the population in which the algorithms will be deployed,” said Dr. Chiou. “This is one of the first in depth experimental studies to evaluate – and try to quantify – the potential biases that arise when state-of-the-art algorithms are applied primarily to patients with darker skin tones. This is part of a broader effort to more rigorously understand the potential implications, both positive and negative, of using AI as a tool for early melanoma and skin cancer detection in a clinical setting.”

To overcome limitations with existing AI algorithms, Dr. Chiou and collaborators first needed to create the Diverse Dermatology Images (DDI) dataset; a pathologically confirmed, benchmarked dataset with diverse skin tones published today August 12, 2022 in the journal Science Advances. 

Key Findings from the L’Oréal-MRA Team Science Award:

• Prior to the Diverse Dermatology Images dataset, there were no publicly available benchmark datasets that included biopsy-proven malignancies in skin of color;
• The DDI dataset can be used to fine-tune previous algorithms to improve performance across diverse datasets; and
• The DDI dataset will provide a useful evaluation benchmark and will allow scientists to improve the way dermatology AI models perform across skin tones.

“It is notable that our study did indeed identify a significant performance gap in algorithm performance in classifying skin lesions between patients with darker versus lighter skin tones, even when accounting for disease rarity,” noted Dr. Chiou. 

“What is equally important, however, is that our machine learning collaborators, Dr. Roxanna Daneshjou and Professor James Zou, were able to demonstrate that there are potential solutions to close this performance gap, such as utilizing our dataset focused on diverse skin tones to fine-tune existing algorithms. We have publicly released the DDI dataset coinciding with our publication, and hope others will contribute similarly to help efforts to ensure more equitable AI performance across diverse patient populations.”

Dr. Roxana Daneshjou noted: “we have seen that datasets previously used for developing AI in dermatology are not representative of all people. With this study, we have shown why this is such a huge issue – algorithms that have not been trained on diverse data perform worse on Brown and Black skin.”

As the global leader in dermocosmetics, L’Oréal is committed to the science of skin health and has been a long-term partner with the Melanoma Research Alliance to help advance crucial research in skin cancer prevention and detection using sophisticated AI technology. 

The topic of potential use of devices to detect of skin cancer is timely, with the US Food and Drug Administration (FDA) convening a two-day panel of experts last month to discuss the state of such devices, including the issues around equity and access for people of all skin tones and colors. 

MRA-funded investigator Dr. Veronica Rotemberg (Memorial Sloan Kettering Cancer Center), a co-author with Dr. Chiou on creation of the DDI dataset and a panelist at the FDA workshop previously noted, “well-designed prospective studies are still needed to understand how consumers and physicians will react to [these emerging] new tools.” She urged that experts have to define what prospective information is needed to be collected and tested in clinical trials before dermatologists will feel comfortable saying these emerging tools could be used in either a layperson’s or primary care physician’s hands.

“Our team is working hard to find ways to increase the representation of all skin tones in future AI datasets in order to address these types of biases.  It is becoming clear, however, that any long-term solution will likely not arise from a single institution,” Dr. Chiou further noted.  

“There are burgeoning consortium efforts to continue to share de-identified, annotated clinical images, which we fully support.  Further, through L’Oréal and MRA’s support, we intend to continue to rigorously evaluate the performance of AI algorithms in more real-world scenarios and settings through the MRA Team-Science Award-funded study under Dr. Roberto Novoa. Our team is optimistic AI can eventually become a valuable tool in the effort for early melanoma detection, though we feel it is important to fully understand its potential risks and biases as well.”

While access to dermatology services remains a challenge today, Drs. Chiou, Daneshjou, Novoa and colleagues' research brings us one step closer to the day when new technology can help all people, regardless of skin tone, get accurate and timely diagnoses of concerning skin lesions.