Michelle Wiest on Avoiding Bias, Maintaining Data Quality, And Collaboration With The FDA

With a rich background in both academia and industry, Michelle Wiest provided fascinating insight into her current role assisting with the development and marketing of early blood-based cancer diagnostics. The conversation covered a lot of ground, from a whistlestop tour of her career to how she views the process of working with regulatory authorities to move diagnostic product development projects forward.

The Interview

Guest Profile

Michelle brings 18 years of research experience in both academia and industry to the podcast. Her first entry into the industry was after graduating from UC Davis, working at a small biotech company in the Sacramento area while finishing her PhD. From there, she worked at the University of Idaho for 12 years, with part of her role running the Statistical Consulting Centre. During this time she spent a period in Australia working as a senior statistician at the Murdoch Children’s Research Institute before leaving the university to return to industry. At the start of the podcast, Michelle gives an insight into wanting to work more closely with patients as a big drive for the move.

The Highlights

Michelle’s interview gives an in-depth perspective on the practical considerations of building datasets for clinical trials and the role of machine learning to facilitate this. However, if you are looking for a quick read, our five highlights are:  

• Using Tailored Diagnostics To Identify Cancer Earlier (5:46): Michelle discusses how part of her motivation to move back into industry from academia was a desire to have a real impact on patient health. She speaks about wanting to push forward products that were going to help with tailored identification of cancer for earlier detection; “taking the signature of someone's tumor and characterizing it, and then developing a test that is going to specifically look for those mutations [using] personalized medicine, to see if their cancer has come back.”
  
  
• Randomization, Balance, and Diversity When Building Diagnostic Data Sets (14:37): When discussing the process of collecting data for a study, Michelle emphasizes the need to ensure as much randomization in the process as possible to avoid bias. The data being used may be a collection of proprietary samples, combined with purchased data from a third-party supplier. Randomization here is essential to ensure that biomarkers are correctly interpreted and avoid false conclusions as a result of other factors, for example, laboratory conditions within one laboratory causing a subset of samples to behave in a certain way. She also discusses the challenge of testing when looking to develop diagnostics for early-stage disease. If the dataset you are initially working with uses patients with late-stage disease, it is important to be careful in assuming that the results will translate into early-stage cancers as well. If accessing early-stage patients is challenging, there are mitigating steps that can be taken, such as weighting data. Here, the dev team needs to work closely with the clinical team. They need to actively collaborate to gather a more balanced training set to ensure correct conclusions are drawn.
  
  
• Maintaining Quality In Data (23:56): Having a good understanding of the history of samples and their metadata is essential to understanding why data may “behave” in a certain way and to interpreting it correctly. There are some datasets you can control in order to ensure all necessary data is in place, but this is not always the case. For the sets out of your control, in some cases you can go back and ask for more data; in other instances, there are different approaches to extrapolate the information needed. Some of these approaches are more robust than others, but the key is trying to ensure the highest quality, best targeted, and most comprehensive data for the diagnostic area being researched.
  
  
• Applying Agent-Based Modelling (29:42): When asked if Michelle feels there are any limits to how agent-based modeling can be used in development of diagnostic tests, she suggests that the best role for this approach is from an economic perspective. In her view, this approach is very good at modeling decisions that people are making and the conditions in which those decisions are made. Using the example of the Ebola epidemic in West Africa, agent-based models were useful when there were fears of this spreading to the US to get an understanding of how to control the disease. This type of use case can translate to the economic side of diagnostic tests in terms of determining uptake, or decisions clinicians may make based on the results of the tests.
  
  
• Collaboration to meet regulatory requirements (31:30): Michelle gives an overview of the process of preparing to present to regulatory bodies like the FDA. She explains the need for close collaboration between in-house regulatory specialists and FDA reviewers to design studies that meet guidelines and address specific questions. Working in a cutting-edge field, it is crucial to speak upfront with your FDA reviewers. Getting their buy-in and feedback on your study designs is incredibly valuable. As an industry, we're learning more about what works, and what doesn't work, and regulators can make suggestions based on what they have seen fail already.

Further reading: Michelle has a number of papers on a broad range of topics from her time as an associate professor at the University of Idaho. You can also read more about work at Freenome through publications listed on their website.

Continuing the Conversation

One of the most interesting aspects of the conversation was discussing her first-hand experience working with the FDA and its attitudes to machine learning.

In the past 12 months, the FDA has been vocal about how it intends to address the rise of Machine Learning (ML) within drug discovery. The regulatory authority has noted a significant increase in the number of drug and biologic application submissions using AI/ML components, with more than 100 submissions reported in 2021. Part of the response to this growing trend, was a paper for discussion in May 2023 looking at the current and potential future uses for AI/ML and the possible concerns and risks associated with these innovations.

One of the big areas for concern and part of our discussion with Michelle was the challenge of avoiding bias in data when training algorithms and how this bias can be mitigated. It also emphasizes the need for human involvement and how to embrace the technologies without increasing the risk to patients using the products of an innovative approach.

This prudent approach can also be seen in a more recent FDA announcement, which states its proposed position on laboratory-developed tests, or LDTs.

LDTs are in vitro diagnostic products (IVDs), and the FDA plans to “make explicit that IVDs are devices under the Federal Food, Drug, and Cosmetic Act, including when the manufacturer of the IVD is a laboratory. Along with this amendment, the FDA is proposing a policy under which the agency intends to provide greater oversight of LDTs, through a phaseout of its general enforcement discretion approach to LDTs.”

The motivation for tighter regulation around LDTs is to encourage responsible innovation, but it is widely acknowledged that this stance could have unwanted side effects, such as increasing the cost of developing new and potentially beneficial LDTs.

This puts more pressure on designers and developers to not waste resources during the development of their diagnostic tool, which will, inevitably, deter them from exploring more speculative paths. There is a distinct possibility that tighter regulation could stifle innovation.

However, the intention of the FDA is not to hamper innovation, but to balance it with safety. Using data effectively can play a role in achieving this balance. Having experimental processes and datasets that are balanced, following best practices, and ensuring that the methods your data team is using are structured and repeatable, results in an efficient learning process. It lowers overhead and supports FDA-compliant practices that are able to tread the line between safety and progress set out by the FDA. More and better curated data can lead to more efficient innovation.

Although we can see that AI and ML are an area of concern for the FDA from a safety perspective, when used in a sensible and well-structured way, they can further scientific progress while supporting the FDA's commitment to patient safety. If you want to explore the potential of data within your organization with experts in implementing data projects within the biotech industry, get in touch for a free analysis and consultation.

Want to listen to the full podcast? Listen here:

• Spotify
• Apple
• Other

‍