Spurt of AI health tools worries experts

Add the letters “A” and “I” to your product or company name, and investors will come flocking – that’s the running joke in tech circles. Claims of innovations leveraging artificial intelligence are flooding the market, even when some fail to explain how AI is actually being used.

While much of this can be dismissed as hype-driven enthusiasm, one sector is raising genuine concerns: MedTech. With numerous startups unveiling AI-driven products and software, experts warn that the lack of clear rulesfor clinical validation could lead to troubling consequences when doctors begin relying on them.

Many of these startups use large language models trained on datasets from diverse global regions, which could result in skewed outcomes when applied to medical technology. As more analytical and predictive tools enter the healthcare space, this creates a clear and present danger.

This is something epidemiologist Rakesh Pillai understands all too well. In 2012, this clinical epidemiology graduate from Trivandrum Medical College participated in a programme initiated by the Indian Council of Medical Research to assess the probability of childhood obesity in India.

He was working in New Delhi at that time after doing research in Biomedical Sciences at the University of Strathclyde in UK.

Skewed Data

Data from approximately 2,800 children across different parts of the country were collected and analysed using a predictive tool that measured body fat and other relevant parameters. As a reference standard, the study utilised Dexa-derived body volume data, a widely accepted benchmark obtained through X-ray machines.

The researchers discovered significant discrepancies between the Dexa data and the results produced by the predictive tool. They realised that the formulas used by the predictive tool were based on data from other societies, which differed substantially from Indian conditions.

“The difference in contextual factors was critical. Any machine developed in a different population context needs to be contextually adapted and validated locally. All the factors influencing the diagnosis must be taken into account,” said Rakesh, who is now the founder-director of Trivandrum-based startup Epifractals Private Limited.

Rakesh was a key contributor to a recent study that warned that while AI has immense potential to enhance healthcare in India, the lack of clear guidelines could create significant challenges for the country’s health sector. Moreover, it could harm the reputation of Indian-made health tools in the international market.

The study, led by Dr Denny John, Professor at the Ramaiah University of Applied Sciences, Bangalore, noted that despite India’s rapid adoption of AI-enabled medical devices, there are no specific provisions for their registration or validation.

The study also highlighted concerns beyond security and privacy, such as the lack of a trained workforce. Many healthcare professionals are not adequately equipped to use AI tools effectively. Algorithmic biases are another critical issue, as they could lead to inequities in diagnosis and treatment.

The researchers evaluated the much-praised Niramai tool, an AI-powered solution for early breast cancer detection that utilises thermal imaging and the proprietary Thermalytixtechnique. While the tool is non-invasive and provides a low-cost screening option, its efficacy and validity have been called into question. The Society of Breast Imaging India meanwhile says it does not support breast thermography as a primary or adjunctive diagnostic tool.

Reliability Concern

Conflicts of interest were noted in studies promoting Niramai, with some authors disclosing financial ties to the company, says the study. Additionally, the AI models used in the product were retrained on limited datasets, raising concerns about their reliability.

While India has taken steps to protect digital health data through initiatives like the Digital Information Security in Healthcare Act of 2018 (DISHA), it has yet to establish clear requirements for AI-integrated medical devices.

In 2018, India established Health Technology Assessment India (HTAIn) and identified institutions such as the All India Institute of Medical Sciences and the Sree Chitra Tirunal Institute for Medical Sciences and Technology as centres for conducting assessment studies. However, the lack of a unified approach has meant that no effective common mechanism is in place.

This gap has become even more critical with the emergence of AI health products that use large multi-modal models entering the market.

In MedTech, Phases 1, 2, and 3 of a clinical trial are typically conducted on a limited basis and serve as experimental exercises. It is only in Phase 3 that the trial is tested on a controlled group of volunteers. Phase 4 involves market entry, where validation occurs through real-world use, and results are followed up and further validated.

Market Evaluation

Rakesh points out that in India, market evaluation is largely carried out by IT firms hired by the product makers. This creates an opportunity for data misrepresentation, as these firms may gather data to validate the results rather than identify discrepancies.

“That approach has cast doubt on Indian validations, as many foreign countries do not accept them as credible evidence. There have even been cases where systematic reviews uncovered instances where adverse factors were deliberately ignored during Phase 3 trials,” says Rakesh.

This could become a major obstacle for Indian MedTech firms, especially as the country seeks to establish itself as a global leader in medical tool manufacturing. Given this situation, it is crucial that regulations for the integration of AI into health devices are swiftly implemented.

Rakesh believes that experts from various domains should be made available to MedTech startups even before they begin working on prototypes. Additionally, medical experts trained in data evaluation should be involved, and the participation of physicians – who are the end users of these tools – is also essential in this process.

The study he co-authored suggests that policymakers, healthcare professionals, end-users, and developers should collaborate to create a comprehensive framework. It advocates for a centralised regulatory body to oversee the assessment and implementation of digital health technology (DHT), ensuring accountability and consistency.

Need for Framework

Rakesh explains that the study participants began working on the paper two years ago when they realised India was falling behind, while countries like Britain and the US had already established well-defined policies.

“Our aim was to create a framework that could be developed with input from multidisciplinary teams, ensuring a standardised approach to the development of AI-assisted health tools,” says Rakesh.

Medical practices evolve alongside advancements in technology, and the reliance on diagnostic machines is steadily increasing. In this context, AI-assisted predictive and diagnostic tools could become standard fixtures in the near future. India must act quickly to ensure that proper guardrails are in place, as we are dealing with a technology that is evolving at a lightning speed.

All set for the Huddle Global 2024

The 2024 Huddle Global opens on November 28, and Kerala Startup Mission (KSUM) says the three-day event will aim to bridge the knowledge gap in the general public about the impact of major initiatives like Graphene research, Hydrogen Park, and the likely effects of the Vizhinjam deep-sea port. KSUM chief Anoop Ambika states that the event will focus on sectors such as agriculture, the food industry, and spacetech, along with a forum for foreign delegates to showcase Kerala’s potential.

The last edition of the event had extraordinary energy, thanks to the inspired choice of Adimalathurai Beach as the venue, which provided a unique vibe. However, the ongoing cash crunch has forced KSUM to take it back to Hotel Leela Raviz, Kovalam. As usual, belt-tightening affects areas where it is least deserved, even though an event like Huddle Global should have been prioritised for its potential to shape the state’s future. Unfortunately, the priorities of a government are often different.

Sound changes are on the way

WhatsApp began as a text-based messaging platform and later added audio messaging capabilities. Now, the social media platform plans to introduce a transcription feature that converts audio messages into text. The feature, which the company says will roll out in a few weeks, will make it easier to grasp messages in noisy environments or avoid irritating those around you by playing audio aloud. Initially, English, Portuguese, Spanish, Russian, and Hindi will be available for Android users, while iOS will offer a much wider selection, according to The Verge.

Venue for purebred coders

Using AI to generate code and video tutorials for learning programming has become the norm, but some experts caution that this approach deprives coders of in-depth knowledge. Addressing this concern, a startup called CodeCrafters has launched a platform designed for those who want to acquire coding skills the right way. Founded by Sarup Bankosta and Paul Kuruvilla, and backed by Y Combinator, CodeCrafters offers developers a hands-on platform to build projects like a BitTorrent client, Git, Redis, Docker, and more. It features challenges across different stages, each with assigned difficulty levels and detailed explanations of how the projects function, reports BestofAI. The startup has garnered support from notable figures, including Instagram co-founder Mike Krieger and Dropbox co-founder Arash Ferdowsi.

Swiss church nails this one

A church in Switzerland now offers visitors the chance to talk directly to Jesus – or at least an AI-powered version of Him. The chapel, located in the city of Lucerne, has installed an artificial intelligence-driven Jesus capable of conversing in 100 different languages. After training the AI program with theological texts, visitors were invited to pose questions to a long-haired image of Jesus, projected through a latticework screen, reports The Guardian. The report states that over 1,000 people – including Muslims and tourists from as far as China and Vietnam – have tried the experience in just two months. About two-thirds of them described it as a spiritual encounter. Wonder when will Indian godmen add AI into their halos?