IBM's Watson is catching up with oncologists in cancer diagnosis accuracy: study

December 12, 2016
by Thomas Dworetzky, Contributing Reporter
When it comes to diagnosing breast cancer Watson is on its way to becoming the perfect second opinion.

At least a recent study suggests that the IBM brainiac software system is catching up with human doctors in accuracy, according to a recent presentation at the San Antonio Breast Cancer Symposium in San Antonio this week.

At the meeting, researchers in India reported on findings that showed that Watson for Oncology (WFO) had shown “a high degree of concordance with the recommendations of a panel of oncologists in a double-blinded validation study,” according to a meeting release.

“Manipal Hospitals recently adopted Watson for Oncology as a tool to support our oncologists in making quality, evidence-based decisions for their patients,” the study’s lead author, Dr. S.P. Somashekhar, chairman of the Manipal Comprehensive Cancer Center of Manipal Hospitals, in Bengaluru, India, told the audience at the meeting.

He said that the study was prompted to see “how it would impact oncologists’ day-to-day practice and to assess how Watson’s recommendations compared to the decisions of our team of experts.”

The study compared the AI system's results with the determinations of the 12-15 oncologists making up the institution's multidisciplinary tumor board.

The researchers reviewed a total of 638 breast cancer patients cared for in Manipal Hospitals.

WFO analyzed the cases and came up with three recommendations – standard treatment (REC); for consideration (FC); and not recommended (NREC).

The study found that the oncologists agreed with WFO about REC and FC recommendations 90 percent of the time.

The type — and complexity — of cancer impacted the rate of agreement. For non-metastatic disease, doctors and machine agreed 80 percent of the time. When metastases occurred, however, agreement was just 45 percent.

WFO and physicians concurred in cases of triple-negative breast cancer 68 percent of the time.

WFO did not fare as well with its analyses of HER2/neu-negative cases, however, matching the doctors in just 35 percent of the cases.

“Including HER2/neu cases opens up many more treatments and variables for consideration,” Somashekhar noted, suggesting that these trickier cases increased “the demands on human thinking capacity. More complicated cases lead to more divergent opinions on the recommended treatment.”

Regarding speed to diagnoses, WFO was a clear leader. People needed about 20 minutes for a fresh case and 12 minutes for those they were more familiar with.

Watson took a mere 40 seconds to collect and review the information and come up with a diagnosis.

Despite the obvious advantage in speed and the sheer number of cases Watson can diagnose in short order – and even given its already impressive performance compared with experienced oncologists, Somashekhar stressed that the system was an adjunct, not a replacement, for doctors.

“There is always an important distinction between what can be done and what should be done based on a variety of very personal factors for the patient,” he said. “We are dealing with human beings, and the context and preferences of each individual patient, the patient-physician relationship, and human touch and empathy are very important. It’s always going to be the decision of the treating oncologist and patient to determine what is truly the best option for the patient,” he concluded.

Its real-world impact, he advised to HealthDay, is that in just a couple of minutes “it will show all of the options, and it will share the evidence," he noted, adding that this means that “then you're in a better place to make a right judgment, especially when you have too many choices and too many variables."

Chief of surgical oncology for Lenox Hill Hospital in New York City, Dr. Stephanie Bernik, concurred.

"It's probably very good for physicians who aren't working in a group," she told HealthDay. "It's almost like getting a second opinion."

Memorial Sloan Kettering Cancer Center and IBM joined forces to create WFO, which scours electronic health records with its natural language-capable AI system to come up with diagnoses and treatment options for lung, colorectal and breast cancers.

The company had launched its partnership deal with over 15 health care groups in June, according to Marketwatch.

“There is strong potential for systems like Watson to help to make radiologists more productive, diagnoses more accurate, decisions more sound and costs more manageable,” Nadim Michel Daher, a medical imaging and informatics analyst at Frost & Sullivan told Marketwatch in June.

Somashekhar has used Watson to treat breast cancer in the field before. In early August, a 37-year-old Indian programmer with a rare type of fast-spreading breast cancer was “seen” by Watson in Bengaluru, India.

“At the click of a button 15 million pages are scanned,” he recounted. “Besides my team, there is one more unbiased person (Watson) whose thinking capacity is infinite,” he said.