Special Interest Groups

Forum Navigation
You need to log in to create posts and topics.

Concordance as evidence in the Watson for Oncology decision-support system

Discussion: concordance as a form of evidence in medical decision-making

As a form of evidence that IBM uses to market its product, concordance is challenging from a medical decision-making perspective. We see concordance as a form of evidence problematic for a number of reasons. Let us consider these problems through the following hypothetical scenarios regarding concordance and non-concordance.

First, if Watson is piloted with a group of oncologists and there is a high level of concordance between the two, what does this high level of concordance suggest? In the best-case scenario, both the Watson platform and the oncologists have chosen treatment options, which give the best survival rate outcomes possible given the state of knowledge at a given time. Watson’s role in this decision-making process then is more to confirm what the oncologists already knew and as such does not provide any new information that can be used to treat patients.

In a second scenario, there is still a high level of concordance between the platform and oncologists. In this scenario, however, both the platform and the oncologists choose treatment options that result in poor survival outcomes, or at least outcomes that are worse than other available treatment options, but neither is “aware” of this. The fact that there is concordance between Watson and the oncologists does not provide evidence that the treatment options are the best ones; only that the two forms of expertise agree. In this scenario, the oncologists are given a false sense of security because of the high concordance level between the two. Furthermore, the developers of the platform are led to believe that because there is concordance with the oncologists, the platform must be getting it “right”, thereby exacerbating the mistake.

In a third scenario, Watson provides the best possible treatment options, but there is a low level of concordance with the oncologists. In this scenario, the oncologists chooses sub-optimal treatment options while the platform does what it is hoped to do: suggest the best possible treatment options available. In this scenario, the oncologists can choose one of two options; either discard their own expertise or discard the expertise of Watson. By following the options suggested by Watson, they end up saving more lives. By discarding Watson’s suggestions, they end up harming patients with sub-optimal treatment options. One could argue that Watson was developed with this scenario in mind, being able to provide new options that the oncologists did not know about, but which end up saving more lives than the treatment options that the doctors are currently using. For the developers of Watson, however, it is unclear what non-concordance implies; is there some evidence that it is not aware of, are the calculations wrong, or are there different values at stake in calculating treatment preferences? And for the doctors there is the crucial epistemological question of whether they have access to the relevant information, data and literature to be able to evaluate Watson’s suggestion and learn from it.

In a fourth scenario, Watson provides poor treatment options, while the options used by the doctors are optimal to saving lives. Here again, the doctors need to choose between following their own expertise or the options that the platform provides. By following their own expertise, patients are saved, by choosing what Watson suggests they end up harming patients. This scenario is similar to that of the third one in that the physicians need to evaluate the validity of their own expertise in relation to that which the platform provides. Again, for the developers, there is a challenge in understanding why there is a low level of concordance between the platform and the oncologists.

Although these scenarios are contrived and do not necessarily represent real-world situations, they none-the-less point to an inherent problem that the use of concordance as a form of evidence has in the marketing of medical decision-making platforms. Concordance does not measure patient outcomes regarding different treatment options. As mentioned, non-concordance does not imply that Watson is wrong and may indeed lead to oncologists learning something new. As a form of evidence, however, concordance says little about the outcomes of chosen treatment protocols, which would be of greater value for determining which options to choose. Nor does concordance give us any information regarding how and why physicians would or would not change their treatment decisions based on Watson’s suggestions.

Watson’s merits lie in the fact that it can cover a great deal of literature in a very short time. One could argue that in the long run, Watson could be a useful tool in comparing the long-term outcomes of different treatment protocols to see which ones may be more effective in treating different types of cancers. Such an undertaking would require, however, a better understanding of how data for cancer studies are collected and analyzed in different context, such as, for example, Denmark vs the US. There are also other important values, which the platform should account for, such as the preferences of patients as to what treatment options are preferable in different situations, which Watson does not account for (McDougall 2018).

    3AI is a not-for-profit association founded with an endeavor of creating the largest community of AI & Analytics professionals
    Email
    info@3ai.in
    Follow us on