A research group that has been widely cited by national health experts and government officials for its COVID-19 models has twice projected enormous increases in COVID-19 deaths in Maine.

But in both cases, the Institute for Health Metrics and Evaluation at the University of Washington published new data within a matter of days that sharply reduced its projected death counts.

The institute published projections last week that Maine could surpass 1,000 COVID-19 deaths by the fall – a 10-fold increase over current deaths – but revised that figure to less than 200 deaths just five days later. And in March, the institute published a projection of more than 3,000 deaths in Maine by August, only to lower that figure to 334 deaths a few days later, after making adjustments to its data.

The wildly different death projections over the course of less than a week raises questions about the value of the modeling and spotlights the potential perils of relying on any one model to anticipate how the coronavirus could spread across the U.S.

Maine’s top public health epidemiologist, Dr. Nirav Shah, said he still lists the institute among the infectious disease modeling groups that he checks regularly as part of the Maine Center for Disease Control and Prevention’s own internal modeling exercises.

But the Maine CDC director added: “It has diminished as a website that I check on a daily basis.”

The institute is among dozens of academic and research groups trying to model how this new, somewhat unpredictable virus will spread across the globe. But IHME’s projections have been used by federal and state public health officials as they make decisions impacting the daily lives of nearly all Americans, as well as the economic health of the country.

Dr. Deborah Birx, the White House’s coronavirus response coordinator, has cited the IHME model several times, as have numerous governors. The institute’s projections, both for the nation and individual states, are often quoted in news articles in a wide array of national publications.

On June 5, IHME released updated state-by-state and national forecasts that projected Maine could see 614 deaths by Sept. 21 – six times higher than the current total of 101 deaths. A forecast from the institute in early April, by comparison, had projected Maine that could experience 115 COVID-related deaths by Aug. 1.

On June 10, the institute’s projection for Maine jumped to 1,076 deaths. But in a June 15 update, IHME researchers slashed the projected number of deaths to 185 by Oct. 1.

Asked about the wild fluctuations over a period of just 10 days, an IHME spokeswoman attributed the swings to changes in Maine’s virus transmission rate. Also referred to as the reproduction rate or the R-naught, a transmission rate of 1 means that every person with COVID-19 would be expected to infect one other person, while a value of 3 means every person would infect three others.

The higher the number creeps above 1, even by decimal points, the faster the situation gets to a point where the virus is increasing exponentially to the point where hospitals and health care systems are overwhelmed.

“In the earlier model, the modeling showed the transmission rate (R naught) going above 1, which leads to fast growth,” Amelia Apfel wrote in an email. “With the additional data in the latest update, we aren’t seeing that increase in transmission rate. The (institute’s) models are pretty sensitive to this type of change.”

Institute staff did not respond to requests for additional information on the data underlying the projections.

Shah, the Maine CDC director, said he saw the dramatic swings in IHME’s projections but attributed it to the institute’s data sources and methodology, as well as Maine’s relatively small number of COVID-19 cases.

Maine did see an increase in its transmission rate several weeks ago, largely because of an outbreak at the Cape Memory Care nursing home in Cape Elizabeth. Over the course of a few days, Cape Memory went from zero cases to more than 60 among residents and staff as the Maine CDC helped test everyone at the facility.

Ultimately, Cape Memory Care had 84 cases with six deaths, to date. But Shah said models such as those run by IHME do not distinguish between isolated and contained outbreaks at facilities and a surge in cases in the community.

“I kind of attributed them to statistical noise,” Shah said Wednesday of the institute’s surging death projections. “What happens in Maine is what I call ‘the law of small numbers.'”

Shah said that Maine has very few cases compared to many other states, with his agency reporting 2,836 cases and 102 deaths as of Wednesday compared to a nationwide total of 2.1 million cases and 117,000 deaths. So when Maine reports 30 new confirmed or probable infections one day but 36 new cases the next day, Shah said, that equates to a 20 percent jump.

Shah said IHME relies largely on historical data, in this case, how COVID-19 spread, faded and may have surged again in other countries and more recently in the U.S. The institute said its most recent models also include such factors as testing rates, mobility data, mask use and population densities.

But while Shah said such models are useful and informative, comparing them to an almanac for weather projections, he has come to prefer more “dynamic” models that he compares to the supercomputers used by meteorologists to make weather forecasts. Those disease models take into account more things happening on the ground and, in some cases, individual policies at the state level.

“But we don’t use models to predict the future,” Shah said. “We use them to project scenarios that might happen and then respond accordingly.”

IHME’s projections for Maine now fall within the range of more than a dozen other modeling organizations tracked by the U.S. Centers for Disease Control and Prevention. Those models – which include projections from researchers at IHME, Johns Hopkins University, the Massachusetts Institute of Technology, UCLA, the University of Massachusetts and other institutions – project Maine will experience between 103 and 151 deaths by July, with an “ensemble” average of between 115 and 123 depending on the confidence interval.

Shah stressed that modeling was a particularly important exercise early on as the state tried to gauge hospital capacity and critical care needs on a range of COVID-19 scenarios. But more than three months later, Maine has so far avoided uncontrolled transmission overwhelming the state’s health care system thanks, in large part, to physical distancing and other restrictions.

Maine CDC epidemiologists still consult outside models, including IHME’s projections, as they try to prepare for future scenarios. But the “prime directives” that influence decisions – such as whether to lift or impose restrictions on businesses and people – include metrics like the daily and rolling averages of new cases and deaths, hospitalization trends, testing rates and the percent of tests that come back positive.

Experts said, however, that models are valuable in infectious disease tracking even if they are inherently imprecise.

“I think models get a bad wrap because they are not perfectly, numerically precise, but that is just the nature of dealing with complex systems,” said Chris Moore, an assistant professor of biology at Colby College.

That imprecision factor becomes even greater, Moore added, when you consider that the progression of COVID-19 and other infectious diseases is often dependent on unpredictable human behavior.

Moore, who taught courses last semester on the evolution of infectious disease and ecological modeling, said weather forecasts are often fairly precise in the short term and can generally predict, based on history, trends over the longer-term. The middle-range is more problematic.

Instead, Moore liked one of his student’s comparisons of infectious disease modeling to using a smartphone for driving directions. Your phone’s GPS should be able to predict with some precision when you will arrive in, say, nearby Fairfield from Waterville. But your ETA in Boston from Waterville depends on traffic, accidents and other factors.

“When you are watching your phone and traveling, it is dynamically updating and taking into account all of these variables,” Moore said.

Mathematical modeling expert Raj Saha said one of his major criticisms of COVID-19 modeling is that organizations need to be clearer about the underlying assumptions and data that are used to build the projections. For instance, the public should be able to clearly see – and understand from non-scientific language – whether a model’s projections take into account policies such as stay-at-home orders and other factors when they see the results.

That’s important because Saha, an interdisciplinary lecturer in environmental geophysics at Bates College, said people could change their behaviors based on projections, with potentially dire consequences during an outbreak of  highly infectious disease.

“People can look at low projections and assume things are OK, but they are OK because of the lockdown measures,” Saha said.

Saha said he would rather see disease projections offer several scenarios based on easily understandable factors. As examples, Saha said one scenario could include a continuation of “lockdown” measures and physical distancing, another scenario where those measures are eased, and one with little or no restrictions.

“Without knowing those assumptions, any projections, regardless of how sophisticated the models are, can be interpreted in unintended ways,” Saha said. “We are all used to seeing weather projections – for example the path of a hurricane which always comes with error zones. But how a public reacts to the projected paths of a hurricane is not going to affect the actual path of the hurricane. The same cannot be said here in the case of disease projections.”

Copy the Story Link

« Previous

On this date in Maine history: June 18

Next »

As fraudulent jobless claims pass 50,000, state sets job search deadlines for claimants