We investigated whether digital contact tracing actually worked in the US

In the spring of 2020, the first versions of covid-19 exposure notification systems were released to the public. These systems promised to slow the disease’s spread by providing automated warnings to people who came into contact with the virus. Now, over a year later, residents in over 50 countries—including half of US states—can opt into these systems.

But the big question remains: how well did this technology work? Some studies suggest answers, but despite such wide rollout, it’s difficult to evaluate whether exposure notifications were actually able to stall covid-19 spread. This is especially true in the US, where many states launched their own apps—a decentralized approach that reflects America’s fragmented pandemic response.

In an attempt to learn more about how this technology fared in the US, MIT Technology Review reached out to every state public health department that launched a digital contact tracing system and examined app reviews left by anonymous Americans. We asked two questions: who is actually using this technology, and how do people feel about it?

The end result of this analysis paints a picture of unexplored potential. Many of the country’s exposure notification apps are underutilized, misunderstood, and not well-trusted—and yet this technology may yet come into its own as a public health tool for future disease outbreaks.

How the technology works

Exposure notifications were first put forward as a complement to traditional contact tracing. Under the traditional manual approach, investigators looking for people who may have been infected ask patients to trace their whereabouts and activities through phone calls and interviews. The new technology promised to scale to cover entire populations automatically rather than just small disease clusters— a distinct advantage for tracking a fast-spreading disease.

You might remember the friend you met for lunch, for example, but not the stranger you stood next to in line for 15 minutes at the grocery store. An exposure notification system does the remembering for you, anonymously using Bluetooth to keep a log of nearby phones and alerting you if one of those phones is associated with a positive test result.

The first wave of this system was designed by cooperatives of developers, most of whom ended up collaborating with Apple and Google to create a uniform standard. The Apple-Google system prioritized privacy for users, anonymizing their data, and did not track users locations. With the backing of the world’s two most dominant phone platforms, this system is the one that’s been most widely adopted, and is used by the vast majority of US states. 

The effectiveness of these systems has been notoriously hard to evaluate. Studies are just now starting to come out about apps in the UK and Switzerland, for example. In the US, evaluation is made even harder by the fact that every state is basically doing its own thing. But our analysis does have a few takeaways: 

• US systems were launched relatively late in the pandemic—when the country’s fall/winter surge was mostly already in progress
• The technology has not been widely adopted, though some states are faring better than others
• A lack of public trust in new technology—coupled with a lack of resources in the public health agencies peddling that technology—hampered both adoption rates and how people used the systems

Who’s using this tech

We tracked exposure notification apps that had been rolled out in 25 states and the District of Columbia. Virginia was the first state to make the technology publicly available to its residents in August 2020, while others are still only getting started now. Massachusetts began testing its app with a pilot in two cities in April 2021, while South Carolina is currently running a pilot program at Clemson University. The state actually started work on its system back in May 2020—but legislators barred the public health department from any digital contact tracing work last summer due to privacy concerns, holding back development.

Even in the states where such apps are available, not everybody can use them. Exposure notifications are only available for smartphone users; and about 15% of Americans don’t have a smartphone, according to Pew Research Center. Still, over half of the US population can now get plugged in. Whether they choose to join those systems is another matter.

As the vast majority of states do not publicly report user data, we reached out to state public health departments directly to ask how many people had opted into the technology.

Twenty-four states and DC shared user estimates, showing that, by early May, a total 36.7 million Americans have opted in to the notifications. Hawaii has the highest share of its population covered, at about 46%. In four more states, more than 30% of residents opted in: Connecticut, Maryland, Colorado, and Nevada. Seven more states have over 15% of their populations covered.

That proportion is important: modeling studies have determined that if roughly 15% of a population opts into the system, it could significantly reduce a community’s covid case numbers, hospitalizations, and deaths. By this metric, 13 states—which together represent about one-third of the US population—have seen some degree of protection thanks to exposure notifications.

The remaining 11 states with exposure notification apps fail to meet this benchmark for success. Of those 11, three states have under 5% of their populations covered: Arizona, North Dakota, and Wyoming. South Dakota, the one state which did not respond to a press request, shares use of the Care19 Diary app with the low-activation states of North Dakota and Wyoming.

Comparing states isn’t perfect, though, because there are no federal standards guiding how states collect or report the data—and some may make very different choices to others. For example, while DC reports an “exposure notification opt-in” number on its Reopening Metrics page, this number is actually higher than its residential population. A representative of DC Health explained that the opt-in number includes tourists and people who work in DC, even if they reside elsewhere. For our purposes, we looked at DC’s activation rate as a share of the surrounding metropolitan area’s population (including parts of nearby Maryland, Virginia, and West Virginia).

Another reason these rates are hard to measure: Several of the states with higher usage rates benefit from a major upgrade that Apple and Google released in September: Exposure Notification Express, or ENX. This framework made it much faster for states to spin up apps, and it also invited millions of iPhone users to avoid downloading anything at all. They could activate the notifications simply by flipping a switch in their phone settings.

ENX activation is much more convenient, and experts say it may seem safer than downloading a new app. It has seriously boosted activation rates for states that use it. Hawaii, for example, saw its users more than double from February to May while rolling out ENX. 

The express system does mean we have less precise user data, though. States aren’t able to track ENX activations directly, and instead need to rely on Apple for their numbers.

Beyond the numbers

Even when a lot of residents have downloaded an app or turned that switch in their iPhone settings, the system still needs to be properly used in order to make a difference in covid cases. So we tried to understand how people were using the systems, too. 

A recent study found that Americans were hesitant to trust digital contact tracing technology. However, this finding was based on surveys conducted before most states even launched their apps. As a proxy for public attitudes towards the US state apps, MIT Technology Review scraped and analyzed app reviews from the Google Play store. We only looked at Google Play reviews (from Android users) to get the most current and consistent data. (Most iPhone users can now turn on notifications without downloading an app.)

Looking at app reviews isn’t a perfect system. Users who chose to review their state’s app are not a representative sample of the EN-activating population—instead, they are those users who want to share strong opinions about the technology.

Still, here’s what we found: 

• Most of the state apps have average ratings between 3 and 4.
• Michigan has the lowest score, at 2.6.
• D.C, California, New York, Delaware, and Massachusetts have the highest scores, over 4. 

Many 1-star reviewers appeared to misunderstand how their state’s app works, didn’t trust in the technology, or were unable to understand how the app fit into the broader public health system. This indicates that, for many Americans, the app wasn’t doing its job even though it was technically in use.

Lessons from negative reviews

Poor reviews provide a glimpse into common issues and misconceptions that the digital contact tracing system faced.

Small glitches made a big difference.
Over and over, reviewers stated that they got tripped up by needing an activation code. To help protect privacy, when you test positive for covid you don’t input your name or other identifying details into the app: instead, you enter a string of numbers that your public health department gives you. Some reviewers state that they don’t know where to get an activation code after testing positive, or that they ran into error messages. We’ve heard from developers in other countries about this issue.

Some US states and other countries have streamlined the process by automating how a code gets sent, but in many cases, users must wait for a contact tracer to call them. This waiting period can decrease trust in the technology, and it significantly slows down digital contact tracing.

“Trust” isn’t just about the app itself. It’s broader than that.
Many app reviewers also mistrust new technology, the government, or both. A Pew Research Center survey conducted in July 2020 found that 41% of Americans would likely not speak with a public health official on the phone or via text message, and 27% said they would not be comfortable sharing the names of recent contacts—both key elements of the contact tracing process. 

Digital contact tracing faces similar challenges. Some reviewers felt so strongly about protecting their privacy that they came to their state app’s pages in order to boast about their refusal to download this technology. Many echoed the sentiments of this reviewer from Pennsylvania: “Open access to my wifi, GPS, and Bluetooth? Creepy. No thanks, Harrisburg.”

Low usage creates a downward spiral of mistrust.
One crucial aspect of digital contact tracing is that you need participation for it to work—at least 15% of the community, but preferably much higher. When people aren’t participating, the chance of getting a match is lower—even if covid levels are high—and so the system likely won’t send out alerts to those small number of people who do have exposure notifications activated.

A few reviews went as far as to beg the other residents of their states to opt into exposure notifications, reminding fellow reviewers that higher usage leads to higher effectiveness in a tone that seemed more reminiscent of a Facebook argument than an app store.