Contact tracing is broken in SF. Which means we’re losing the fight. Which doesn’t bode well for the rest of the US.
Read this post if you don’t want more social distancing in your future. It is about what is going wrong with SF’s response to COVID-19. Ultimately, I will focus on the contact tracing loop. I will be talking about this loop both from my excavation of SF’s datasets and also from the personal experience of a friend who just got diagnosed with COVID-19. If you keep reading there’s a bit of a bombshell at the end, but before I get to the ugly stuff, let me talk about the good.
San Francisco has a lot going for it in this pandemic.
San Francisco’s response is informed by science.
San Francisco was scarred by the AIDS crisis and learned some hard lessons which we are applying to COVID-19.
San Francisco has a community spirit and a populace willing to sacrifice for their community.
San Francisco has a woman leader which seems to help. (NY Times - Women leaders)
SF’s dashboard looked good in early June.
San Francisco’s dashboard metrics have all looked quite good especially compared to much of the rest of the country and especially back in mid June when the surge started. (One caution in looking at these metrics is that the x-axis aren’t the same on all the graphs, so they don’t represent the same period of time. This is true even for the two contact tracing graphs. You need to hover over points to see the data and line the graphs up.)
There are a number of metrics like PPE available and hospital beds available on the dashboard. These are reactive metrics. These metrics indicate whether we can deal with shit hitting the fan. However, they don’t indicate whether we’ve turned the fan around to avoid the shit in the first place.
Right now, I’m not interested in these reactive metrics. Instead, I’m interested in the metrics that should indicate active control of COVID-19. These include
percentage of tests that are positive,
total numbers of tests run, and
contact tracing.
All these active control metrics looked really good in early June. Apart from the new case count, most of them still look relatively good even now.
So why is SF facing a crisis?
Given all that San Francisco has going for it, and given that all the metrics above looked really good back in early June, why in the world is our city facing down a surge of COVID-19? When the pandemic hit the US back in March, I was pretty sure that my kids would be out of school through the rest of the spring. But in early June, looking at San Francisco’s dashboard, I had a reasonable expectation that my kids would be back in school in the fall. Perhaps school would look different but they would be physically back in school. To explain why reality diverged from my expectation you need to know what the contact tracing cycle looks like.
The Contact Tracing Cycle
Here is the contact tracing cycle. The purpose of this cycle is to break the chain of transmission without resorting to a lockdown. Phoenix Data Project- Insight not health vs economy and Phoenix Data Project- Basic TTSI
Please let me know whether this diagram is self explanatory and I’ll add more explanation if it isn’t.
At its best, contact tracing is a way of controlling the virus while our community remains open. If you want to get back to life as normal (or at least closer to normal), than this cycle needs to function well. If this contact tracing cycle breaks, then the only way we have to control the spread of COVID-19 is to be more socially distant and locked down.
When is Contact Tracing Effective?
This cycle is only effective when it is
fast,
accurate,
pervasive, and
thorough.
Contact tracing has to be fast enough that it can stop someone from spreading COVID-19. This entire cycle ideally needs to be completed in a day. If the cycle takes any longer than three days, it probably isn’t effective.
Contact tracing needs to be accurate. Any tests being returned with false negative results will allow COVID-19 to spread.
Contact tracing needs to be pervasive. Everyone who tests positive for COVID-19 needs to be reached.
Contact tracing needs to be thorough. If the contact tracer doesn’t reach all of a patient’s contacts, then those neglected contacts could easily spread COVID-19.
Is SF’s contact tracing fast? No. Delays are at least 6 to 9 days not 1 to 3 days
Dataset excavation - There is no direct dashboard data on this metric, but I did some excavation and I’ve developed my own metric from San Francisco’s dataset. Phoenix Data Project- dataset dig. This dataset indicates that it is taking 6 to 9 days to go from a COVID-19 test to having some sort of government contact with an individual who tested positive. Santa Clara’s dashboard indicates the same thing. The turnaround time just for tests in Santa Clara has grown from 1.5 to 4.5 days. If just getting test results back is taking more than 3 days, then this loop is definitely broken.
Personal experience - I have a friend who found out that she was positive for COVID-19 ten days after taking a test at Kaiser in San Francisco. She was briefly contacted by a contact tracer twelve days after taking the test. This is so far beyond the bounds of what is acceptable that face palm. (I’d love a good gif here.)
Is SF’s contact tracing accurate? Anecdotal information says perhaps not.
Unfortunately, we really have no metrics indicating whether our testing is accurate. Anecdotally, this friend who just tested positive thinks that she may herself have gotten it from a friend who had many of the classic COVID-19 symptoms, shortness of breath and fatigue, but this same friend twice tested negative. The first negative test of the potential carrier might not have been too surprising, given that she was not symptomatic at the time. The second negative test was taken when the potential carrier had symptoms, yet the test still came back negative. Perhaps, and this is just pure speculation, the test swabbing wasn’t done effectively enough. This potential carrier friend was one of the most significant contacts that my friend had had, but with the two negative tests, we are left to wonder if there another route of transmission.
Unfortunately, there was no thorough investigation done around my friend’s infection and so we really don’t know how she caught COVID-19. This anecdotal account of a potential false negative is not enough to say that accuracy is actually a problem. This potential example is enough to illustrate that if tests aren’t accurate then we’ve got a real problem on our hands.
Is SF contact tracing pervasive? No
Dataset excavation - See the following post !!!Link for a detailed explanation of this data, but suffice it to say that if San Francisco’s contact tracing effort was making a dent in this virus then the new community counts numbers in this graph would be continually dropping. Unfortunately the community transmission rate is higher now than at the start of June.
Furthermore, if contact tracing was working even during a surge of cases then the ratio of the number of community spread cases to the numbers of known contact cases should be dropping. Neither of these curves is lower than they were in June.
Personal experience - My friend’s test results came back way too late and no one is following up with her contacts. The long test delay means that contact tracing on my friend has become incredibly difficult. COVID-19 has used her and moved on to whomever else its going to infect. That doesn’t mean that we should give up. If we had an awesome contact tracing system, we could play catchup with the particular transmission chain that is my friend’s case. Someone would talk to my friends contacts and interview them and get them tested and then talk to their contacts interview them and get them tested and repeat until you’ve caught everyone. Places like South Korea are doing contact tracing to this extent.
Right now, I am the one playing contact tracer. I am encouraging my friend to contact her friends (which she had done.) I am the one coaching them on how to work the system and hopefully get results faster than 10 days. I’m hoping that my knowledge of the testing system Phoenix Data Project- A rant about testing will help them get tested quickly and fast through the lab associated with the city, Color, rather than through Kaiser. As of the time that I write this it looks like that gambit worked and no-one around her has caught COVID-19. So that is a bullet dodged.
Is SF contact tracing thorough? Graph looks good but what is the standard?
The contact tracing metric shown in the adjacent graph tracks a measure of the thoroughness San Francisco’s contact tracing effort. Specifically it indicates how many of a person’s contacts the contact tracer has been able to reach. The problem with this metric is that we don’t know the baseline. We don’t know the standards that San Francisco’s contact tracers are trying to achieve. Are San Francisco’s contact tracers only looking for the most obvious transmission or are they able to try to find transmissions that might not be obvious.
If you are only looking for the obvious ways that COVID-19 is spreading, than you won’t find the currently hidden ways that COVID-19 is spreading. Does the contact tracer only ask for the names of people you have shared an indoor space for longer than 15 minutes? Are they only limiting themselves to people who share indoor spaces for long periods of time that weren’t wearing masks? If that is the case, that would be an unacceptably high bar. Is this metric for only contacts that can be named? What about unknown people who might have been exposed?
As an example what if the positive patient worked in a store for an entire day? Would the contact tracer follow up with everyone who visited that store? What if the patient was eating dinner outdoors upwind of some other tables? Would the contact tracer get the credit card data from the restaurant and followup with the patrons? If a contact tracer doesn’t followup in these sorts of scenarios that are currently presumed to be low risk than how do we know that these sorts of activities are actually low risk?
Because we aren’t getting any contact tracing reports coming out of the SF Department of Public Health, I strongly suspect that San Francisco is looking for its house keys / COVID-19 cases under the light and not really searching thoroughly for all types of COVID-19 transmission.
There has been a promise on the bottom of this graph to provide the dataset associated with it. Unfortunately this promise has been up on the city’s dashboard for weeks now.
Testing allocation and FEMA Bombshell.
My friend asked the Kaiser rep who contacted her why the results took ten days. The representative told her that FEMA requisitioned testing materials from Kaiser which brought test processing to a halt. If this is true, this is the worst sort of self defeating behavior on the part of the federal government.
How do firefighters tackle a fire? They first set a perimeter to try to keep the fire from burning new areas. With the perimeter protected they turn the firefighting resources toward putting out the core inferno. In the case of COVID-19 you don’t abandon the places where COVID-19 is out of control, but at the same time you don’t waste testing resources on those areas. For out of control spread you use the big hammer of a lockdown to treat these regions. If we try to use testing resources to control COVID-19 in a saturated area like Texas, then we are wasting testing resources. This is doubly true if the testing resources are being used in regions that have poor contact tracing. Precious testing resources should be used wisely to contain the perimeter of COVID-19. With the perimeter contained those testing resources can be turned inward toward the more saturated regions. (And by the way, I can’t believe that six months into the pandemic testing I am calling testing resources “precious.”)
This allocation of testing resources was clearly outlined back in April and May in the pandemic testing plan Safra Center- Pandemic Resilience Roadmap Supplement. At the time, they were advocating for New York to remain under lockdown for longer so that testing resources could be freed up for the rest of the country which at the time was green, yellow, and orange. I encourage you to at least skim through the report but suffice it to say that on page 15 in its advice to local policy makers the report says
If I’m in a red zone, I should keep only essential services open … Recognizing that I will be the #3 priority of my state, I need to expect that stay-at- home advisories will last longer for me than for my peers in yellow and green zones. My zone will require a huge volume of tests to clear the disease and I recognize that we can collectively get there faster, if we first protect all green zones and help all yellow zones achieve clearance and reach green, before we return to the project of fully clearing the disease from red zones. That said, I should focus within my zone on directing all available resources to vulnerable populations in the first instance.
I also know that I will ultimately get the lion’s share of total testing resources, in exchange for waiting in line just a little longer. …
At this point, our entire country is best served by using test results in the yellow and orange areas not saturated with COVID so that we can secure the perimeter of this fire.
By taking testing resources away from San Francisco, FEMA is shooting itself and the entire country in the foot.
The contact tracing cycle isn’t robust and has been overwhelmed.
In a pandemic it pays to be pessimistic. San Francisco did not build its contact tracing program up sufficiently enough to handle the modest surge in cases that occurred June till now. This also illustrates the power of exponential growth. If we open the door to COVID-19 a crack, exponential growth can quickly turn that small crack into a yawning chasm. From looking at how the metrics are headed, the contact tracers in San Francisco can’t keep up. COVID-19 is spreading and is not being sufficiently checked.
If the contact tracing cycle isn’t working effectively, then the only way we have of controlling this virus is social distancing and lockdown. Masks help with social distancing but without effective contact tracing we don’t know exactly how effective they are. Without proper contact tracing, all the advice and directives that our government is giving us is just rough guesswork. Outdoor dining, hair stylists outdoors, shopping with masks indoors: we know that all of these activities are safer than how we used to do them, but are they truly safe, that’s anyone’s guess.
I know I don’t want more social distancing anymore than any of you, but with broken systems, it is the only really effective way of controlling the virus.
I have bashed SF contact tracing effort, but the contact tracers themselves deserve love and appreciation.
I significantly under estimated the challenges of good contact tracing. I read various reopening proposals back in April and didn’t give the specifics of contact tracing a lot of thought. Contact tracing is much harder than its simple description. Convincing someone to trust you with the names of their friends, family, and their entire itinerary is really hard. To be successful, the contact tracers have to be a part of the community that they are trying to work for. At a bare minimum, you need contact tracers that can speak all sorts of languages, but you really need contact tracers who are culturally aware of the communities that they’re working in. I didn’t appreciate this initially, but I think that given the AIDS crisis, San Francisco’s Public Health Department did get this. I’m guessing that San Francisco has a diversity of contact tracers that can work in a wide range of communities.
As much as I criticized San Francisco’s contact tracing graphs, our contact tracing numbers look much better than most anywhere else in the Bay Area and/or the country. Unfortunately, this is exactly the reason why I say if contact tracing isn’t working in San Francisco right now, then the rest of the country is screwed. If your region is on the cusp of considering additional reopening measures, make sure that your testing and contact tracing loop is solid and can withstand the strain of many additional cases.
What the city can do.
The city needs more contact tracers and needs to make the process of contact tracing more transparent. I shouldn’t need to try to read the tea leaves of San Francisco’s datasets to determine whether contact tracing is working. I want proof that it is. Keep the metrics extremely clear and transparent. As one example San Francisco’s dataset should tell me how many new people did SF’s contact tracers successfully get to get to quarantine on any given day. The dataset should tell me how long it took for those people in temporary quarantine to get a test result back and the number of people in the temporary quarantine who tested positive and needed to quarantine for longer. These are much more straight-forward numbers. They aren’t percentages of something that we don’t know what it is.
Some of San Francisco’s DPH resources should be devoted to epidemiological studies. I want to see some case studies in the spread of COVID-19 not just from abroad but from our very own city. Such studies would be relevant and motivating to me and everyone in this city.
What we can do.
Put yourself through the thought experiment of what you would tell a contact tracer if they phoned you and told you you had tested positive for COVID-19. Would your conversation with them be short or long? How long would your list of specific individuals be? How many people would you have been indoors with without masks? How many businesses have you been inside?
As you are considering the amount of social isolation that you can put up with, think about making the contact tracer’s life easier. Ideally our social isolation would be more like we were doing back at the middle of May start of June. If you have bubbled up with other people or other families since March and April, make sure that the walls of those bubbles are sound. Don’t have leaky bubbles in which people from your bubble are also bubbled up with other people too. I anecdotally I hear about bubbles that include girl friends or boy friends who then themselves are part of different bubbles.
Stay on your toes. Let the city metrics drive the extent of your socialization. Don’t let COVID-19 use you to spread.