Review: The new Apple Watch says my lungs may be sick. Or perfect. It can't decide.

Sometimes the new Apple Watch Series 6 reports my lungs and heart are the picture of health, pumping blood that's 100% saturated with oxygen.

At other times, it reports my blood oxygen is so low I might be suffering from emphysema. (I am not.)

The watch can't decide. This much is clear: Don't buy one of these $400 devices in the hopes of monitoring your lung health.

An Apple oxygen check a day will not keep the doctor away, at least not yet. The way consumer tech companies are marketing health capabilities is getting ahead of what their gadgets can actually, reliably do. That's a dangerous trend, and it jeopardizes the potential positive effect that collecting body data could have on our health.

And it's particularly deceptive at a time when many people are looking to health monitors for any clue that they may have COVID-19, the illness caused by the novel coronavirus.

For the past week, I've been wearing a smartwatch on each wrist, all day and all night long. On the right I have the Apple Watch Series 6, and on the left I wear the new $330 Fitbit Sense, which went on sale this week.

There are many reasons that people buy wearable gadgets. I wear an Apple Watch for fitness motivation and to receive phone notifications, and an Oura Ring to track my sleep. But this fall's smartwatch upgrades from Apple and Fitbit are all about health. Apple's slogan reads: “The future of health is on your wrist.”

These watches also read heart rate and rhythm, but I'm focusing this review on the headline addition to the Apple watch and the Fitbit: an oximeter, which measures the oxygen in your blood. Doctors are increasingly treating oxygenation as a vital sign (alongside pulse and temperature) because it can help reveal aspects of conditions including sleep apnea, pulmonary embolism and COVID-19. That certainly sounds helpful to have on your wrist.

And that's what Apple vice president for health Sumbul Ahmad Desai implied at Apple's prerecorded launch event. “Adding blood oxygen brings another valuable health measurement to users. Blood oxygen and pulse oximetry are terms that we've heard a lot about during the covid pandemic,” she said.

But you start to get a different picture when you read what both companies say in their disclaimers. Neither device is approved by the Food and Drug Administration.

The tiny type at the bottom of Apple's website says its blood oxygen app is “not intended for medical use” and is “only designed for general fitness and wellness purposes.” Fitbit's small print says its blood-oxygen app is “not intended to diagnose or treat any medical condition” and is useful to “help you manage your well-being and keep track of your information.”

There are important differences in the blood oxygen data that Apple and Fitbit report. But in my experience, neither company's measurement serves much purpose at all. You should know what you're buying, because it might do more harm than good.

- - -

To understand my frustrating Apple Watch readings, I called pulmonologists who haven't had a chance to test the new watches but understand the science. When doctors test blood oxygen, they often use sensors on fingers called pulse oximeters. These devices shine light through the skin and nail to detect the color of the blood as a measure of how much oxygen is there. They produce a measure called SpO2; most healthy people range between 95% and 100%.

The finger oximeters used by doctors are approved by the Food and Drug Administration. To compare my smartwatch results, I bought a finger oximeter for $60 from Medline Industries that is FDA approved and reports an error rate of plus or minus two percentage points.

Unlike finger pulse oximeters, these two smartwatches try to read your blood oxygen from your wrist. And they're conspicuously silent about accuracy.

Apple's new watch has lights on the bottom to generate signals that are reflected back from the blood in your wrist and read by sensors. An app lets you do spot checks any time and also runs on its own while you sleep. You have to hold really, really still for 15 seconds to get a reading.

The first time I tried this on the Apple Watch 6, it said my oxygen level was 88% - shockingly low, given that I am in good health and wasn't wheezing. Five minutes later, I tested again and it said my SpO2 was 95%. I kept trying it and kept getting different readings - and, frequently, an “unsuccessful measurement” error message.

I told Apple about my experience, and it sent me a new watch. My first measurement on my second Apple Watch 6 reported my SpO2 as 100%. If these readings were accurate, my lungs were having a really wild Wednesday.

Over several days of comparing my second Apple Watch's measurements to my FDA-approved finger oximeter, Apple's readings most often differ by two or three percentage points - though they've also sometimes exactly matched, and sometimes been as much as seven percentage points lower.

Is it just me? Skin, fat and blood vessels do vary. Apple would not comment on the error rate of its sensor, but spokeswoman Amy Bessette said it “has been rigorously tested across a wide spectrum of users and across all skin tones.” (When I tested the Apple Watch on a colleague whose skin is darker than mine, the results were also off from the finger pulse oximeter, but less wildly so.)

Bessette also said, “For a small percentage of users, various factors may make it difficult to get a blood oxygen measurement including motion, watch placement on the wrist, skin temperature and skin perfusion, and the blood oxygen app provides dynamic feedback to help users get the best reading possible.”

The company sent me additional Apple watch straps - eight in total - to wear while testing its second watch. This year, Apple is selling a new kind of stretchy band that is called the Solo Loop and comes in a variety of sizes. Going down one size (to a model that leaves a slight imprint on my wrist) did eliminate some but not all of the “unsuccessful measurement” error messages.

With the Fitbit, I've had less-erratic results, but the device also provides a lot less information. You can't ask the Sense to run spot checks. Instead, it measures your SpO2 while you sleep and provides a nightly average.

My oxygen level, Fitbit reports, is typically in the range of 95% to 97%. That sounds believable, though I can't compare it to results from my finger pulse oximeter because I'm not awake to turn it on.

In an interview, Fitbit's director of research, Conor Heneghan, said the company decided the overnight view was a more reliable piece of information. “It's a pretty hard technical problem to measure SpO2 on the wrist,” he said. Unlike fingers, which have many blood vessels near the surface that offer a strong signal, the wrist is prone to obstructions and poor readings.

“You move a little bit, or even just you are a little bit colder than normal, you can get a very weak signal,” Heneghan said. “We've gone after long-term averaging, so that way, when we take overnight measurements, we can comfortably exclude the periods when we feel that signal is too noisy or weak to be reliable.”

Heneghan still wouldn't disclose the Fitbit's exact error rate. But he said it beats the range set by an international standards organization. That's not much to brag about: It would allow someone with a true SpO2 reading of 95% to be told they're at 91%.

He was forthcoming on the testing Fitbit did, such as working with a lab at the University of California at San Francisco to test the device on volunteers, including people with different skin tones. “We tried to overrepresent darker-skin toned people in our testing to make sure that it's not skewed toward a particular tone,” he said.

- - -

Let's be clear: These companies are marketing a device with medical functions while winking and insisting they're not medical functions. OK, so then what else, exactly, are we supposed to use oxygen apps for?

Fitness? You can't use these sensors while you work out. Just the slightest bit of movement - even breathing too heavily - sends my Apple Watch into error mode. And neither Apple nor Fitbit makes any effort to explain how your SpO2 levels might be linked to your workouts. (SpO2 is different from another oxygen indicator called VO2 Max, which measures how your body uses oxygen while you exercise.)

That leaves us with the industry's term “wellness.” So, are we supposed to get together with friends over drinks and talk about O2 stats? “Hey, bud, my hemoglobin works better than yours!”

Whatever the fine print might say, some people are going to treat these as medical devices - and that's a concern.

“Pulse oximeters can tell you in a trending situation if your oxygen is in the normal range,” said Albert Rizzo, the chief medical officer for the American Lung Association. But it's not necessarily a leading indicator of problems, including COVID-19. “Nobody should be waiting for their pulse-ox to go down before calling their doctor,” he said.

There could be consequences if consumers actually believe the hype about these devices. “I agree with you that it is a dangerous trend for technology companies to release medical devices that don't meet FDA standards and claim that they are not medical devices,” said Brian Clark, a pulmonologist and professor at the Yale University School of Medicine.

The most common negative consequence is likely to be people calling their doctors too often because of false low readings. “But the more concerning and potentially dangerous scenario is when the devices provide false reassurance and people don't seek health care when they really need it,” Clark said.

Apple was more upfront in 2018 when it added an electrocardiogram, or ECG, app to its watch. It did get FDA clearance (not quite the same as “approval”) for its app, and worked with researchers to publish studies on its accuracy. But still, there's fine print: the Apple Watch's irregular-rhythm notification is not intended for use by “those who have been previously diagnosed with atrial fibrillation (AFib).”

Fitbit said an ECG app it added to the Sense this year also received FDA clearance. Why not do the same for the oximeter? “If we were to make a claim, like we could detect sleep apnea, we would definitely go through the regulatory process and be very clear on our messaging and very clear on the limitations,” said Fitbit's Heneghan.

A release-with-disclaimers approach could leave consumers without guardrails as more body sensors come to market. To the Sense, Fitbit also added a skin temperature sensor and an electrodermal activity sensor - similar to what's in a polygraph - that it says “may indicate your body's response to stress.” Neither of those sensors has been FDA cleared.

Questions about accuracy also interfere with the work of academics combing through the body data from smartwatches to see if it can be used to detect disease. This summer, I wrote about promising early results from academics using heart rate and temperature data from the Oura Ring and Fitbit to predict the onset of COVID-19 symptoms.

Several of those researchers told me they were excited by the addition of blood-oxygen data - but there's not enough information about its validity. “We have toys, and we have things that are used for clinical purposes. And it really needs to be a clear distinction,” said Duke University's Jessilyn Dunn, an assistant professor of biomedical engineering who is helping to lead a study called Covidentify.

It should not be acceptable for giant tech companies to market devices that take readings of our bodies without disclosing how those devices were tested and what their error ranges might be.

I believe that collecting accurate data about our bodies can help advance our health. But the key word here is “accurate.”