Are they really worth the hype? A 9-day minute-by-minute sleep tracker
by Jina Yoon — firstname.lastname@example.org
Three years ago, sleep specialist and physician Dr. Christopher Winter posted an article titled “Personal Sleep Monitors: Do They Work?”. Today, with over 500 options from more than 15 brands available, I was curious to conduct a similar study three years later on how the data from personal sleep tracking devices compare with each other. I wore 10 of some of the most popular sleep trackers (including one research-standard actigraph as a control) across 10 nights and then graphed the data against each other. If you’re only interested in the results, feel free to skip to the end, but here’s a rundown of the devices used, and what data I was looking at. I’m also including links to guides and source code if you’d like to make a similar chart yourself!
In general, sleep happens in five phases, but these are often categorized by consumer-level devices into two to three stages: light sleep, deep sleep, and REM sleep. Most wearables and apps in the market do not claim to recognize REM sleep since it is virtually impossible to detect without much more advanced sleep studies using polysomnography, a type of test that records your brain waves, blood oxygen levels, heart rate, breathing, as well as eye and leg movements during the study. It requires the placement of various electrodes on your head and is usually performed in a clinic for just 1 night.
Because polysomnography is very expensive and inconvenient, these personal activity trackers offer simpler, more affordable, and longer-term options for people like you and me. They typically work by using accelerometer sensors (how much you move around), noise sensors (how much sound you make), or some combination of both, but because they are less complex, they are not nearly as accurate as the devices used in clinical sleep studies.
Here’s a quick rundown of each of the devices I used in my experiment. *I originally wanted to include the BASIS Peak that "won" Dr. Winter's original experiment, but decided against it after the device was recently recalled due to safety concerns.
|AMI MotionLogger||$795, without software||Wristband||Accelerometer, Temperature, Heart Rate Monitor||Awake, Asleep||Research-standard actigraph. Our “control”/”gold standard”. We chose to compute sleep using the most commonly known Sadeh algorithm.|
|Fitbit Alta||$130||Wristband||Accelerometer||Awake, Restless, Asleep||One of the top-selling activity tracker brands on Amazon.|
|Jawbone UP (discontinued)||$34||Wristband||Accelerometer||Awake, Light Sleep, Deep Sleep||Another top-selling brand.|
|Microsoft Band (discontinued)||$125||Wristband||Accelerometer, Heart Rate Monitor, Skin Temperature||Awake, Light Sleep, Deep Sleep||We are not sure which sensors are used for sleep stage computation because their algorithms have not been publicly released.|
|Microsoft Band 2||$224||Wristband||Accelerometer, Heart Rate Monitor, Skin Temperature||Awake, Light Sleep, Deep Sleep||We are not sure which sensors are used for sleep stage computation because their algorithms have not been publicly released.|
|Sleep Cycle iOS||Free ($9.99/yr Premium)||iPhone App||Microphone||Awake, Sleep, Deep Sleep||Was once named the top paid app in the App Store. Northcube AB does not clarify what thresholds are used for classifying sleep stages, so our chart presents our best estimates based on the given images from the app.|
|Sleep as Android||Free ($3.99 Unlock)||Android App||Acclerometer, Microphone||Awake, Light Sleep, Deep Sleep, REM Sleep||The top downloaded sleep app in the Play Store. Claims to track REM data, unclear how.|
|SleepCoacher iOS||Free||iPhone App||Accelerometer||Awake, Asleep||A specially developed app by Brown University HCI that offers personalized sleep recommendations based on your habits and experiments. You can sign up to participate in our beta launch here!|
|SleepCoacher Android||Free||Android App||Accelerometer||Awake, Asleep||A specially developed app by Brown University HCI that offers personalized sleep recommendations based on your habits and experiments. You can sign up to participate in our beta launch here!|
|Hello Sense (without Voice)||$149||Tabletop Sense + Clip-on Pill||Accelerometer, Light Sensor, Microphone||Awake, Light, Deep||A unique system that tracks not only your motion, but also your sleep environment during the night. The data output from their app is a bit rough, so some of the data on our visualization is also estimated.|
Gathering the data was simple in theory: I just had to use all ten devices simultaneously for ten nights. So, I wore all five wristbands on the same arm (Fitbit Alta, Jawbone UP, Microsoft Band, Microsoft Band 2, AMI MotionLogger), three phone apps placed side-by-side on my bed (Sleep as Android, SleepCoacher iOS, SleepCoacher Android), one phone app placed on my nightstand (Sleep Cycle iOS), one device clipped to my pillow (Hello Sense Sleep Pill) and one product that required two sensors clipped onto my pillow and placed on my nightstand (Hello Sense). It was, as expected, not very comfortable -- which might be why my sleep patterns seem quite restless over the 10 nights!
Extracting the data from the devices proved to be a bit of a challenge. If you’re interested in a detailed explanation of how I did this, you can check out my data extraction guide. It’ll walk you through how to unlock your minute-by-minute (or stage-by-stage) data from your device. This isn't always as straightforward as you would expect -- in fact, we were of the first to unlock data from the Hello Sense, and you can find the guide for that here (thanks Jeff)!
As for the data visualization, I used this d3 chart. I tweaked the margins and the legend just a tiny bit for this project, so if you'd like to check out my version, you can find it here (they are virtually the same).
Below is the visualization of the data I retrieved. There are a few things to note that you can find in the limitations section later. You can hover over sections of each night to see timestamps, and click through the links to different nights.
While the data I gathered was better than random, the results from my experiment should still be taken with a grain of salt. My experience with these 10 devices may differ from yours, and each device has its strengths and weaknesses. If we consider only the results from the AMI MotionLogger as the “gold standard” for our accuracy study, the Fitbit Alta seems to be the most accurate among the other 9 in terms of sleep versus awake data. However, each device varies in price, convenience, design, popularity, and plenty of other factors more that you might consider differently based on your lifestyle.
If you’re interested in learning more about sleep tracking and would like to sign up for SleepCoacher, the Android and iPhone app being developed by researchers at Brown University, check out our interest form here! By signing up, you’ll receive an invitation to be part of the first group to receive personalized recommendations through our app. SleepCoacher works by guiding you through small experiments to find the optimal sleep environment and schedule for you. The more you use it, the better the recommendations get. It’s been presented at research conferences, and our algorithms have been adjusted to Our goal is to provide convenient, affordable, and accurate sleep improvement recommendations to help you figure out what works best for your lifestyle and habits.
In the end, we found that most devices were similar to each other if they were based on the same type of sensor (accelerometer vs. noise) and the device type (phone vs. wristband vs. other). Ultimately, sleep is best tracked through polysomnography, but these devices offer much more accessible and convenient options for casual users to track their activity and learn more about their habits during both the night and day. These devices offer some powerful data tracking tools, but it’s best to let the experts analyze your sleep if you have any chronic conditions. Our findings tell that these consumer-level sleep reports should be taken with a grain of salt, but regardless we’re happy to see more and more people investing in improving their sleep -- after all, there’s nothing better than a good night’s rest!