An (almost) fully automated exporting of HealthKit data

Published: 18 July 2020
Posted in:  

The following is part of a series of posts called "Building a household metrics dashboard".

This series details the various steps I took setting up a household dashboard to give the family better insights in a number of different areas such as finances and health, as well as a few other useful metrics and information.

The next data source I wanted to implement in my on-going dashboard project was my Health and Screen Time data from Apple. This turned into a significant rabbit hole and while I managed to put together a solution, it is much less than ideal due to the way Apple imposes boundaries on what you can do with your device.

As an aside, my wife and I settled on the Apple ecosystem a number of years ago and there are still days where I regret this decision. Unlike other prominent companies, I feel Apple play hard ball when it comes to vendor lock in strategies. Got an Android and want to switch to Apple, no problem, Android have facilities to migrate. That’s the way it should be in any industry. Are you with Apple and want to switch to a different platform? Good luck!

I’ll save this discussion for a different post, but suffice to say in the long term, industry players in any vertical that try to retain customers through implementing high switching costs always face eventual attrification issues due to diluting goodwill over many years. Anyhow…

Pulling your HealthKit data in any type of automated fashion from Apple just isn’t possible and Apple stop any attempts to do this. Screen Time data isn’t even an option, the only way of accessing this is via the Screen Time app UI which is very disappointing.

The method I’m using involves the creation of a simple cordova app via a few simple commands and then loading it on your device as a development app. This doesn’t require Apple to review the app so you can pretty much do what you need to do with the internal HealthKit API.

Obviously, if you’re familiar with putting apps together in Objective C or Swift, it’s likely you can do something much more elegant (let me know if you do!).

To be able to add a development app on your device though, you will unfortunately need a Apple developer license if you don’t have access to a existing development team. This does cost about $99USD per year and yes, you need to pay Apple for the ability to run your own code on your own device.

The code I’ll explain in a moment only exports HKCategoryType and HKQuantityType sample classes and if you’re interest, you can find more information about the different classes of data available here). If you have a device such as a Apple Watch, the amount of data contained across these two classes is quite extensive and great timeseries data for building dashboards and analytics, which is my goal. The cordova plugin that I’ve used DOES NOT have the ability to access work out routes which is a bit of a bummer, but you can however access these via the standard download on your phone if you really need them (they come as a GPX file per workout). I’m keen to try and flex my Objective C and/or Swift muscles in future and maybe add this to the plugin, but time is short for me these days.

The app as I’ve built immediately starts requesting all data from the last 72 hours and sends the results to a backend. After completing the last 72 hours, it waits for one second, and then forcefully quits the app. If the user taps the screen at any stage, this forceful exit is cancelled and the user can select a custom number of days to run a back up for (e.g. If you have been offline for a week, or when setting up initially and you want to download the last 5 years).

The 3 day back up lasts only a few seconds so it’s not too much of a hassle having to do this once per day. The issue is remembering to open the app for it to do it’s backup. I’ve found that by setting an automation in the Shortcuts app to open the HealthKit backup app at a particular time each day, you will either get a modal asking if you want to open the app (if you are using your phone at the selected time), or you will get a notification if you phone is locked at the given time. In the case of being locked, it’s then simply a case of clicking on the notification that will be waiting for you when you come back to your phone and waiting a few seconds for the app to do it’s thing. In either case, it’s a one tap interaction.

You will also need to create a backend for the app to communicate with, but I’ll cover this off when we get to it. I’ve stuck with the idea of having the backend service running on my local home network as opposed to having a cloud based solution just to limit the exposure of the backend as it holds some pretty sensitive information.

Ok, lets dive in and build the app first.

Building the exporter app

NOTE: I haven’t put the following into a repository as cordova projects tend to become deprecated

and broken very quickly due to the breadth of underlying API’s which are constantly evolving. In the given case, running a handful of commands to create an up to date app is much easier than cloning an old project and trying to solve issues.

The initial things do is to make cordova available on your system. I’ll assume that you have NodeJS installed at this point (if not, it’s a simple process to get installed).

Install cordova globally using the following:

$ npm install -g cordova

Once installed, create a project with the following command. Change the <BUNDLE_IDENTIFIER> here to something meaningful, in my case I used me.ianbelcher.healthkitexporter. It doesn’t matter too much what you choose, but keep with the reverse domain name structure to avoid any unforeseen issues.

$ cordova create health-kit-exporter <BUNDLE_IDENTIFIER> HealthKitExporter

The next step is to install the required plugins. This is a simple case of the following.

$ cd health-kit-exporter/

$ cordova plugin add com.telerik.plugins.healthkit --variable HEALTH_READ_PERMISSION='App needs read access' --variable HEALTH_WRITE_PERMISSION='App needs write access'

$ cordova plugin add cordova-plugin-exit

Once the plugins are installed, add the ios platform to the project with the following command:

$ cordova platform add ios

At this point, you should also check that all the required dependencies for cordova are installed. When I ran this, I found that my CocoaPods version was an older incompatible version so I needed to update and that the Xcode version that I had selected was the command line tools. You may need to update or install some other dependencies here as well.

Running the requirements command will output something like the following:

$ cordova requirements
Requirements check results for ios:
Apple macOS: installed darwin
Xcode: not installed
Error: xcodebuild: Command failed with exit code 1 Error output:
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
ios-deploy: installed 1.10.0-beta.3
CocoaPods: not installed
Error: Cordova needs CocoaPods version 1.8.0 or greater, you have version 1.2.1. Please install version 1.8.0 or greater from https://cocoapods.org/
Some of requirements check failed

For updating or installing CocoaPods, it was simply a case of using gem to update:

$ sudo gem install cocoapods

To fix the Xcode, I already had the full version downloaded, but had the command line version selected. To switch I needed to run the following. At this point, you may need to download the entire Xcode package if you haven’t already.

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

Following this, running requirements again returned no errors.

$ cordova requirements
Requirements check results for ios:
Apple macOS: installed darwin
Xcode: installed 12.0.1
ios-deploy: installed 1.10.0-beta.3
CocoaPods: installed 1.9.3

At this point, the project is a ‘Hello world’ cordova app that can be run. To test, use the run command and this will start the base cordova app in a local simulator. It has been a while for me so I don’t remember too well, but you may need to download and install a simulator within Xcode for this to happen, or it may already have one installed saving another download.

The following command needs to be run at this stage without an actual phone connected to run on a simulator at this point. If a device is connected, cordova will try to run the application on that device and it most likely fail.

$ cordova run ios

Once you’ve tested that this works, you can try installing the base app on your phone. This is where things can get difficult for those without Xcode and app signing experience, it can even be difficult with those with plenty of experience as well.

To create the Xcode project, use the build command.

$ cordova build ios

This will create an Xcode project directory in the platforms/ios directory. Open this directory in Xcode.

You’ll need your phone to be added as a device on your Apple Developer account in order to use it, and you’ll also need Xcode to be aware of your developer account and have all the correct certificates. When setting this up, there can be many things that go wrong which require a lot of digging for solutions, so good luck! Hopefully it’s quick and easy and you get to the point of being able to select your connected device in the selector in the top header of Xcode.

Selecting an attached device

Clicking the Play button to the left of this selector should build the app in Xcode, install it on your phone and then run it. Prior to doing this the first time, you will likely need to also select the signing team under the Signing and Capabilities tab in Xcode. This can be difficult to find the first time using Xcode, but you can get to it by clicking the folder icon in the top of the left side navigation, clicking on the top item (HealthKitExporter) and then clicking the Signing and Capabilities tab in the center panel.

At this point, the next step is add the exporter code. To keep things simple, I’ve constrained this to the one html file to make it a simple copy paste situation.

Replace the contents of the www/index.html file with the following code. You will need to read through and adjust it to your given requirements, but the main point is that it will get all the data for the given period and send it to the backend and we’ll look at next.

After replacing the index.html code, the app needs to be built again via the previous build command and then run within Xcode again to update the app.

<!DOCTYPE html>
<html>
<!-- 
  Written for iOS 11+ which supports const, arrow functions, spreads, fetch and async/awaits.
  
  This app does a quick query for most HKCategoryType and HKQuantityType values from health kit
  from the last seven days automatically, sends the results to a backend server for storage,
  waits for a second and then closes the app. The user can also interrupt the closing action by
  tapping the screen. This will allow them to run a backup for a custom number of days in the past.

  This intention for this app is to be run once daily via an iOS shortcut, with the app only 
  staying open for a number of seconds. The use of a shortcut means that a reminder will be shown
  on the users notifications, requiring them to click once on the reminder to open the app, with 
  the app closing quickly after being opened.

  The cordova plugin that this app is based off doesn't implement the required API's to allow for 
  you to make HKWorkoutRouteQuery queries, so it can't get the actual geographical route. This 
  shouldn't be too hard to implement by editing the plugin objective c code if you have the time.

  The amount of information available from the implemented API is quite extensive though, and 
  allows for a breadth of different approaches for analysis. This app from originally put together
  quickly to serve a Grafana implementation so similar systems should find a number of different
  ways of displaying the data.

  For reference, the following units can be used on the unit constants below
  This is from https://developer.apple.com/documentation/healthkit/hkquantitytypeidentifier
  and units come from https://developer.apple.com/documentation/healthkit/hkunit
  with unit strings defined at https://developer.apple.com/documentation/healthkit/hkunit/1615733-unitfromstring?language=objc
  
  const MASS_UNITS = ['g', 'oz', 'lb', 'st']; // gram, ounce, pound, stone
  const LENGTH_UNITS = ['m', 'in', 'ft', 'yd', 'mi']; // meter, inch, foot, yard, mile
  const TIME_UNITS = ['s', 'min', 'hour', 'day']; // second, minute, hour, day
  const ENERGY_UNITS = ['J', 'cal', 'Cal', 'kcal']; // joule, small calorie, large calorie, kilocalorie
  const TEMPERATURE_UNITS = ['K', 'degC', 'degF']; // kelvin, celsius, fahrenheit
  const VOLUME_UNITS = ['l', 'fl_oz_us', 'fl_oz_imp', 'cup_us', 'cup_imp', 'pt_us', 'pt_imp']; // litre, us fluid ounce, imperial fluid ounce, us cup, imperial cup, us pint, imperial pint

  Prefixes can take the following for for reference as well
  const prefixes = [
    'p', // pico 0.000000000001
    'n', // nano 0.000000001
    'mc', // micro 0.000001
    'm', // milli 0.001
    'c', // centi 0.01
    'd', // deci 0.1
    'da', // deca 10
    'h', // hecto 100
    'k', // kilo 1000
    'M', // mega 1000000
    'G', // giga 1000000000
    'T', // tera 1000000000000
  ];
 -->

<head>
  <meta charset="utf-8" />
  <meta name="format-detection" content="telephone=no" />
  <meta name="viewport"
    content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width, height=device-height, target-densitydpi=device-dpi" />
  <meta name="msapplication-tap-highlight" content="no" />
  <title>HealthKit Exporter</title>
</head>

<body>
  <div>
    <label for="days">Number of days to run backup for</label>
    <input id="days" name="days" type="number" min="1" step="1" value="3" />
    <button onclick="runBackup()">Run Backup</button>
    <button onclick="cordova.plugins.exit()">Close</button>
    <pre id="progress" style="white-space: pre-line;"></pre>
    <pre id="log"></pre>
  </div>
  <script type="text/javascript" src="cordova.js"></script>
  <script type="text/javascript">
  /* global window, document, fetch, cordova */
  // Configuration variables
  const BAR_LENGTH = 10;
  const QUERY_LIMIT = 1000;
  const MASS_UNIT = 'g';
  const LENGTH_UNIT = 'm';
  const TIME_UNIT = 's';
  const ENERGY_UNIT = 'J';
  const TEMPERATURE_UNIT = 'degC';
  const VOLUME_UNIT = 'l';
  const SERVER_ADDRESS = 'http://10.0.0.200:5000';

  let willExit = true;

  const SAMPLE_TYPES = [
    ['HKCategoryTypeIdentifierAppleStandHour', 'AppleStandHour'], // A category sample type indicating whether the user has stood for at least one minute during the sample.
    ['HKCategoryTypeIdentifierCervicalMucusQuality', 'CervicalMucusQuality'], // A category sample type for recording the quality of the user’s cervical mucus.
    ['HKCategoryTypeIdentifierMenstrualFlow', 'MenstrualFlow'], // A category sample type for recording menstrual cycles.
    ['HKCategoryTypeIdentifierIntermenstrualBleeding', 'IntermenstrualBleeding'], // A category sample type for recording spotting outside the normal menstruation period.
    ['HKCategoryTypeIdentifierSexualActivity', 'SexualActivity'], // A category sample type for recording sexual activity.
    ['HKCategoryTypeIdentifierOvulationTestResult', 'OvulationTestResult'], // A category sample type for recording the result of an ovulation home test.
    ['HKCategoryTypeIdentifierAudioExposureEvent', 'AudioExposureEvent'], // A category sample type for audio exposure events.
    ['HKCategoryTypeIdentifierLowHeartRateEvent', 'LowHeartRateEvent'], // A category sample type for low heart rate events.
    ['HKCategoryTypeIdentifierHighHeartRateEvent', 'HighHeartRateEvent'], // A category sample type for high heart rate events.
    ['HKCategoryTypeIdentifierIrregularHeartRhythmEvent', 'IrregularHeartRhythmEvent'], // A category sample type for irregular heart rhythm events.
    ['HKCategoryTypeIdentifierMindfulSession', 'MindfulSession'], // A category sample type for recording a mindful session.
    ['HKCategoryTypeIdentifierSleepAnalysis', 'SleepAnalysis'], // A category sample type for sleep analysis information.
    ['HKCategoryTypeIdentifierToothbrushingEvent', 'ToothbrushingEvent'], // A category sample type for toothbrushing events.
    ['HKCategoryTypeIdentifierHandwashingEvent', 'HandwashingEvent'], // A category sample type for handwashing events.
    ['HKQuantityTypeIdentifierStepCount', 'StepCount', { unit: 'count' }], // A quantity sample type that measures the number of steps the user has taken.
    ['HKQuantityTypeIdentifierDistanceWalkingRunning', 'DistanceWalkingRunning', { unit: LENGTH_UNIT }], // A quantity sample type that measures the distance the user has moved by walking or running.
    ['HKQuantityTypeIdentifierDistanceCycling', 'DistanceCycling', { unit: LENGTH_UNIT }], // A quantity sample type that measures the distance the user has moved by cycling.
    ['HKQuantityTypeIdentifierPushCount', 'PushCount', { unit: 'count' }], // A quantity sample type that measures the number of pushes that the user has performed while using a wheelchair.
    ['HKQuantityTypeIdentifierDistanceWheelchair', 'DistanceWheelchair', { unit: LENGTH_UNIT }], // A quantity sample type that measures the distance the user has moved using a wheelchair.
    ['HKQuantityTypeIdentifierSwimmingStrokeCount', 'SwimmingStrokeCount', { unit: 'count' }], // A quantity sample type that measures the number of strokes performed while swimming.
    ['HKQuantityTypeIdentifierDistanceSwimming', 'DistanceSwimming', { unit: LENGTH_UNIT }], // A quantity sample type that measures the distance the user has moved while swimming.
    ['HKQuantityTypeIdentifierDistanceDownhillSnowSports', 'DistanceDownhillSnowSports', { unit: LENGTH_UNIT }], // A quantity sample type that measures the distance the user has traveled while skiing or snowboarding.
    ['HKQuantityTypeIdentifierBasalEnergyBurned', 'BasalEnergyBurned', { unit: ENERGY_UNIT }], // A quantity sample type that measures the resting energy burned by the user.
    ['HKQuantityTypeIdentifierActiveEnergyBurned', 'ActiveEnergyBurned', { unit: ENERGY_UNIT }], // A quantity sample type that measures the amount of active energy the user has burned.
    ['HKQuantityTypeIdentifierFlightsClimbed', 'FlightsClimbed', { unit: 'count' }], // A quantity sample type that measures the number flights of stairs that the user has climbed.
    ['HKQuantityTypeIdentifierNikeFuel', 'NikeFuel', { unit: 'count' }], // A quantity sample type that measures the number of NikeFuel points the user has earned.
    ['HKQuantityTypeIdentifierAppleExerciseTime', 'AppleExerciseTime', { unit: TIME_UNIT }], // A quantity sample type that measures the amount of time the user spent exercising.
    ['HKQuantityTypeIdentifierAppleStandTime', 'AppleStandTime', { unit: TIME_UNIT }], // A quantity sample type that measures the amount of time the user has spent standing.
    ['HKQuantityTypeIdentifierHeight', 'Height', { unit: LENGTH_UNIT }], // A quantity sample type that measures the user’s height.
    ['HKQuantityTypeIdentifierBodyMass', 'BodyMass', { unit: MASS_UNIT }], // A quantity sample type that measures the user’s weight.
    ['HKQuantityTypeIdentifierBodyMassIndex', 'BodyMassIndex', { unit: 'count' }], // A quantity sample type that measures the user’s body mass index.
    ['HKQuantityTypeIdentifierLeanBodyMass', 'LeanBodyMass', { unit: MASS_UNIT }], // A quantity sample type that measures the user’s lean body mass.
    ['HKQuantityTypeIdentifierBodyFatPercentage', 'BodyFatPercentage', { unit: 'percent' }], // A quantity sample type that measures the user’s body fat percentage.
    ['HKQuantityTypeIdentifierWaistCircumference', 'WaistCircumference', { unit: LENGTH_UNIT }], // A quantity sample type that measures the user’s waist circumference.
    ['HKQuantityTypeIdentifierBasalBodyTemperature', 'BasalBodyTemperature', { unit: TEMPERATURE_UNIT }], // A quantity sample type for recording the user’s basal body temperature.
    ['HKQuantityTypeIdentifierHeartRate', 'HeartRate', { unit: 'count/min' }], // A quantity sample type that measures the user’s heart rate.
    ['HKQuantityTypeIdentifierRestingHeartRate', 'RestingHeartRate', { unit: 'count/min' }], // A quantity sample type that measures the user’s resting heart rate.
    ['HKQuantityTypeIdentifierWalkingHeartRateAverage', 'WalkingHeartRateAverage', { unit: 'count/min' }], // A quantity sample type that measures the user’s heart rate while walking.
    ['HKQuantityTypeIdentifierHeartRateVariabilitySDNN', 'HeartRateVariabilitySDNN', { unit: TIME_UNIT }], // A quantity sample type that measures the standard deviation of heartbeat intervals.
    ['HKQuantityTypeIdentifierOxygenSaturation', 'OxygenSaturation', { unit: 'percent' }], // A quantity sample type that measures the user’s oxygen saturation.
    ['HKQuantityTypeIdentifierBodyTemperature', 'BodyTemperature', { unit: TEMPERATURE_UNIT }], // A quantity sample type that measures the user’s body temperature.
    ['HKQuantityTypeIdentifierBloodPressureDiastolic', 'BloodPressureDiastolic', { unit: 'mmHg' }], // A quantity sample type that measures the user’s diastolic blood pressure.
    ['HKQuantityTypeIdentifierBloodPressureSystolic', 'BloodPressureSystolic', { unit: 'mmHg' }], // A quantity sample type that measures the user’s systolic blood pressure.
    ['HKQuantityTypeIdentifierRespiratoryRate', 'RespiratoryRate', { unit: 'count/min' }], // A quantity sample type that measures the user’s respiratory rate.
    ['HKQuantityTypeIdentifierVO2Max', 'VO2Max', { unit: 'ml/(kg*min)' }], // A quantity sample that measures the maximal oxygen consumption during incremental exercise.
    ['HKQuantityTypeIdentifierDietaryBiotin', 'DietaryBiotin', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of biotin (vitamin B7) consumed.
    ['HKQuantityTypeIdentifierDietaryCaffeine', 'DietaryCaffeine', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of caffeine consumed.
    ['HKQuantityTypeIdentifierDietaryCalcium', 'DietaryCalcium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of calcium consumed.
    ['HKQuantityTypeIdentifierDietaryCarbohydrates', 'DietaryCarbohydrates', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of carbohydrates consumed.
    ['HKQuantityTypeIdentifierDietaryChloride', 'DietaryChloride', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of chloride consumed.
    ['HKQuantityTypeIdentifierDietaryCholesterol', 'DietaryCholesterol', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of cholesterol consumed.
    ['HKQuantityTypeIdentifierDietaryChromium', 'DietaryChromium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of chromium consumed.
    ['HKQuantityTypeIdentifierDietaryCopper', 'DietaryCopper', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of copper consumed.
    ['HKQuantityTypeIdentifierDietaryEnergyConsumed', 'DietaryEnergyConsumed', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of energy consumed.
    ['HKQuantityTypeIdentifierDietaryFatMonounsaturated', 'DietaryFatMonounsaturated', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of monounsaturated fat consumed.
    ['HKQuantityTypeIdentifierDietaryFatPolyunsaturated', 'DietaryFatPolyunsaturated', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of polyunsaturated fat consumed.
    ['HKQuantityTypeIdentifierDietaryFatSaturated', 'DietaryFatSaturated', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of saturated fat consumed.
    ['HKQuantityTypeIdentifierDietaryFatTotal', 'DietaryFatTotal', { unit: MASS_UNIT }], // A quantity sample type that measures the total amount of fat consumed.
    ['HKQuantityTypeIdentifierDietaryFiber', 'DietaryFiber', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of fiber consumed.
    ['HKQuantityTypeIdentifierDietaryFolate', 'DietaryFolate', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of folate (folic acid) consumed.
    ['HKQuantityTypeIdentifierDietaryIodine', 'DietaryIodine', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of iodine consumed.
    ['HKQuantityTypeIdentifierDietaryIron', 'DietaryIron', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of iron consumed.
    ['HKQuantityTypeIdentifierDietaryMagnesium', 'DietaryMagnesium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of magnesium consumed.
    ['HKQuantityTypeIdentifierDietaryManganese', 'DietaryManganese', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of manganese consumed.
    ['HKQuantityTypeIdentifierDietaryMolybdenum', 'DietaryMolybdenum', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of molybdenum consumed.
    ['HKQuantityTypeIdentifierDietaryNiacin', 'DietaryNiacin', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of niacin (vitamin B3) consumed.
    ['HKQuantityTypeIdentifierDietaryPantothenicAcid', 'DietaryPantothenicAcid', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of pantothenic acid (vitamin B5) consumed.
    ['HKQuantityTypeIdentifierDietaryPhosphorus', 'DietaryPhosphorus', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of phosphorus consumed.
    ['HKQuantityTypeIdentifierDietaryPotassium', 'DietaryPotassium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of potassium consumed.
    ['HKQuantityTypeIdentifierDietaryProtein', 'DietaryProtein', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of protein consumed.
    ['HKQuantityTypeIdentifierDietaryRiboflavin', 'DietaryRiboflavin', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of riboflavin (vitamin B2) consumed.
    ['HKQuantityTypeIdentifierDietarySelenium', 'DietarySelenium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of selenium consumed.
    ['HKQuantityTypeIdentifierDietarySodium', 'DietarySodium', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of sodium consumed.
    ['HKQuantityTypeIdentifierDietarySugar', 'DietarySugar', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of sugar consumed.
    ['HKQuantityTypeIdentifierDietaryThiamin', 'DietaryThiamin', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of thiamin (vitamin B1) consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminA', 'DietaryVitaminA', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of vitamin A consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminB12', 'DietaryVitaminB12', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of cyanocobalamin (vitamin B12) consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminB6', 'DietaryVitaminB6', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of pyridoxine (vitamin B6) consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminC', 'DietaryVitaminC', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of vitamin C consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminD', 'DietaryVitaminD', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of vitamin D consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminE', 'DietaryVitaminE', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of vitamin E consumed.
    ['HKQuantityTypeIdentifierDietaryVitaminK', 'DietaryVitaminK', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of vitamin K consumed.
    ['HKQuantityTypeIdentifierDietaryWater', 'DietaryWater', { unit: VOLUME_UNIT }], // A quantity sample type that measures the amount of water consumed.
    ['HKQuantityTypeIdentifierDietaryZinc', 'DietaryZinc', { unit: MASS_UNIT }], // A quantity sample type that measures the amount of zinc consumed.
    ['HKQuantityTypeIdentifierUVExposure', 'UVExposure', { unit: 'count' }], // A quantity sample type that measures the user’s exposure to UV radiation.
    ['workoutType', 'Workout'], // Gives you workout information.
    // ['HKWorkoutRouteTypeIdentifier', 'Route'], // Gives you route general information.
  ];

  const log = (message) => {
    const logElement = document.querySelector('#log');
    logElement.innerText = `${logElement.innerText}\n${JSON.stringify(message, null, 2)}`;
  };

  window.onerror = (message, source, lineno, colno, error) => {
    willExit = false;
    log('window.onerror');
    log(message);
    log(error);
  };

  const runBackup = async (givenTimePeriodDays) => {
    log('Started backup');

    const timePeriodDays = givenTimePeriodDays || document.querySelector('#days').value;
    const progressElement = document.querySelector('#progress');
    const progress = new Array(SAMPLE_TYPES.length).fill('-'.repeat(BAR_LENGTH));

    const processSampleType = async (configuration, index) => {
      const [sampleType, sampleName, queryProperties = {}] = configuration;

      let sampleTypeCount = 0;
      const now = new Date();
      const backupPeriodMs = 1000 * 60 * 60 * 24 * timePeriodDays;
      const earliestEpochDate = now.getTime() - backupPeriodMs;
      let endDate = now;
      let timeRangeMs = backupPeriodMs;

      while (endDate.getTime() > earliestEpochDate) {
        const startDate = new Date(endDate.getTime() - timeRangeMs);
        /* eslint-disable-next-line no-loop-func */
        const data = await new Promise((resolveSampleQuery) => {
          window.plugins.healthkit.querySampleType(
            {
              startDate,
              endDate,
              sampleType,
              limit: QUERY_LIMIT,
              ...queryProperties,
            },
            (sampleTypeItems) => {
              let results = [];
              try {
                if (sampleTypeItems.length >= QUERY_LIMIT) {
                  // If we received the QUERY_LIMIT or more samples back, halve the current
                  // query time range window, and run the request again so we don't miss
                  // anything.
                  timeRangeMs /= 2;
                } else {
                  // Otherwise, we know we got all the samples for the given time range. Add
                  // them to the results to be returned and update our counters.
                  sampleTypeCount += sampleTypeItems.length;
                  endDate = startDate;
                  results = sampleTypeItems;
                  // Also, if we didn't receive any results for this range, lets double the
                  // next range so we can 'accelerate' through ranges where there are no
                  // samples. As samples can be highly clustered, this heavily decreases the
                  // time required to find all samples compared to moving through in a linear
                  // fashion.
                  if (sampleTypeItems.length === 0) {
                    timeRangeMs *= 2;
                  }
                }
                // Update the progress object for this sampleType
                const completed = Math.min(
                  BAR_LENGTH,
                  parseInt(BAR_LENGTH * ((now.getTime() - endDate.getTime()) / backupPeriodMs), 10),
                );
                const remaining = BAR_LENGTH - completed;
                progress[index] = `${'='.repeat(completed)}${'-'.repeat(remaining)} ${'&nbsp;'.repeat(6 - `${sampleTypeCount}`.length)}${sampleTypeCount} ${sampleName}`;
                progressElement.innerHTML = JSON.stringify(progress, null, 1);
              } catch (querySampleTypeHandlerError) {
                log('querySampleTypeHandlerError');
                log(querySampleTypeHandlerError.message);
              }
              resolveSampleQuery(results);
            },
            (error) => {
              log(`${sampleName}: Error`);
              log(error);
              resolveSampleQuery([]);
            },
          );
        });
          // If this time range contained samples, send them back to the server.
        if (data.length) {
          try {
            const response = await fetch(
              SERVER_ADDRESS,
              {
                method: 'POST',
                headers: {
                  'Content-Type': 'application/json',
                },
                body: JSON.stringify({
                  sampleName,
                  data,
                }),
              },
            );
            const { result, error } = await response.json();
            if (result === 'nack') {
              log('request not acknowledged');
              log(error);
            }
          } catch (requestError) {
            log('requestError');
            log(requestError.message);
          }
        }
      }

      // On completion of this sampleType, finalize the progress bar.
      progress[index] = `${':'.repeat(BAR_LENGTH)} ${'&nbsp;'.repeat(6 - `${sampleTypeCount}`.length)}${sampleTypeCount} ${sampleName}`;
      progressElement.innerHTML = JSON.stringify(progress, null, 1);
    };

    const sampleTypePromises = SAMPLE_TYPES.map(processSampleType);

    // Wait for all SAMPLE_TYPES to complete
    await Promise.all(sampleTypePromises);
    log('Completed backup');
  };

  const main = async () => {
    const killAppIfRequired = () => {
      willExit = false;
      document.removeEventListener('click', killAppIfRequired);
      log('Interaction detected, disabling automatic exit');
    };
    document.addEventListener('click', killAppIfRequired);

    // Automatically request authorization to the required health kit types.
    await new Promise((res) => {
      window.plugins.healthkit.requestAuthorization(
        {
          readTypes: [
            'HKWorkoutTypeIdentifier',
            'HKWorkoutRouteTypeIdentifier',
            ...SAMPLE_TYPES.map(([i]) => i),
          ],
        },
        () => {
          log('requestAuthorization success');
          res();
        },
        () => {
          log('requestAuthorization failure');
          res();
        },
      );
    });

    // Always run a backup of the last 7 days of data on startup. This is a bit of overkill if
    // you are running the app once a day, but it is quick and may pick up data that is not
    // added immediately.
    await runBackup(7);

    // Wait three seconds after the initial completed backup before exiting if there is no
    // user interaction
    setTimeout(() => {
      if (willExit) {
        cordova.plugins.exit();
      }
    }, 1000);
  };

  document.addEventListener('deviceready', main, false);
  </script>
</body>

</html>

Building the backend

As I’m running only on my local network, I’ve only put together a very quick and simple solution which involves only three files.

Firstly, create a Dockerfile that handles all the backend dependencies and a run.sh bash command to bring everything up (you could also create a docker-compose file for this, I’m doing this all very quick and loosely so have left it as bash).

Dockerfile

FROM node:12-slim
WORKDIR /server
RUN npm install express cors body-parser mysql
COPY index.js index.js

run.sh

#!/usr/bin/env sh

docker network create health-kit-exporter

docker build -t health-kit-exporter .

docker rm -fv health-kit-exporter-mysql || true
docker run \
--detach \
--rm \
--network health-kit-exporter \
--name health-kit-exporter-mysql \
-v $(pwd)/data/mysql:/var/lib/mysql \
-p 127.0.0.1:3306:3306 \
--env MYSQL_ALLOW_EMPTY_PASSWORD=true \
--restart unless-stopped \
mysql \
--default-authentication-plugin=mysql_native_password

docker rm -fv health-kit-exporter || true
docker run \
-it \
--rm \
--network health-kit-exporter \
--name health-kit-exporter \
-p 0.0.0.0:5000:80 \
-v $(pwd)/data/debug:/data/debug \
-v $(pwd)/index.js:/server/index.js \
--restart unless-stopped \
health-kit-exporter \
node .

The next step is adding an express server which will take requests from the app and insert the data into a MySQL database. Again, this code should really be seen as a template for you to build your own implementation.

index.js

const { writeFile } = require('fs').promises;
const express = require('express');
const cors = require('cors');
const bodyParser = require('body-parser');
const mysql = require('mysql');

const SHARED_SECRET = 'RANDOM_STRING_OF_CHARACTERS';

(async () => {
  const app = express();
  app.use(cors());
  app.use(bodyParser.json({ limit: '100mb' }));

  // Wait 5 seconds to ensure that mysql has started correctly.
  await new Promise((res) => setTimeout(res, 5000));

  const connection = mysql.createConnection({
    host: 'health-kit-exporter-mysql',
    user: 'root',
    password: '',
    multipleStatements: true,
  });

  connection.connect();

  const runQuery = (query, parameters = []) => new Promise((res, rej) => {
    connection.query(query, parameters, (error, results) => {
      if (error) rej(error);
      res(results);
    });
  });

  runQuery(`
    CREATE DATABASE IF NOT EXISTS health_kit_data;
    USE health_kit_data;
    CREATE TABLE IF NOT EXISTS combined (
      id VARCHAR(100) NOT NULL,
      timestamp INT(12) NOT NULL,
      property VARCHAR(100) NOT NULL,
      value DOUBLE NULL,
      startedAt INT(12) NOT NULL,
      meta JSON,
      UNIQUE KEY id (id)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
  `);

  app.post('/', async (req, res, next) => {
    try {
      const { secret } = req.headers;
      if (secret !== SHARED_SECRET) return next();
      const { sampleName, data } = req.body;
      await writeFile(`/data/debug/${sampleName}.json`, JSON.stringify(data, null, 2));
      const queryPromises = data
        .map(({
          UUID, endDate, startDate, quantity = '', value = '', duration = '', ...rest
        }) => ([
          UUID, // id
          `${new Date(endDate).getTime() / 1000}`, // timestamp
          `${new Date(startDate).getTime() / 1000}`, // startedAt
          sampleName, // property
          `${quantity}` || `${value}` || `${duration}`, // value
          JSON.stringify(rest), // meta
        ]))
        .map(async (parameters) => {
          try {
            await runQuery(
              'INSERT INTO combined SET id=?, timestamp=?, startedAt=?, property=?, value=?, meta=? ON DUPLICATE KEY UPDATE id=?, timestamp=?, startedAt=?, property=?, value=?, meta=?',
              [
                ...parameters,
                ...parameters,
              ],
            );
            return null;
          } catch (error) {
            return {
              error: error.message,
              parameters,
            };
          }
        });

      const errors = (await Promise.all(queryPromises)).filter((i) => i);
      if (!errors.length) {
        return res.json({ result: 'ack' });
      }
      return res.json({ result: 'nack', error: errors });
    } catch (error) {
      return res.json({ result: 'nack', error: error.message });
    }
  });

  app.listen(80, () => {
    console.log('Listening for incoming connections');
  });
})();

Once you have created these three files, Dockerfile, run.sh and index.js together in a directory, ensure the run.sh is executable via chmod +x run.sh and then run it via ./run.sh. Once started you should see a log message saying Listening for incoming connections.

At this point, your current machine will have the backend listening on port 5000 or which ever port is set in the run.sh command if you have changed it.

Connecting the app to the backend

As mentioned previously, I’m only running the backend service on my local home network, and only expect the app to be able to work when I am at home and connected to the WiFi.

There are a number of ways for doing this so this might be something that you’ll need to figure out for yourself if you are playing along at home. For my situation, I’ve found the best solution based on the equipment that I have is to set a static IP for the dashboard as resolving hostnames can be a little touchy with my router / network.

I have previously limited my routers DHCP range when setting up a local Kubernetes cluster so I know that I can assign static IP addresses within a given range without the worry of conflicts (as long as I keep a good record of who is using what). This then makes it a simple case of updating the app code so that the SERVER_ADDRESS variable represents the address of the backend service. The port also needs to be included here, so the value should look something similar to http://10.0.0.200:5000.

At this stage, you should be able to open the app and see progress bars for each of the different samples types as the app requests them and sends them to the backend. Again, this is a very quick and scrappy system, but it gives a template from which to build to your own use case. While the MySQL queries are parameterized, there is still a number of security measures that should be added as far as further mitigating injection vectors, using encryption in the communication between the app and backend and hardening up the backend interface. For now, running on the local network means that these are low risk factors though.

Result

When running a backup over the last 3 years since I started using Apple, there are just shy of one million results. I feel that a lot of this is due to the fact that I also got an Apple watch not long after getting the phone, and while I have only used the watch about two thirds of the time, it still collects a huge amount of data.

I haven’t yet had a chance to slice and dice the data too much at this point, but I’m sure there are going to be a number of interesting insights that I can discover.

As far as what is on the dashboard, I now can display pretty much anything that is found in the Fitness or Health apps. I’m also working on a few aggregates for quick reference, and I’m also working to incorporate my wifes data as well so that we can compare against each other which helps to keep us focused on our health and fitness goals.

In summary, it really should not be as difficult as this to access your own data in a way that is useful. Hopefully this post helps you unlock your own data in some useful way as well.

If you’re interested in doing something similar, please feel free to contact me through the contact form on my site. It is likely that this project will mature as I use it more and will be more than happy to help anyone who is interested in doing the same sort of exporter.

Posted in: