What are you afraid of? I would be more worried about the data in your phone. There is more damage to be made with GPS location and all the other private data.
I doubt that knowing what time you sleep, when you exercise, and your hearth rate can add a lot more risks. Maybe the hearth rate combined with what you are doing on your phone can be used to measure the effect of what you are reading or watching...
Well, this data would been incredibly valuable to insurance companies. If you could tie specific demographic information (which Google has mastered) to sleep and exercise routines, this could be used in a risk model.
I'm fairly confident this will eventually happen, as I had a professor explain to me that he wears a fitness tracker in exchange for a premium reduction. I think the incentive of the insurance company is that they found individuals wearing these devices to be more conscientious of their own health (I don't think they request any data at this point).
Even if you don't partake, these risk models will certainly generalize your demographic to their set of data.
From the Ad-revenue perspective, it's easy to see how they could target you for selling melatonin if your sleep pattern supports it. Or, maybe you're active, which brings about plenty of products to target you for.
I doubt that knowing what time you sleep, when you exercise, and your hearth rate can add a lot more risks. Maybe the hearth rate combined with what you are doing on your phone can be used to measure the effect of what you are reading or watching...