The device is definitely not constantly listening and uploading to the cloud (or at least they certainly do not intend it to). This keyword recognition thing everyone is doing is to make interaction more intentional, reduce misrecognition, bandwidth consumption, and processing costs.

Keyword recognition is done locally on the device at minimal processing cost. Compared to actually recognizing your speech, it's trivial to determine whether or not something you said is a given keyword. It's rather like biometric confirmation vs. identification, where confirming whether a given fingerprint is that of a given individual can be done quickly and with a certain probability distribution but determining which of millions of fingerprints corresponds to the one provided is incredibly difficult/unreliable.

The more difficult task of recognizing what you say after the keyword is done in the cloud, but only once you've said the keyword. It would incur considerable cost for you and them to constantly record and upload all audio, especially with their parametric microphone array for sound localization/isolation.

The real concern should be the device's own security, because hacking the device and enabling constant recording would be very harmful.