> Having worked in data/engineering at bigtech, it's not like there's a human on the other side reviewing what each user is doing
There doesn't need to be a human inspecting stuff, just a computer doing the filtering and reacting to it. The example that the article is using is already mentioning that they found the nefarious purpose. Given the current trend of politics, notably in the US, its not far-fetch to draw a future where searching for things like, say, abortion, becomes illegal, and tech companies get coerced into sharing that kind of information.
The fact that MSFT found that co-pilot was being used by hackers suggests that they already have the entire set of tools to already do that.
> Given the current trend of politics, notably in the US, its not far-fetch to draw a future where searching for things like, say, abortion, becomes illegal, and tech companies get coerced into sharing that kind of information.
This is the weakest of the privacy arguments to me. If we are in the "evil government" hypothetical future, why do they need my real searches? They can just as easily lie, make something up, use some other piece of data to persecute who they want, or do away with all of that pretense because they don't need it.
You're imagining a fictional, united evil government acting as a single entity. In real life right now, any given government has numerous bad actors at various levels who can and do use data to persecute people and groups they dislike. Most of the time, they still need real data, but it's a messy multidimensional spectrum and sometimes they can fabricate it. Another thing you're missing is that they're using this data to find the people they want to persecute.
There doesn't need to be a human inspecting stuff, just a computer doing the filtering and reacting to it. The example that the article is using is already mentioning that they found the nefarious purpose. Given the current trend of politics, notably in the US, its not far-fetch to draw a future where searching for things like, say, abortion, becomes illegal, and tech companies get coerced into sharing that kind of information.
The fact that MSFT found that co-pilot was being used by hackers suggests that they already have the entire set of tools to already do that.