“It Is Fucked Up What Is Happening”: Signal’s Whittaker on AI in the Operating System
Set Trending Topics as a preferred source on Google.
Meredith Whittaker, president of the messaging app Signal, delivered an unusually blunt assessment of the relationship between privacy and artificial intelligence at the TechBBQ conference in Copenhagen. Her argument: today’s AI boom is the product of the advertising and surveillance business model that has carried the tech industry since the 1990s. And the next stage of that model, AI assistants embedded deep inside operating systems, could pull the ground out from under Signal as it exists today.
Roughly a billion people currently use ChatGPT, and about as many use Google’s Gemini, which is also deeply integrated into Android. Billions more will soon have access to Siri AI in the next version of iOS. Anthropic, meanwhile, is heading toward an IPO that could make the maker of Claude the seventh most valuable company in the world.
From Data Collection to Inference
Whittaker begins by describing a shift in what privacy even means. For a long time, the guiding question was which data a company could collect about a person. Today the question is what models can infer from it. “I think the surface area for data creation and collection and the kind of inferences and modeling of who we are and our behavior has expanded dramatically,” she said.
That expansion, she argues, is a product of AI, and AI in turn is a product of a specific business model: “a very particular business model that was built around collecting huge amounts of data about users, scaling platforms, scaling tech products (…) in order to create models of types of people that they could sell advertisers access to.” She traces its origins to the 1990s, when the regulatory framework for the internet industry took shape in the United States: an advertising-funded tech sector that underwrote the privatization of the internet and remains the economic engine of the industry.
Old Algorithms, New Volumes of Data
From there Whittaker derives her explanation for the rise of the AI companies. The deep learning methods themselves are old, she notes. Backpropagation dates back to the 1970s and 1980s and was long regarded as experimental and largely ineffective, “largely because they weren’t matched with huge amounts of data and huge amounts of compute.”
She locates the turning point in the early 2010s, when Google, Facebook and others had both: enormous training corpora such as YouTube videos and user data, plus the server capacity they had already built to collect and process that data. “Suddenly they were reanimated with the huge amounts of data (…) These algorithms, these AI approaches were really, really good at things like calibrating an engagement-driven social media feed.” Her conclusion: “This revival of AI actually rests on the basis of this mass data collection, this business model that itself was a contingency of 1990s regulatory and policy decisions, not the inevitable shape of tech today.”
By now the mechanism runs in both directions, she says, with AI intensifying the hunger for data. Behind product promises such as voice assistants and AI glasses, Whittaker sees first and foremost an apparatus for data capture: “We’re talking about a data collection apparatus that they’re hoping marketing will lull us into without thinking about the collateral consequences.” Her call to users and companies is to ask concrete questions: where the data sits, where it is processed, who owns it, and which other data it gets joined with.
Why Signal Is a Nonprofit
Asked whether a company like Google or Meta could ever be fully privacy-respecting, Whittaker points to economics: “Given the current business model they cannot be.” Technically the question is settled, she says, with Signal as the reference point: “The ideas are not scarce. What we’re talking about are political economic realities.”
Signal costs around 50 million dollars a year in bandwidth, servers and staff, according to her, and is funded by donations. She puts the reason plainly: “I cannot have a board member coming back from Davos being like, carve out a little bit of your privacy promises so that you can actually beat those revenue goals.” In the AI era, the pressure has grown: “There is an atmosphere of almost psychedelic hype around AI that makes it harder and harder to just defend these fundamentals on technical grounds.”
On the regulatory front, Signal continues to face attempts to weaken encryption. Whittaker speaks of “zombie bills, like chat controls,” meaning efforts such as the EU proposal that justifies blanket scanning of communications as a way to protect children. Signal’s announcement that it would leave the EU and the UK if such an obligation passed is meant seriously: “If we puncture the network at one point, we have poisoned it for everyone (…) we’ll go home.”
“It Is Fucked Up What Is Happening”
The danger Whittaker considers most underestimated is the integration of AI assistants and agents into operating systems. Here she gets explicit: “Allow me this morning to be a little bit spicy, but it is fucked up what is happening.”
Her example is the summarization feature in Siri under iOS 27. Ask the assistant to summarize what is on screen, and it takes a screenshot of the app in the foreground. “It is taking a screenshot if the Signal is there and you haven’t blocked it as a user. And it is then sending a screenshot of your Signal message off-site potentially to be processed by an LLM so Siri can say your message says dinner is at 7.” Whether the processing happens on the device or on a server is not clearly documented, she says. She sees comparable capabilities in Gemini and in the integration of ChatGPT with iMessage.
The point of attack thus moves away from encryption itself, which in Signal’s case has been openly documented for more than a decade and scrutinized by the security community. “That is a battleship if you want to get through that. Suddenly, there is a hole in the battleship that is typing that information into an insecure database, off-site, into the hands of a third party, because the operating system is the ocean we and every other application developers swim in.” App makers would have control over this only if Apple, Google or Microsoft built in toggles to opt out.
Without such options, Whittaker sees Signal’s foundation at risk: “I’m not being dramatic here: if this trajectory holds in a year or two Signal will not be able to operate with integrity. We will not be able to make privacy promises that we as developers who are independent can keep, because the water we swim in will have been poisoned.”
How Signal Itself Uses AI
Signal has been conspicuously restrained about adding AI features. Whittaker’s reasoning, delivered to the room: “Raise your hand if you want an annoying chatbot in your messenger.” No hands go up. And further: “We don’t have data. So we’re not going to turn over data for an inference for a chatbot, and we don’t have data to train it on.” Then there is the context of use: “You’re in Ukraine, you need to quickly talk. You don’t need like, let me summarize your dinner plan.” Signal has “a duty of care to serious people,” she says.
There is one AI feature at Signal: a face detection model that runs locally on the device and recognizes whether a photo contains a face. It powers the automatic blurring in the media editor. “If I posted that on social media, I would not be exposing your biometrics to a big tech company. That is a mission-aligned use of AI, but it is very precise.”
The proliferation of AI features in other products she attributes to internal pressure: “A lot of what you’re seeing (…) is like a desperation by teams who’ve been told from higher up that you have a KPI this quarter to ship AI. Figure it out. Before lunch, they get in a conference room, like Chad, Brad, and Mike are like, I don’t know, I guess we put a chatbot in this.” The promise is a magic assistant, she says, while the result is often a flawed summary that books the wrong flight.
Privacy as a Business Model for European Startups
Asked from the audience what she would recommend to European founders, Whittaker offers a market-gap logic: “Look at what the big tech model can’t do well (…) What is the scale at all costs, AI is everything model not able to do? And one thing is privacy.” She sees demand in banking, in government and around European defense concerns: “There is still a huge market need for privacy (…) And particularly with the sovereignty concerns, we will have customers here.”
She backs this with a security argument: “LLMs themselves are not secure architectures. The data extraction attacks, data poisoning, there’s a lot of problems with just that form in terms of security and privacy.” She is even more critical of the systems built around them, which are designed for sweeping access to user data and broad authority to act: “That looks a lot like malware if you’re looking at it from a sort of software architecture perspective.” Her appeal to the industry is to make security and privacy the “apex functions” of design.
What makes Whittaker optimistic is the widening gap between marketing and reality: “We’re in a place where the hype seems almost divorced from reality (…) Every private conversation I have with someone kind of off the record, people are like, yeah, this is weird.” Her hope: “People will find the muscle memory, find the strength to sort of stand up and confront a grim map with a strategy that is worthy of it.”
