1. The energy inside the Four Seasons Hotel on Seattle’s downtown waterfront felt less like a typical tech conference and more like a room of people trying to glimpse a future that hasn’t quite arrived yet. The occasion was Madrona’s IA40 Summit, an annual gathering where the venture capital firm brings together AI startups, investors, and executives from giants like Microsoft, Amazon, Anthropic, and Stripe. For the most part, the people in the room agreed on the broad strokes: AI agents—software that can act on your behalf, browsing, clicking, and making decisions—are about to transform how businesses and customers interact. But one question kept surfacing, stubbornly unresolved, like a splinter no one could remove: when an AI agent becomes the middleman between a company and its customer, who actually owns the relationship? And more importantly, who gets to keep the data that flows through that relationship? Madrona’s managing director, Matt McIlwain, put it plainly in his closing remarks. The question of who gets to use the data generated by people’s engagement with AI systems came up “over and over and over again” throughout the day. It was the kind of puzzle that doesn’t have a clean answer yet—and that might explain why so many smart people in that room seemed to circle it with equal parts excitement and dread. The future, they all seemed to agree, will be amazing. But it’s also going to be messy, especially when it comes to trust, ownership, and loyalty in a world where a machine is doing the talking.
2. Perhaps the most vivid picture of that future came from executives who described a world where apps as we know them start to disappear into the background. Microsoft’s Charles Lamanna, the executive vice president who oversees Microsoft 365 and the platform behind Copilot, made a bold prediction: most business software will eventually be used by AI assistants on behalf of a person, not by people clicking around themselves. When that happens, software makers lose a lot of leverage, starting with pricing power. If an AI agent can do the work without needing a company’s neatly designed interface, why would anyone pay the old prices for that interface? Lamanna used the term “thin app” to describe the lightweight shells of software that people might only touch briefly, if at all. Jean-Denis Greze, CEO of Town, a startup building an AI assistant for work, went even further. He said that since around July, AI has become capable of operating a browser or a computer almost as well as a human being, and at a reasonable cost. That means an AI assistant can work through an app’s user interface exactly the way a person would, without needing a special API from the software maker. “Everything is going to become a thin app because the better the AI gets at using the computer, the less the app matters as a unit of software,” Greze said, repeating Lamanna’s phrase with almost a sense of inevitability. The same shakeup is happening in retail, too. Maia Josebachvili, Stripe’s chief revenue officer for AI, described a striking pattern: commerce handled by AI agents on Stripe stayed roughly flat for eight or nine months, then shot up sharply in the past six weeks. That sudden surge signals a sea change, but not an entirely comfortable one. Many merchants make their real money by getting shoppers to add extra items at checkout or by showing them tailored ads. When a tireless AI agent simply buys what was requested and leaves, those opportunities vanish. Josebachvili said that model won’t work over the long term. The tension is already playing out in real time, most publicly in a clash between Amazon and Meta: Amazon recently blocked Meta’s Muse assistant from shopping on its site. It’s a glimpse of a world where your assistant’s ability to shop for you depends on whether the platform recognizes the assistant as a trusted ally or treats it as an intruder.
3. Underneath all of this lies a messy, unresolved argument about data. It came to the surface in a session with Anthropic’s Chief Technology Officer, Rahul Patil, during a conversation moderated by Raphaëlle d’Ornano. She asked a probing question: who owns the record of an AI agent’s work, including its mistakes, its corrections, and the way it learned from them? Does that record belong to the customer? She mentioned that she had never gotten a clear answer to that question. Patil didn’t exactly provide one either. Instead, he explained that each company supplying the software for running agents will naturally work to improve those systems, and, in his words, “will use every data that’s available to them to make it better.” He added that it’s good for the ecosystem if agents actually improve. But that answer leaves open the essential tension: your agent’s learning process might involuntarily feed the platform’s bottom line. Amazon Web Services vice president Swami Sivasubramanian brought a different angle, pointing out that one of the most overlooked parts of making AI agents work isn’t the model itself—it’s giving the agent the right company data for the task at hand. AWS announced a service in June called AWS Context, which maps the relationships in a company’s data so that agents can use them properly. Sivasubramanian said the service is designed to work across different platforms and cloud providers, and he expects this kind of “context layer” to become a basic building block for the next twenty years, just as cloud storage and databases were two decades ago. But even with technical solutions emerging, the human question remains. Who gets to learn from your decisions and mistakes? Who gets to profit from the intimate picture that an AI assistant builds of your preferences, your patterns, and your intentions? The room didn’t have a satisfying answer, just a shared sense that everyone wants to be the one collecting that information—and nobody wants to be the one giving it away.
4. There was also a widely held feeling, almost like a collective sigh, that the humans in the equation aren’t keeping up with the machines. Patil revealed that Anthropic now writes roughly 200 times as much code as it did just eighteen months ago. Some of the company’s customers have doubled or even tripled their output, but almost none of them have seen gains anything like that. The gap is not because the technology is failing. It’s because organizations are still built around processes, roles, and habits that predate AI. Archana Vemulapalli, global head of AI product management at Goldman Sachs, put it bluntly: “The bottleneck is actually not AI. The bottleneck is human.” That sentiment echoed throughout the day. Sivasubramanian said that teams inside Amazon had shown him impressive AI agents built in just two or three weeks. But when he asked when those agents could actually be rolled out, almost every team said the same thing: they still had to work out security, identity, and monitoring. The creativity of the model isn’t the problem. The guardrails are. McKinsey senior partner Lari Hamalainen offered an unsettling statistic: about 40% of companies report that AI has increased their profits, but only 6% describe the increase as substantial. In other words, there’s a lot of noise, but not as much signal as the hype suggests. Even the people building the technology admit that more capable models aren’t enough. As McIlwain said, there’s still a lot of work to do to close the gap between “the goals, the capabilities, and what we all are aspiring to do.” That gap isn’t just a technical problem. It’s an organizational problem, a leadership problem, and ultimately a trust problem. Companies are learning that you can’t just drop an AI agent into a messy workflow and expect it to fix everything. The agent might be ready, but the culture, the policies, and the people aren’t. Even OpenAI’s own safety chair, Carnegie Mellon professor Zico Kolter, pointed out that the ability to control AI systems has to keep pace with their capabilities—which might mean deliberately developing them more slowly than we actually could. That’s a hard thought for a world obsessed with speed, but it’s one that more and more executives seem to be taking seriously.
5. Another major theme of the summit was the uncomfortable dependency many companies are building on a single AI provider. Patil made a somewhat contrarian argument: large companies, he said, spend way too much time and energy preserving the ability to switch between AI providers. In trying to stay flexible, they build for what all the models have in common, which means they settle for the lowest common denominator. They pull talented people away from work that only their own company can do, and in the process they miss out on the benefits when a more capable model arrives. It’s an argument for commitment, not caution. But on the same panel, Carlos Guestrin, co-CEO of AI startup Noeri and a Stanford computer science professor, offered a very different vision. Guestrin, who previously founded Seattle machine learning startup Turi before Apple acquired it in 2016, argued that intelligence shouldn’t be controlled by one or two companies that own the models. Every company, he said, should be able to build and own its own AI systems. That debate—stick with a lab vs. build your own—played out in other conversations too. Eno Reyes, co-founder and CTO of Factory, a startup making AI coding tools, said many businesses don’t see a way to build their future without ceding control to one AI lab. Thomas Dohmke, the former GitHub CEO who now leads startup Entire, said developers always want choice. Amazon, for its part, seems determined to play both sides. The company is a major Anthropic investor and partner, but it also launched a managed agents product with OpenAI the same week, and it sells its own product for running AI agents, called AgentCore, which overlaps with Anthropic’s Managed Agents. Dan Grossman, Amazon’s vice president of business and corporate development, pointed out on an investor panel that the company is comfortable working with more than one AI lab. That sort of hedging might be smart business, but it also shows how unsettled the landscape still is. Nobody wants to bet everything on one horse in a race where the finish line keeps moving.
6. The financial reality hanging over all of this is staggering—and deeply concentrated. According to Madrona’s data, the companies on this year’s IA40 list have raised a combined $410 billion since they were founded. But OpenAI, Anthropic, and Databricks account for 92% of that money. Anthropic alone raised $143 billion of its $161 billion total in the twelve months ending August 15, a figure that includes debt. The company has also filed confidentially for an initial public offering and is reportedly seeking a valuation of around $2 trillion, with some reports suggesting it could go public as early as mid-November. Meanwhile, capital spending by Amazon, Alphabet, Microsoft, Meta, and Apple is projected to rise 74% this year, to roughly $742 billion—an almost incomprehensible sum. Lamanna noted that Microsoft’s heaviest AI users among its software developers are each on track to spend more than $1 million a year on AI usage, even with the company’s internal discount. That level of spending suggests the technology is becoming essential, but it also raises questions about who can afford to play. Madrona’s closing slide tried to sound an optimistic note: returns on AI are still early for most companies, but they look inevitable based on what the earliest adopters are seeing. Yet even with that optimism, the day ended with the same uncomfortable question that had been haunting it from the start. In a world where AI agents stand between businesses and customers, who gets to keep the relationship, and who gets to keep the data? The tools are evolving faster than the agreements that govern them. For all the talk of breakthroughs and trillion-dollar valuations, the real work ahead might be figuring out how to share the most valuable thing these systems create: not the intelligence, but the trust. And in that crowded room on the Seattle waterfront, no one had quite figured out how to do it yet.
