Wow. The pressure to retrofit governance onto agentic systems after deployment is going to be significant, messy, and expensive. And most enterprises won’t have the option to simply pull them. Once agents are embedded in workflows, the integrations calcify. “Cancel” becomes “replace,” which requires the same governance work that was skipped the first time, plus the operational cost of explaining why a capability went offline. 😬
The rollout of agents in reminds me of other non-AI tech deployments. Everyone wants the new shiny tool before their competitors (usually before understanding the data, processes or users - key inputs for governance 🙄). Except rollout of AI/agentic tech changes the risk calculation making traditional deployment approaches unacceptable.
Good point. I project manage refurbs sometimes. When I do a refurb, I am more intersted in the bones, and structure, and foundational stuff. Electrics, plumbing, damp issues, how the house functions, etc. And far less interested in wall colours, floors, cushions.
I've been called out to many properties to try and fix a refurb gone wrong.
But remedial work can only ever go so far. Without starting again. Which for most people is not feasible. It's just pretty much impossible to fix after the fact.
All this hype around agents reminds me of the tulip bubble.
I also do not believe for one second that"it will get better" when agents/the industry/tech "matures"
I'm guessing that remedial patches just introduce more noise, and the agents get even more neurotic. I hope I'm wrong.
It's an unfortunate fact that the primary legal obligation of a company is to make profit for shareholders. In most scenarios in most sectors this ignores externalities and often results in many types of harm, eg. Oil companies, things owned by asset management, privatised utilities... Etc etc. this situation goes unchallenged...
It is nuts that the whole system operated this way round. Here is an example I was saying elsewhere earlier:
We need a joined up approach to aviation - there is still no government policy on that and associated emissions, and taxpayer pays at every level:
In aviation, there are 21 airport expansions going ahead which citizens fight at local level but get bounced back by government as there is no regulation on expansion. Airports are owned by foreign investors. The taxpayer pays at many levels for them to profit;
i) citizens pay for legal challenges to corporations and governments - vastly unfair, they never win.
ii) There is proven and calculable cost to NHS of lifetime burden (all documented) on citizens due to noise pollution, air pollution, removal of green space, depression and associated proclivity to comorbidities - lifetime cost to NHS, taxpayer foots the bill.
iii) emissions from aviation expansion have not been factored into net zero or even jet zero. Even if they were, the taxpayer would have to pay to remove them by CCS.
We are therefore paying for basic human rights, as well as paying taxes to profit shareholders abroad. We have so far not approached the topic like this.
I love it, I don’t know why, when machines behave like us. Maybe bec it’s a reminder that the items that displease us about AI, they already existed in *us.*
I think this also subtly points towards why dissent should be built into a team. The friction leads to debate, and I think ultimately a stronger outcome.
I also love nothing more than when I call customer service and speak to a human. My challenge is unique! If I couldn’t handle it online myself, I need another human to help :)
I find it fascinating too, when they behave like us. It’s one of those things that is both exciting and unsettling at the same time. Sam, thank you for such a thoroughly researched piece! It gave me a lot to think about.
you really shine a beam of light on stuff that matters, Sam. So appreciated by many. You know me a bit so of course I'm not in the direct line of working in any system using multi-agents - but this post is for all of us - we all depend on social systems that ask us to trust them... banks, insurance, healthcare. Your piece speaks to all of us. I don't know a single person that hasn’t directly experienced what you describe in so-called normal, functional and day to day admin.
Yes - those loops - the being sent around in circles wherein something quite prosaic, but nevertheless important in the management of, well, almost everything, increasingly mangles a poor human being’s thinking into madness. You said the phrase 'gas-lighting'. That is what it engenders, so attempting to sort an uncomplicated item on my to-do list, often leaves me feeling as if I've lost my mind. We know there's no-one there hearing us - such a basic thing that used to work imperfectly, often taking a bit of persistence, but still mostly getting there eventually.
Thank you for laying it out.
Thank you for concrete advice on identifying the tell before you've unraveled, and the pragmatic pushbacks.
& respect to Anthropic making unpalatable to the very competitive, incomprehensible scale of money in the game, research findings visible - I'm a committed realist, so I know Anthropic are not just the good guys. I know they are part of it all - nevertheless, I do think there are real ethics within Anthropic worth noting. I don't think we can ever reduce this co-created chaos of existence into villains and heroes - at least once we're beyond the tooth fairy and father christmas stage anyway...
Wow. The pressure to retrofit governance onto agentic systems after deployment is going to be significant, messy, and expensive. And most enterprises won’t have the option to simply pull them. Once agents are embedded in workflows, the integrations calcify. “Cancel” becomes “replace,” which requires the same governance work that was skipped the first time, plus the operational cost of explaining why a capability went offline. 😬
In the wise words of Sam, “go slow.” Very slow.
Exactly this. And honestly I think this ma have been going on for longer than many of us (certainly me!) expected. 😢
The rollout of agents in reminds me of other non-AI tech deployments. Everyone wants the new shiny tool before their competitors (usually before understanding the data, processes or users - key inputs for governance 🙄). Except rollout of AI/agentic tech changes the risk calculation making traditional deployment approaches unacceptable.
Exactly! FOMO is going to be VERY costly for some of these companies...
Good point. I project manage refurbs sometimes. When I do a refurb, I am more intersted in the bones, and structure, and foundational stuff. Electrics, plumbing, damp issues, how the house functions, etc. And far less interested in wall colours, floors, cushions.
I've been called out to many properties to try and fix a refurb gone wrong.
But remedial work can only ever go so far. Without starting again. Which for most people is not feasible. It's just pretty much impossible to fix after the fact.
All this hype around agents reminds me of the tulip bubble.
I also do not believe for one second that"it will get better" when agents/the industry/tech "matures"
I'm guessing that remedial patches just introduce more noise, and the agents get even more neurotic. I hope I'm wrong.
It's an unfortunate fact that the primary legal obligation of a company is to make profit for shareholders. In most scenarios in most sectors this ignores externalities and often results in many types of harm, eg. Oil companies, things owned by asset management, privatised utilities... Etc etc. this situation goes unchallenged...
Oh how I wish this weren't so Alice. 😢
If we are to have a policy for Agents why don't we have a policy in the real world?? Bonkers.
☹️💯
It is nuts that the whole system operated this way round. Here is an example I was saying elsewhere earlier:
We need a joined up approach to aviation - there is still no government policy on that and associated emissions, and taxpayer pays at every level:
In aviation, there are 21 airport expansions going ahead which citizens fight at local level but get bounced back by government as there is no regulation on expansion. Airports are owned by foreign investors. The taxpayer pays at many levels for them to profit;
i) citizens pay for legal challenges to corporations and governments - vastly unfair, they never win.
ii) There is proven and calculable cost to NHS of lifetime burden (all documented) on citizens due to noise pollution, air pollution, removal of green space, depression and associated proclivity to comorbidities - lifetime cost to NHS, taxpayer foots the bill.
iii) emissions from aviation expansion have not been factored into net zero or even jet zero. Even if they were, the taxpayer would have to pay to remove them by CCS.
We are therefore paying for basic human rights, as well as paying taxes to profit shareholders abroad. We have so far not approached the topic like this.
I love it, I don’t know why, when machines behave like us. Maybe bec it’s a reminder that the items that displease us about AI, they already existed in *us.*
I think this also subtly points towards why dissent should be built into a team. The friction leads to debate, and I think ultimately a stronger outcome.
I also love nothing more than when I call customer service and speak to a human. My challenge is unique! If I couldn’t handle it online myself, I need another human to help :)
One million percent the same. This is all why I tend to phone customer service rather than email them. That and the fact I am an old man. 😂
It’s so ironic. Email was supposed to speed things up, now it has its own wait time! And however old you are, I’m older! 💁🏻♀️
😎😎😎
I find it fascinating too, when they behave like us. It’s one of those things that is both exciting and unsettling at the same time. Sam, thank you for such a thoroughly researched piece! It gave me a lot to think about.
Very welcome Karo. And absolutely. Uncanny valley!
It makes it easier to see our faults. First is "oh you silly machine," second is "oh but that's just like us," third is "oh we are silly." 🤣
Hallucinating agent + hallucinating agent + hallucinating agent + no one thinking about ethics = (lawsuit x ∞)
Thanks Heather. That equation is spot on. 😬😬😬
I did not know that about the 6 scenarios in use. It's essential that decisions remain appealable!
you really shine a beam of light on stuff that matters, Sam. So appreciated by many. You know me a bit so of course I'm not in the direct line of working in any system using multi-agents - but this post is for all of us - we all depend on social systems that ask us to trust them... banks, insurance, healthcare. Your piece speaks to all of us. I don't know a single person that hasn’t directly experienced what you describe in so-called normal, functional and day to day admin.
Yes - those loops - the being sent around in circles wherein something quite prosaic, but nevertheless important in the management of, well, almost everything, increasingly mangles a poor human being’s thinking into madness. You said the phrase 'gas-lighting'. That is what it engenders, so attempting to sort an uncomplicated item on my to-do list, often leaves me feeling as if I've lost my mind. We know there's no-one there hearing us - such a basic thing that used to work imperfectly, often taking a bit of persistence, but still mostly getting there eventually.
Thank you for laying it out.
Thank you for concrete advice on identifying the tell before you've unraveled, and the pragmatic pushbacks.
& respect to Anthropic making unpalatable to the very competitive, incomprehensible scale of money in the game, research findings visible - I'm a committed realist, so I know Anthropic are not just the good guys. I know they are part of it all - nevertheless, I do think there are real ethics within Anthropic worth noting. I don't think we can ever reduce this co-created chaos of existence into villains and heroes - at least once we're beyond the tooth fairy and father christmas stage anyway...
You are so welcome Caro. To be honest it wasn't until I researched this article that the loops I talk about began to make 'sense'...
Wow! I am definitely saving this for later. Amazing work!
Thank you Ileana. 🙏