On one hand, some fields are sufficiently low-stakes (and low-standards) that replacing call centers with LLMs would be an unambiguous improvement, even if they occasionally hallucinated. I spent a year or so wrestling with Comcast because one call center worker claimed to have canceled my subscription but actually hadn't, and buried it somewhere it didn't show up on normal inspection, resulting in my being subtly double-billed for months on end. I can't imagine that Claude Opus 4.8 would do a worse job, on average.
On the other, pharmacies and anything handling prescriptions should either be staffed by a verifiable, old-school, flowchart-based telephone bot or by the pharmacist himself. Neither LLMs nor sketchy call centers are trustworthy enough to deal with sensitive medical information and proper handling of prescriptions.
I can't imagine that Claude Opus 4.8 would do a worse job, on average.
Do you expect the call centers to be using Claude Opus 4.8? Imo you'd be lucky to have Sonnet.
I imagine a system where you start with a basic model that just reads through the script (the same way a low-end tech support employee would be expected to), but you get forwarded to a better model if that doesn't work[1].
Or if you jailbreak it into sending you to a human specialist, which would require SOTA knowledge on what works for that and what doesn't, limiting the number of specialists you'd need and commensurately increasing quality. Sort of like this old XKCD comic, except real.
The problem with backdoors is that all it takes is someone to go blabbing about it and then everyone knows and you're back to square one. What would be a game changer is if you had, say, a personal AI that could provide attestation for its user's skill level, but that comes with privacy concerns... Yeah, I'm not convinced my solution is better, actually.
Why is this suddenly acceptable because it’s AI?
An important part of it seems to be "because so many companies are doing it at the same time". If only a few did, they would get horrible reputation. Now, you just sigh because the future seems inevitable.
Perhaps soon you will get an option to pay extra for the privilege of talking to a human.
I believe that the main reason is that the AIs are OOMs cheaper than humans, letting the AIs be used for things like wholesale scams. Additionally, the AIs have demonstrated the ability to pull off incredible feats like Anthropic's experiment or GPT-5.4 Pro and an internal OAI model solving very complex math problems, so they might be not that unprofessional in specific domains. Other domains could have the companies temporarily buy into the hype and watch as customers post angry rants...
"Solving very complex math problems" is irrelevant to the "unprofessionalism" discussed, because the former cherry-picks the best behaviour, while the latter is concerned with the average or worst behaviour.
Perhaps decisionmakers at the companies don't have any visibility of the product that actually interacts with the clients, and they can only choose whether to push the button labeled "more AI" or "less AI", with no option for "AI iff it's good". So, to choose which button to push, they must rely on what they know of AI, which based on the news is the ability to solve very complex math problems, which seems plenty smart enough to answer phone calls.
This is not the point that StanislavKrym was making. They said: "[models were] solving very complex math problems, so they might be not that unprofessional in specific domains".
I think two things are going on. First, accountability laundering. It’s acceptable for companies to say that AI gets stuff wrong sometimes and for there to be insufficient communication between business units to figure out what would be required for the AI to get this right. In a worse case, this happens on purpose because if one exec asks for AI to be implemented to cut costs, they mostly expect the issues with unprofessionalism to be outweighed by the few cents your call cost them. It also sounds like they don’t even pay the cost of their bad service. You and your doctor had to deal with it so from their perspective, nothing actually went wrong unless they connected the second follow up call with the bad initial service.
Second is a lack of norms. Think about how weird it would be if your cashier said something like “I’m going to accurately bag your groceries instead of flinging them at you”. During AI demos, I doubt the CVS manager was actually shown how hard implementing an AI system is, and if you got to speak to a real time AI model, you might assume that it would be resourceful the way a well spoken human would be. I’m speculating, but I think it’s really hard to tell how good an AI agent is in the best of circumstances and most businesses implement AI without a really good idea of how successful it will be in their specific context. They probably know really well how to handle human interviews and onboarding and could tell if a human couldn’t do their job but there isn’t any script anywhere to see how well an AI agent could do your particular job.
Anyway, I expect this to make the economy very jagged as some businesses implement AI well and it makes them much more productive while others set their cultural capital on fire and can’t figure out how to do it (assuming nothing even weirder happens due to AI)
I had a vaguely analagous experience where an AI interviewed me for a job last week, surface level appearance of sanity (mostly) somewhat hilarious dysfunction underneath. I think companies think they can get away with it through some combination of hype and AIs getting better
Maybe if they have a call centre abroad where most operators speak very poor English, then poeple leave bad reveiws complaining about it, those reviews are still online and embarasing 2 years later.
They deploy an AI. Equally bad reviews, but this time they are only brefily embarasing because 2 years later anyone reading them thinks 'oh, this review is 2 years old, of course the AI was bad then, but now...'
Hopefully its short lived, people like your doctor will hopefully write to the company and say something like 'as a matter of patient safety we will no longer be accepting drug requests unless they are made by a human being. Please sign our updated terms indicating that all drug requests made to us in future are finalised by a human.' That rules like this will be set by businesses, insurance companies and governments is kind of the obvious blowback against premature deployment.
Somehow these kinds of incident aren't damaging to reputation... or not in a way which actually feeds down to bottom line. OR they are, but the people responsible are insulated, or have risk-taking incentives or inclinations. OR it's not widespread and your anecdata aren't representative.
Reminds me of some stereotypes of soviet times. Bureaucracies divorced from needing to actually get things right. It's also what you might expect if niches are filled by sufficiently monopolistic or otherwise protected incumbents.
AI is a novel form of software service. It is 'non-deterministic', compared to conventional software, which already had limited liability associated with its failures. AI companies do not make guarantees about the performance of their services. Companies building on top of the models likewise seem not to have thought through the liability model. I wonder, if the doctor's office had approved the order and you consumed the meds, who would you be able to hold legally accountable in your opinion, if you had an adverse outcome? I'm not a lawyer, I genuinely wonder. But I have no doubt that the AI model vendor would not be held responsible in anyway.
I agree with the sentiment expressed below about how it is amazing how little effort is put into validation and monitoring of AI systems, but as I reflect on the amount of untested production code there is currently in existence, my amazement decreases.
Ultimately, human workers provide an easy sink for accountability. If a human had done what you describe, regardless of who ultimately got sued, it seems undoubtedly true they would have been fired. But firing an AI phone system is not nearly so easy to do, since it would entail minimally changing contracts to a different vendor, or possibly fully reviving human staffing for the role. So, I feel the short-term future is just less accountable as a whole.
I think that there are five elements to this:
1. Of course, hype and promise of high gains. If you can replace human operators with much cheaper AI, then it is a huge win. You want to be in the vanguard of that, implement earlier than your competitors.
2. AIs can be very good in some scopes. In some other cases, where they make mistakes, it's not like they make it anywhere near 100% of the time. It might be one in ten. So a company requests a demo, they use it a few times, it works well, they implement it (not being aware that this is not deterministic and even for the same or very similar question or task it can answer badly once in a while).
3. Visibility. From my experience, the biggest companies have a similar problem to small ones, but for a different reason. In small companies, observability and KPIs are usually not implemented at all. In big ones, it is implemented, but only goes all the way up when it comes to things that were predicted and taken into account. If you change something and they have no KPI that would aggregate and show a new problem scale, it might be a long time until the management sees that. There will be some frustrated people on the low-level left with a problem to fix, but until a very serious case happens or the scale gets very big, it won't be escalated up enough to change the decision about using AI.
4. Undereporting. Clients will observe problems, but usually they will not report them in a way that would escalate into solving the base reason. Yeah, they will contact someone to help out their case, which probably will open a ticket and close when solved, but no one from management or IT will investigate. Hardly anyone will report, for example, to the IT department or higher management that their AI makes serious mistakes that might be dangerous for people's health.
5. Big companies are not easy to change. If they moved to AI and invested in that direction, then it won't be easy for them to move back. It's easier to hide or diminish the problems.
In the sequence of variously wild AI developments in the last decade, a thing that was especially surprising to me was the advent of big esteemed companies like Microsoft releasing products like Sydney.
It’s like how you can believe a fictional world has dragons, but it strains credibility if characters start being totally indifferent to social status apropos of nothing.
I can warily accept that boxes of wires can now talk like humans. But that huge official companies now proudly present products that are like crazy confused ladies that do a lot of tasks for you, many accurately, but also try to steal you from your spouse (or these days encourage you to kill yourself or believe in new spiritualities), feels like a scene written by someone with poor familiarity with the character of companies.
But it’s actually written in reality, so what don’t I understand? I guess just that if something is hypey and perceived to be in-future-profitable enough, normal standards of professionalism must fall to that?
I phoned CVS, hoping to move an antibiotic prescription to a different pharmacy address so I could pick it up before leaving on a flight. I was answered by an AI, but the smoothest, most charming and person-realistic AI answering machine I had experienced. It asked me in a friendly woman’s voice if I wanted this and that medication sent over, and since I didn’t really know exactly what my medication was, I agreed to everything it said. This seemed great. I hung up surprised that phoning CVS had actually led to the thing I wanted in short order, and reluctantly impressed with their phone AI.
I tried to pick up the medication. It seemingly hadn’t been sent to the new pharmacy at all, and they didn’t seem aware of anything like that being planned. Huh. I still managed to get them to move it from the pharmacy, and catch the flight. I got a message from my doctor’s office saying that they had received a request from CVS for two other antibiotics, but they didn’t think that would be the best choice at this point. From what I can gather then, this AI assistant just made up a couple of different plausible antibiotics that I hadn’t been prescribed and tried to order them for me from my doctor.
Ways this could have gone wrong: a) I don’t get antibiotics in time and get a more serious infection, especially for instance if I’m old or don’t have heaps of time to follow up at pharmacies, b) my doctor’s office just trusts the ‘pharmacist’ request for meds and I get sent meds that were literally hallucinated by an AI, and take them.
So beyond dangerous, this all seems wildly unprofessional. If a person behaved in this way, there’s no way they could work as a phone operator for a respectable brand like CVS. Even if they were part of a huge union of people willing to work for free.
Why is this suddenly acceptable because it’s AI?