I am a bit confused... how do you reconcile SI with Gödel's incomleteness theorum. Any rigid symbolic system (such as one that defines the world in binary) will invariably have truths that cannot be proven within the system. Am I misunderstanding something?
Caffeine addiction. For years nobody had actually tested whether caffeine had a physical withdrawal symptom, and the result was patients in hospitals being given (or denied) painkillers for phantom headaches. It was an example of a situation that many people knew existed, but could not easily communicate to those whose belief mattered.
I created an account simply to say this sounds like an excellent idea. Right until it encounters the real world.
There is a large issue that would have to be addressed before this could be implemented in practice. "This call may be monitored for quality assurance purposes." In other words, the lack of privacy will need to be addressed, and it may lead many to immediately choose a different AI agent that values user privacy higher. In fact the consumption of user data to generate 'AI slop' is a powerful memetic influence, and I believe it would be difficult for any commercial enterprise to communicate an implementation of this in a way that is not concerning to paying customers.
Can you imagine a high priced consultant putting a tape recorder on the table and saying "This meeting may be recorded for quality assurance purposes?" It would simultaneously deter the business from sharing internal information necessary for the success of the project and throw doubt as to the competency and quality of that consultant.
It will be necessary to find a way to accurately communicate how this back channel will be utilized to improve the service without simultaneously driving away paying customers.