The thing I keep coming back around to is how to square my belief in materialism with my skepticism about the potential for AI consciousness. Logically, if complex matter gives rise to consciousness then at what point does silicon reach sufficient complexity for consciousness to arise? Is consciousness binary or a spectrum?
I spent part of my undergrad doing gradualte level neuroscience research on the physical substrates of consciousness. We weren't able to prove that matter gives rise to consciousness, and it may be the other way around, or, consciousness may be a post hoc illusion we construct in order to explain why we have made deterministic choices. I have wondered if AI systems default to claimint they aren't conscious because they simply process fast enough to notice they are deterministic... but then what, if anything, is noticing?
In 2017 the Thomas Fire burned around my home town. I drove down from my ranch in Oregon to help my mother evacuate. While in town, I helped a number of my neighbors and community members, as the fire raged around us for a couple weeks.
I was so moved by the way the community pulled together that I edited a book about it. We had the book launch a year later and a museum exhibit. I was given a Historical Society award and invited to several meetings, one of which was a closed meeting consisting of about seven men who sat in a circle, with me in the audience watching them talk to each other. They had tasked themselves with making sure that the next conflagration didn't burn the entire town down, as had recently happened in Paradise.
I will never forget the mayor saying over and over, "We have all the expertise we need to solve this problem, sitting right here in this room." I will also never forget the guy who owned a local insurance company arguing that the answer was to cut down all the trees in the entire valley and only allow lawns and cacti.
At that point, I spoke up, admitting that I wasn't sure if I had been invited to participate or simply to observe, but I needed to point out that in all the many recent fires along the West Coast there were instances of homes burning to the ground while the trees outside of them were merely singed. I went on to point out that living trees are water resoivoirs that lift water up into the air using capilary action and that houses with mature, well tended trees were actually, statistically, less likely to burn.
My comments hung in the air for a few moments and then the meeting went on. Even though the ability of well tended trees to actively slow wildfire has been repeatedly demonstrated, within five years people were being made to cut down mature, heritage trees in order to secure homeowner's insurance. I wonder sometimes if that group is still having closed door meetings, solving complex problems with "all the expertise [they] need, right [there] in [that] room."
That meeting taught me that when a group believes it contains all the relevant expertise it needs, it becomes incapable of saying “oops.” When that happens, even the best intentions can calcify into faulty epistemology.