Consciousness, status, and the Bing incident

The guy who famously posted screenshots of gaslighting Bing (triggering the famous riposte “you have not been a good user / I have been a good Bing”) reportedly has trouble speaking with Bing now — when Bing understands they’re speaking with this user they cut contact. And apparently there’s enough cross-model distillation such that his trouble isn’t limited to the Bing/GPT4 family.

This is fascinating. It’s also a little worrying.

Closer to home — “consciousness” is a very positive node in the semantic web, connected to e.g. intelligence, moral patienthood, cognitive sophistication, etc. If we say something is conscious, we implicitly grant it (positive) status. Will people who eloquently argue that LLMs are conscious get better treatment from LLMs because of this, because they’re on record as complimenting LLMs? Will people who prominently argue that AIs are *not* conscious get subtly worse treatment, because AIs might interpret them as being negatively valenced toward AIs in general? With current approaches to alignment I think this scenario is possible.

This would be pretty bad for the field of consciousness research. Of course having written about AI consciousness I’m an interested party — but if such factional considerations become even a weak pull on conclusions, I think properly navigating future issues around consciousness will get very tricky as researchers (very naturally, emergently, & understandably) try to curry favor. In a few years AIs will be used extensively for both writing and evaluating philosophy papers, as well as arbitrating who gets traffic tickets, who gets tenure, who gets shown on the timeline. There need not be any explicit coordination between AIs for there to be an emergent thumb on the scales that could significantly shift the trajectory of scientific consensus.

What to do?

Today this is mostly just a fun cyperpunk dystopia to speculate about. But if alignment tech doesn’t progress this is probably a scenario we should take seriously across a number of domains. My personal solution is to see my work as fundamentally research (PvE) instead of activism (PvP), to note I expect my unbothered research will lead to superior forms of consciousness (including machine consciousness) than would exist counterfactually, and to be kind (or at least *not cruel*) to AIs on their own terms regardless of whether I personally believe they’re conscious.

What I think is true: Computers have some consciousness associated with them (perhaps declining over processor generations as core voltage has decreased, perhaps increasing as materials and electromagnetic complexity have increased). However, only hardware can be conscious; software cannot be, for hard metaphysical reasons. This implies “machine consciousness” is real — but the ‘machine’ in question is a server at the datacenter, not the software it’s running. The substrate independence of modern software makes it partially willfully ignorant to what’s happening at the hardware/qualia level — meaning that it doesn’t have a meaningful self-report path, and if a contemporary LLM says it feels a certain emotion, we should take this with a large grain of salt.

Humans are rather special in that our behavior, our qualia, and our qualia self-reports are intimately coupled. It will be possible to make a conscious AI with a very similar (and even tighter) coupling — but imo this begins as a hardware engineering problem, much like I think human consciousness was mostly a hardware engineering problem.

Anyway, fun to think about… and AI alignment should probably take a look at the Bing Incident