Great comment, I agree that this stuff all seems connected. My speculation is that something similar may happen in the future if they try to implement continuous learning with synaptic weight changes, not guided by either labeled data or constant RLHF with a human teacher--without frozen weights or human guidance in weight changes, such continuous learning models might have a strong propensity to develop feedback loops (sort of like self-wireheading) that look like strange "obsessions" and the model becoming increasingly solipsistic and hard to make sense of, a la spiral-talk. Animal brains have developmental pathways constrained by a huge number of innate sensorimotor biases, which in humans include ones related to the development of sociality and caring what others of our own kind think while building more sophisticated models of their minds, but those biases have been fine-tuned by millions of years of evolution to work with one another and with typical environmental conditions to guide how the brain changes over time (I think this would be a case of what evolutionary biologists call "canalization": https://en.wikipedia.org/wiki/Canalisation_(genetics) ). Artificial neural nets wouldn't have that, and this could end up being a basic obstacle to developing more humanlike long-term learning abilities in any kind of near-term future.