Raw LLM Responses

AI for physics researcher here! There are a few claims here that I'm a bit "eh" on: 1) there are some papers that suggest that "reasoning traces" aren't necessarily correlated with how models are actually operating (one is from Anthropic, called "Reasoning Models Don't Actually Say What They Think"). Personally, I think reasoning models are a bit of a false path for interpretability- they don't say what they think, they spit out text that convinces you they know what they think. They don't actually think at all! 2) Hallucination doesn't really have much at all to do with what AI "knows"- it doesn't have a stateful internal memory, it doesn't "know" things are true or false at all. Training a model is kind of like trying to carve a rock so it replicates human speech when the wind blows through it at a certain angle- it's a system designed to produce a response that is believably human, but it doesn't have a survival drive or a genome or neurotransmitters, it's a system of little plinko pegs that maps an input to an output (which sometimes is still super useful, for things like building better simulators where the hard part is sampling from a big gnarly difficult distribution!) 3) I really caution against anthropomorphizing AI in any sense, because the way these things work is not like a brain- I doubt we'll ever have an interpretability chart that looks like the one described, but we will have progress that looks like information-theoretic stuff about like when different features emerge during different training stages (which we can talk about as like... blobby Arrival-alien style phase transitions in a crystal). Look into things like "Transformer Circuits" or "grokking" (not the Elon chatbot, the phase transition) if you want to get a sense of some recent progress in interpretability research along these lines! Even thinking about models as having "side goals" that they follow to help minimize their loss function isn't really accurate- they're not following a goal drive, the singing rock doesn't know anything about what humans are, it's just subject to quirks of erosion. However, sometimes *we* as humans can extract useful knowledge from how models arrange stuff- models that learn math can teach us weird new ways to think about math- but that's a whole other can of worms 🙂 Also, I tend to believe that a lot of the Everyone Dies-ness can be averted by JUST NOT PUTTING AI IN SO MUCH STUFF. Superintelligence isn't a meaningful concept because human intelligence isn't linear (what does making a model that is "better than the best painter" or "the best mother or "the best dentist" even mean???). These things don't work like Ultron where they get in the wires and go rogue- the only way you get an LLM that launches nukes is if you give an LLM the keys to the nukes, so we should just stop putting them in charge of stuff that we don't expect to go wrong. Josh Clymer has a good "safety case" framework about when it makes sense to put AI in a system-- if failure happens, who is harmed and how do you catch it, when do you have a human in the loop and what is their role, etc.- that I like a lot. Anyway text wall over! I'm really glad you're doing scicomm about this, this is a niche that has been really under-communicated about for a while and the grifters have been having a field day about it