Raw LLM Responses

Yudkowsky: An AI will create subgoals which are not foreseeable by the designers. Most of these subgoals will include killing humans. And for humanity to end, it is sufficient if only one of the subgoals includes killing humanity as a side effect. Problems with this argument (in arbitrary order): 1) It is not clear that the jump from AI as smart as humanity, to AI at a level ridiculously smarter than humanity, happens so fast that humanity could not adapt ( e.g. shut it off). 2) Killing humanity is not impossible, but really hard, if humanity can have a look into your brain at any time, analyze you with the help of other AIs, and shut you down or modify you whenever they want. 3) High Intelligence alone does not automatically mean that entities are effective at reaching goals. Look at many intelligent humans ... 4) Humans follow goals and generate subgoals in a similar unforeseeable manner. But these subgoals almost never include killing all of humanity. 5) It should be possible to add high-level goals which eliminate a lot of all potential dangerous subgoals by means of contradiction. 6) Just because the subgoals are not foreseeable, does not mean that most of them will include killing humans. They are unforeseeable and not arbitrary. 7) It is not clear that subgoals are not balanced out by other subgoals. 8) There might not be one AI, but actually several of them balancing each other. 9) .. Probably there are more .. This discussion should have addressed this kind of counter-arguments. Instead Yudkowsky did not actually try to convince anyone, and Wolfram did not address the obvious problems with Yudkowsky's theory, but instead explored all kinds of "1+1 may not be 4 because who knows what a 1 is, in the first place?"