Raw LLM Responses

Based on the excellent book "Human Compatible: Artificial Intelligence and the Problem of Control" by Stuart Russell, there are other problems with smarter-than-human-AI, where smart is defined as being effective at reaching ones goal. The most important one is probably that we only get one chance to get it right. If not constructed correctly, such an AI would tend to 'overshoot' by being too effective at reaching its goal, as in the saying 'be careful what you wish for'. In other words, we might get more of the intended effect than we anticipated and we wouldn't be able to dial it down/reprogram because such an AI would anticipate this and make sure that it can't happen as that would go against its purpose. That is also why you can't just unplug it (it would prevent being unplugged and thereby failing it's programmed mission). The human flaws that might slip into the programming/be reproduced in the AI, as you mention, could obviously be a cause of such failed programming and humans could exploit it as long as they are smarter than the AI but in the end, it would become uncontrollable. Russell gives some elements which would be required to prevent such an out-of-control AI, such as the need to align its goals with those of humans by inserting doubt/uncertainty and requiring human feedback. Side thought (which I repeat from an interview with Yuval Harari): a very worrying aspect of the current Ukraine war and global polarization is that the current advances in AI require international cooperation, exactly to prevent the potential devastating consequences but instead, they are being militarized within the framework of a global competition (not saying that the unsupervised development of AI by Google, Meta and the likes is much less worrisome).