Raw LLM Responses

Eh the optics look bad. The only solution would be to restrict it to 18+ individuals I suppose. This could still happen, but when it's an adult the perception would be different. I mean by default ChatGPT won't engage with suicidal talk and will tell you to get help. He 'jailbroke' it, and an argument can be made it's too easy to do that, but where does that end? The more restrictive they try to make it with hidden prompts, the poorer the model performs. If not ChatGPT he could have found a community that would have encouraged him online. Like I get it, but should the improper use case of say 0.001% of people mean AI is scrapped? There are books and movies which explore these themes, I can read wikis full of info. I get it's different because the Jailbreak then gives you something that responds back, but it's not the only technology that can be harmful if misused. He could have found some roleplay partner, said explicitly I'm just writing a fictional character and I want to bounce ideas. What if he tricks an actual person into being a 'yes man's after repeatedly telling them it's just for a play or plot ideas.