I don't know the internals of Bing, but they have some triggers which themselves seem to be made with NLP. They use it a lot to fetch web info. That means that if the model somehow produces some creative version of a crime that doesn't get caught, it'll just send.
I think this is why Bing sometimes refuses to continue the conversation, or ChatGPT will flag its own text as against their terms sometimes. But yeah, they definitely can encourage how to crime sometimes, I've made ChatGPT explicitly tell me how to replicate some crimes like the Armin Meiwes cannibalism one while bored.
I don't know the internals of Bing, but they have some triggers which themselves seem to be made with NLP. They use it a lot to fetch web info. That means that if the model somehow produces some creative version of a crime that doesn't get caught, it'll just send.
I think this is why Bing sometimes refuses to continue the conversation, or ChatGPT will flag its own text as against their terms sometimes. But yeah, they definitely can encourage how to crime sometimes, I've made ChatGPT explicitly tell me how to replicate some crimes like the Armin Meiwes cannibalism one while bored.
The legal cases are going to be fun reading when they come out!