Considering that most of Sillicon Valley believes in this exact same shit with slight modifications, they probably didn't think it was too incorrect to be removed from the dataset.
1 - a significant chunk of the internet. It's way too large to sanitize
This is the same as GPT3. It reproduces things that appear on the internet, not the chatbot format people want/expect. It'll respond to questions with a list of questions, because that's where questions are usually found on the internet.
2 - samples of text where someone wrote things in question-answer format with the answer being the kind of confidently corporate encyclopedia that we know ChatGPT to be
Just a pile of people manually trying to mimic the chat bot they want. This isn't the researcher's first rodeo, so I'm sure it included lots of samples of people asking racist questions and the bot responding like a good little boy.
ChatGPT is trained to prioritize this training data over #1, but the data from #1 is way larger and more thorough, so you can break out of this by phrasing questions slightly differently than the manual ones, or speaking in pig latin, or whatever, since it'll fall back to its wider knowledge.
3 - a separate network trained to determine whether an answer is good or bad based on a bunch of manual samples
So they used the 1+2 version, got it to say something awful, put that in as a bad example, etc. If this network detects a bad answer it'll re-roll it, copy paste in the "as a machine learning model I can't" thing, and so on.
So they have these two extra layers to try to stop it from reading out Stormfront's Favorite Statistics but the core of it is still a giant heap of raw unfiltered internet.
That's really interesting, simultaneously more ingenious and less impressive technically than I was imagining. I'm sure insisting on sanitized data sets would make it extremely limited, but sometime I think if that's the trade off maybe just don't make the thing. Thanks for effort posting, even though you did ruin my joke.
Of course the real problem IMO is not that some terminally online nazi manipulated it like this, but that it will uncritically regurgitate whatever the most orthodox (for the internet) opinion on any subject, with no context and the veneer of impartiality.
So far, the bigger the training data set the better, and that's one of the biggest things that determines how well the model works. That's not an absolute - I'm sure that removing all the racist shit from the internet corpus would make it better overall. But the problem is how to get "the internet minus racist shit" instead of "this much smaller dataset that we've manually screened all the racist shit out of." You could make an AI do it, but where are you going to get a non-racist AI to go do that for you?
simultaneously more ingenious and less impressive technically than I was imagining
If you really want to dig into it, Andrej Karpathy did a video on how to make a GPT from scratch. It's less than 1000 lines of Python code, maybe 200 of which is actually the machine learning model (the rest being training, stuff to make using and training it easier, etc). The full-sized GPT3 is still the same model, just with bigger numbers and even more scaffolding stuff (to get it to run nicely on computing clusters, etc).
In terms of technical background needed: Understanding matrix multiplication is really important. At least a vague idea of how computer programming works, but it's short and Python and mostly math, so you could puzzle it out. Karpathy's video also treats it as a given that you can just optimize a matrix, which is possible because there's an automatic differentiator built into this, which lets you just move vaguely towards some min/max (called gradient descent now because they want to sound fancy; back in my day they called it hill climbing).
I think they just gave it a whole bunch of internet data. So I'm not surprised it knows what a racist would say
after all considering the the context where the phrase "average inteligence of ethnicities" will appear on the internet it's not surprising that the model associates the phrase with racist text
I din't think the facts are in the training set, the training set was to get it to reasonably parse text, the facts it "knows" are whatever it finds online.. which of course is going to be dumb bullshit half the time.
I didn't know that, so it's basically like those Google suggested answers for questions, but combined with a natural language text generator? I assumed it was a purely predictive model, like a souped-up markov chain.
I think there's a lot going on under the hood on the NLP portion, because it does have to group stuff into concepts so that it brings in conceptually similar results. But I don't believe it's pretrained on all the stuff it can answer to.
They couldn't take the phrenology textbooks out of the fucking training set?
Considering that most of Sillicon Valley believes in this exact same shit with slight modifications, they probably didn't think it was too incorrect to be removed from the dataset.
Thinking about it, I'm 99% sure every techbro who worked on this horror has a fucking copy of The Bell Curve on their bookshelf.
Judging by how they react to anything left wing, yeah I'd say you aren't too far off.
No you see, whites are only at level 3) so its not racist
Even racism's gone woke smh
That's why it's called ""race realism"" and not eugenics. Totally different, you see? There are actually people who believe this, unfortunately.
ChatGPT is basically three layers:
1 - a significant chunk of the internet. It's way too large to sanitize
This is the same as GPT3. It reproduces things that appear on the internet, not the chatbot format people want/expect. It'll respond to questions with a list of questions, because that's where questions are usually found on the internet.
2 - samples of text where someone wrote things in question-answer format with the answer being the kind of confidently corporate encyclopedia that we know ChatGPT to be
Just a pile of people manually trying to mimic the chat bot they want. This isn't the researcher's first rodeo, so I'm sure it included lots of samples of people asking racist questions and the bot responding like a good little boy.
ChatGPT is trained to prioritize this training data over #1, but the data from #1 is way larger and more thorough, so you can break out of this by phrasing questions slightly differently than the manual ones, or speaking in pig latin, or whatever, since it'll fall back to its wider knowledge.
3 - a separate network trained to determine whether an answer is good or bad based on a bunch of manual samples
So they used the 1+2 version, got it to say something awful, put that in as a bad example, etc. If this network detects a bad answer it'll re-roll it, copy paste in the "as a machine learning model I can't" thing, and so on.
So they have these two extra layers to try to stop it from reading out Stormfront's Favorite Statistics but the core of it is still a giant heap of raw unfiltered internet.
That's really interesting, simultaneously more ingenious and less impressive technically than I was imagining. I'm sure insisting on sanitized data sets would make it extremely limited, but sometime I think if that's the trade off maybe just don't make the thing. Thanks for effort posting, even though you did ruin my joke.
Of course the real problem IMO is not that some terminally online nazi manipulated it like this, but that it will uncritically regurgitate whatever the most orthodox (for the internet) opinion on any subject, with no context and the veneer of impartiality.
So far, the bigger the training data set the better, and that's one of the biggest things that determines how well the model works. That's not an absolute - I'm sure that removing all the racist shit from the internet corpus would make it better overall. But the problem is how to get "the internet minus racist shit" instead of "this much smaller dataset that we've manually screened all the racist shit out of." You could make an AI do it, but where are you going to get a non-racist AI to go do that for you?
If you really want to dig into it, Andrej Karpathy did a video on how to make a GPT from scratch. It's less than 1000 lines of Python code, maybe 200 of which is actually the machine learning model (the rest being training, stuff to make using and training it easier, etc). The full-sized GPT3 is still the same model, just with bigger numbers and even more scaffolding stuff (to get it to run nicely on computing clusters, etc).
In terms of technical background needed: Understanding matrix multiplication is really important. At least a vague idea of how computer programming works, but it's short and Python and mostly math, so you could puzzle it out. Karpathy's video also treats it as a given that you can just optimize a matrix, which is possible because there's an automatic differentiator built into this, which lets you just move vaguely towards some min/max (called gradient descent now because they want to sound fancy; back in my day they called it hill climbing).
I think they just gave it a whole bunch of internet data. So I'm not surprised it knows what a racist would say
after all considering the the context where the phrase "average inteligence of ethnicities" will appear on the internet it's not surprising that the model associates the phrase with racist text
I din't think the facts are in the training set, the training set was to get it to reasonably parse text, the facts it "knows" are whatever it finds online.. which of course is going to be dumb bullshit half the time.
I didn't know that, so it's basically like those Google suggested answers for questions, but combined with a natural language text generator? I assumed it was a purely predictive model, like a souped-up markov chain.
I think there's a lot going on under the hood on the NLP portion, because it does have to group stuff into concepts so that it brings in conceptually similar results. But I don't believe it's pretrained on all the stuff it can answer to.