Frankly, not sure where to begin with this one. Some here have already pointed out how it can easily be trained to produce racist/biased data which is a big red flag to begin with. But am I the only one thinking about how this deep-learning, AI algorithm is going to render millions obsolete at the behest of capital? As has been the case with almost everything developed under capitalism , what a marvelous discovery that will in all likelihood be used for nothing but exploitation. And do I even have to mention how our leaders are geriatric and couldn’t regulate this shit to save their lives?

Unless this is somehow made open-source (and soon), we’re fucked.

    • BabaIsPissed [he/him]
      ·
      2 years ago

      Yep, no disagreement about the fine-tuning stuff of course. I actually misremembered some stuff that bothered me about a claim in the paper. I like to annotate stuff as I read and oftentimes I'll complain about something just to have it answered a page or so later.

      TL;DR: I'm dumb

      Our speculation is that a language model with sufficient capacity will begin to learn to infer and perform the tasks demonstrated in natural language sequences in order to better predict them, regardless of their method of procurement.If a language model is able to do this it will be, in effect, performing unsupervised multitask learning.

      Maybe (probably) I'm dumb but I thought: can they really claim that? If a model sees, for example, a bunch of math operations and produces the correct output for such tasks, is it more likely that it picked up in some way what numbers are, what math operators do and how to calculate or that it simply saw ('what is 2+2?', '4') a bunch of times? Can we really say it's like a multitask model where we know for a fact it's optimizing for multiple losses? The catch is that they did some overlap analysis later on and their training set covers at most 13% of a test dataset and the model did pretty well in a zero-shot context for most of the tasks, so seeing the answers in the training set doesn't really explain the performance. So yeah, I guess they can claim that lol.

    • hexaflexagonbear [he/him]
      ·
      2 years ago

      Since BERT the state of the art for almost any NLP task has been taking these pre-trained large language models and fine-tuning them for the specific task you want to do.

      I might be mistaken, but I believe it's more than just fine tuning. It's fine tuning so it picks up on the different context it's getting used in, but foe any non-trivial application there are additional machine learning systems attached to it. So for example drawing based on prompts would have to have a system capable of doing the "draw X in the style of Y" type tasks.