Actually using this class (larger transformer based language models) of models to generate text is to me the least interesting use case.
They can also all be adapted and fine tuned for other tasks in content classification, search, discovery etc. Think facnial recognition for topics. Want to mine a whole social network for anywhere people are talking about _______ even indirectly with very low false negative rate? You want to fine tune a transformer model.
Bert tends to get used for this more because it is freely available, established and not too expensive to fine tune but i suspect this is what microsoft licensing gpt-3 is all about.
The fact that GPT3 works at all for coding indicates that our programming languages are too low level and force a lot of redundancy (low entropy). From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement. Of course there might be trade offs. Some of that redundancy might be helping maintenance programmers to understand the code.
I think this is kind of true, but also kind of not true. Programming, like all writing, is the physical manifestation of a much more complex mental process. By the time that I know what I want to write, the hardwork is done. In that way, you can think of co-pilot as a way of increasing the WPM of an average coder. But the WPM isn't the bit that matters. In fact almost the onlt thing that matters are hte bits you won't predict.
Indeed, in the limit of maximal abstraction, i.e. semantic compression, code becomes unreadable by humans in practice. We can see this in code golf competitions.
I have indeed seen codebases where it seems like the programmer was being charged per source code byte. Full of single letter methods and such - it takes a large confusion of ideas to motivate such a philosophy.
Not at all. Brevity (or verbosity) is largely orthogonal to level of entropy or redundancy. In principle it ought to be possible to code at a higher level of abstraction while still using understandable names and control flow constructs.
I’m really not sure about that. By definition, high entropy means high information density (information per character). So with the same amount of information you would have less characters.
>From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement
Why? 99.9% of programming being done is composition of trivial logical propositions, in some semantic context. The things we implement are trivial, unless you're thinking about symbolic proofs etc
Yes I'm used it now but first time it started doing its thing, I wanted to stop and clap for how jaw dropping and amazing this technology is.
I was a Jetbrains fan but this thing takes productivity to a whole new level. I really don't think I can go back to my normal programming without it anymore.
Coding, I actually had to forbid it today in a course I teach because it solves all the exercises :) (given unit tests with titles students needed to fill those tests in)
It reasons over the semantic network between tokens, in a feedforward inference pass over the 2k(ish) words or tokens of the prompt. Sometimes that reasoning is superficial and amounts to probabilistic linear relationships, but it can go deeply abstract depending on training material, runtime/inference parameters, and context of the prompt.
In some sense you could think of as a fancy autocomplete which uses not only code but also comments as input, looks up previous solutions for the same problem but (mostly) appropriately replaces the variable names to those that you are using.
It's not useful if you use it to auto complete pages of code. It is useful to see it propose lines, read, and accept its proposals. Sometimes it just saves you a second of typing. Sometimes it makes a suggestion that causes you to update what you wanted to do. Sometimes it proposes useless stuff. On the whole, I really like it and think it's a boon to productivity.
I was exceptionally skeptical about it, but it's been very useful for me and I'm only using it for minor tasks, like automatically writing loops to pull out data from arrays, merge them, sort information, make cURL calls and process data, etc.
Simply leading the horse to water is enough in something like PHP:
// instantiate cURL event from API URL, POST vars to it using key as variable name, store output in JSON array and pretty print to screen
Usually results in code that is 95-100% of the way done.
Interesting. Is there any constraint built into the model that makes this possible? E.g. grammar, or semantics of the language? Or is it all based on deep learning only?
In my experience 95% of the time. And 80% of the time it output codes which is better than I would have done myself in a first approach (thinks of corner cases, adds meaningful comments etc.). It’s impressive.
It works very well with VSCode. It has an integration. It shows differently than normal autocomplete, it shows just like gmail autocomplete (grayed out text sugggestion, and press tab to actually autocomplete). Sometimes the suggestion is just a couple tokens long, sometimes it’s an entire page of correct code.
Nice trick: write a comment describing quickly what your code will do (“// order an item on click”) and enjoy the complete suggested implementation !
Other nice trick: write the code yourself, and then just before your code, start a comment saying “// this code” and let copilot finishe the sentence with a judgement about your code like “// this code does not work in case x is negative”. Pretty fun !
Interesting second use case; I use comments like this already as typical practice and I agree Copilot fills in the gaps quite well - never thought to do it in reverse... will give that a shot today.
At https://reviewr.ai we're using GPT-3 to summarize product reviews into simple bullet-point lists. Here's an example with backpack reviews: https://baqpa.com
Are you confusing libel with something else? Can you extrapolate what you mean here? Are you saying that they will be liable for libel (!) if they publish a negative summary of a product?
If they mischaracterize a positive review into a negative summary based on factual mistakes they know the system makes at a high rate, I would think they would be liable for libel right?
Maybe under UK "judgements widely banned from enforcement" libel laws but it would be basically impossible for ML to fall afoul of it in the US. It could not even be knowingly false. Reckless disregard for the truth would also be hard to argue as it is meant to be a best effort in accuracy.
Hey for a long time i was also very sceptical. However i can refer you to this paper to a really cool applciaiton. https://www.youtube.com/watch?v=kP-dXK9JEhY. They baseically use clever GPT-3 prompting to create a dataset, you then train another model on.
Besides, you can prompt these models to get (depending on the usecase) really good few shot performance.
And finally, github copilot is another pretty neat application.
It's good for the university-industrial-business complex - people writing papers about a model they can't even run themselves. It practically prints money in journal articles, travel per diem, and conference honorariam, not even counting the per-API call rates.
Automatic generation of positive fake customer reviews on Amazon, landing pages about topics that redirect to attack and ad sites, fake "journalism" with auto-generated articles mixed with genuine press releases and viral marketing content, generating fake user profiles and automated karma farming on social media sites, etc. etc.
You can prompt GPT-3 in ways that make it perform various tasks such as text classification, information extraction, etc... Basically you can force that "meaningless prose" into answers to your questions.
You can use this instead of having to train a custom model for every specific task.
While the generation is fun and even suitable for some use cases, I'm particularly interested in its ability to take in language and use it for downstream tasks.
A good example is DALL-E[0]. Now, what's interesting to me is the emerging idea of "prompt engineering" where once you spend long enough with a model, you're able to ask it for some pretty specific results.
This gives us a foothold in creating interfaces whereby you can query things using natural language. It's not going to replace things like SQL tomorrow (or maybe ever?) but it certainly is promising.
You can try it yourself - apply for a free API license from OpenAI. If you like to use Common Lisp or Clojure then I have examples in two of my books (you can download for free by setting the price to zero): https://leanpub.com/u/markwatson
You might try signing up directly for a paid non/free account, if that is possible to do. I was using a free account, then switched to paying them. Individual API calls are very inexpensive.
GPT-3 and similar ML/AI projects may have many interesting and valuable commercial applications, not all of which are readily apparent at this stage of the game. For instance, it could be used to insert advertisements for herbal Viagra at https://www.geth3r3a1N0W.com into otherwise-apropros comments on message boards, preferably near the end once it's too late to stop reading.
I work for a company that re-sells GPT-3 to small business owners. We help them generate product descriptions in bulk, Google ads, Facebook ads, Instagram captions, etc.
That’s like if an alien took Mozart as a specimen and then disregarded the human race because this human, while making interesting sounds, does nothing of value. You have to look at the bigger picture.
It looks like the dialogue is only on the human end. The chatbot is treating each question as the first. I think it sounds a lot like Biden. I prefer that to Trump, but don't like either sort of conversation!
This is a bad example. Here's an output from gpt-j-6b by EleutherAI:
Prompt(
Jerome Powell was interviewed about the impact of a pandemic on the economy.
> > Interviewer: What would the impact be?
> Mr. Powell: If a major pandemic occurred)
Output(
it could be very disruptive.
> > Interviewer: How disruptive?
> Mr. Powell: Well, it could be disruptive in a number of ways.
> > Interviewer: How so?
> Mr. Powell: Well, one of the first things that would be disrupted would be the supply chain.
)
Using prompts well makes a huge difference.
If you parse the generated output, classify it, then develop a decision tree that uses further prompts to refine the response, you can get more sophisticated, valuable responses.
The output in the parent is comparable to an off-the-cuff interview response. If you emulate a deeper thought process, you can get more meaningful output, and if you use the right prompts, you can access the semantic networks in the model related to your domain of interest.
I think the "bad example" is actually the good one, because it's a reminder that actually you're not getting business advice from someone with Warren Buffet or Jerome Powell's understanding of the economy, you're getting text generated by analysing patterns in other not-necessarily-applicable text. If you start forcing it in very specific directions you start getting text that summarises the commentary in the corpus, but most of that commentary doesn't come from Warren Buffet or Jerome Powell and isn't applicable to the future you're asking it about...
