A few months ago, OpenAI unveiled GPT-2. It’s a machine learning model built from 8 million web pages. GPT-2 ingested these pages and modelled the relationship between their words. It could then regurgitate the words in new ways. Including by writing convincing fake news.
So OpenAI was careful. The full version of GPT-2 has 1.5 billion parameters. (The more parameters, the more complex the model.) OpenAI began by releasing a 117 million parameter model. (About 8% of the full version.) We didn’t see an exponential increase in fake news. So they released a 345 million parameter model. (About 23% of the full version.) We still weren’t inundated with machine-generated nonsense. So now, after 6 months, OpenAI has released a 774 million parameter model. (More than 51% the full version.)
I wanted to see how much better each model is at helping me write blog posts. So, here’s a test, using the prompt “In today’s blog post, I’m going to write about.” How did the models do?
117 million parameter model: Loses the plot
In today’s blog post, I’m going to write about Gulf states hosting $58 trillion worth of oil. Of course, I know there are more productive ways to do it, but one thing is clear: there are other ways to roll them up. 41. Fossil fuels for rural development The forests served by every pipeline plant will account for more than 50,000 miles of roads. Most of them will be…
345 million parameter model: Better, but inconsistent narrative
In today’s blog post, I’m going to write about how to rig items out of the Top Pick:Character selection option. We primarily use this option to choose Vehicles for Engineers based on their faction. By setting the :Choose for vehicles option to ‘True’, it forces the :When this option wasn’t invoked before allowing the “If this was interrupted before that cutoff or if all subsequent requests failed when finishing the specific task, simply…
774 million parameter model: Coherent, interesting, and creative
In today’s blog post, I’m going to write about how I reinvented an old MacBook accessory. Lately I’ve been working on transforming a standby USB charger into a USB 3 hub. The idea was to combine the operation of an iPad, a phone, car charger, mouse, desktop keyboard, PC, and laptop to easily remove the original equipment of a Mac. (They’d be distorted and disregarded!).”
Granted, none of these are great ideas for a blog post. But you can see in the 774 million parameter model the ability to construct a logical narrative. You can also see the ability to be creative. It proposed an idea (though unworkable) for a USB 3 hub built from various gadgets.
I’m looking forward to experimenting with ever-larger models.
PS: You can play with this yourself in Google Colaboratory. Someone made this notebook. You just need to add the 774 million parameter model to the download area at the top.