(author here)
That's an interesting take (which I agree with).
Providing a quick way to stress test the model is definitely a double edge sword.
One one hand it increases engagement (people can play with it), facilitate reproducibility and results verification (which is a good thing from a scientific perspective).
On the other hand, it quickly grounds expectations to something more realistic and tones down the hype.
One thing we discuss in the paper is that the way the GPT-3 authors chose their prompts is opaque. Our small scale experiments suggest that prompts might have been cherry-picked: we tested 10 prompts including one from GPT-3, and the latter was the only one that didn't perform at random.
Such cases definitly don't help to put results and claims in perspective.
> Providing a quick way to stress test the model is definitely a double edge sword.
I hope you don’t second guess or regret the choice to make the announcement so accessible. It’s a really good thing to have scientific communication accurate and accessible, especially when those two things go together.
I don't have exact numbers for latency but the inference widget is currently on a TPU v3-8 (which if I am not mistaken could roughly be compared to a cluster of 8 V100). That gives you a rough idea of the latency for short inputs.
Providing a quick way to stress test the model is definitely a double edge sword. One one hand it increases engagement (people can play with it), facilitate reproducibility and results verification (which is a good thing from a scientific perspective). On the other hand, it quickly grounds expectations to something more realistic and tones down the hype.
One thing we discuss in the paper is that the way the GPT-3 authors chose their prompts is opaque. Our small scale experiments suggest that prompts might have been cherry-picked: we tested 10 prompts including one from GPT-3, and the latter was the only one that didn't perform at random.
Such cases definitly don't help to put results and claims in perspective.