Exploring Model Scaling in Natural Language Processing and Discussing Non-Differentiable Information Storage

Natural language processing expert and PhD student Melanie Subbiah sits down with Jon Krohn to discuss GPT-3, its strengths and weaknesses, and the future of NLP.

In this episode you will learn:

• What is GPT-3? [6:24]
• The strengths and weaknesses of GPT-3 [14:38]
• What is autoregression? [18:03]
• GPT-3's new fine-tuning abilities [20:02]
• Bias issues with GPT-3 [22:47]
• The future of natural language processing models [27:54]
• How Melanie ended up working at OpenAI [38:13]
• Melanie’s self-study process [42:19]
• Melanie's work on OpenAI API [45:45]
• How to address the climate change and bias issues that cloud discussions of large natural language models [49:40]
• Why Melanie chose to do a PhD at Columbia University [1:01:17]
• The machine learning tools Melanie’s most excited about [1:08:09]

Additional materials: www.superdatascience.com/559