Vanishing Gradients Episode 31: Rethinking Data Science, Machine Learning, and AI
20 snips
Jul 9, 2024 In this discussion, Vincent Warmerdam, a senior data professional at :probabl, challenges conventional data science approaches with innovative insights. He emphasizes the importance of real-world problem exposure and effective visualization. The conversation dives into framing problems accurately and determining if algorithms truly solve them. Vincent advocates for simple models, discusses the role of UI in data science tools, and examines the potential and limitations of LLMs. He highlights the need for community knowledge sharing through blogging and open dialogue.
AI Snips
Chapters
Transcript
Episode notes
Non-Mutually Exclusive Classes Example
- spaCy's approach to NLP classification questioned the default mutual exclusivity assumption.
- Vincent found non-mutually-exclusive labels better reflect real-world situations like images containing both cats and dogs.
Limitations of Probability as Confidence
- Prediction probabilities alone are unreliable confidence measures.
- Combining confidence scores with outlier detection improves decision automation safety.
Logistics Optimization by Redefining Problem
- World Food Organisation improved logistical allocations by focusing on nutrients instead of specific foods.
- This insight led to significant cost reductions with simpler algorithms aligned to reality.