Goodhart's law and metrics pitfalls

Devansh warns that metrics become manipulable; he examines resilient metrics and the need for non-deterministic methods.

Play episode from 11:39

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Guest

Devansh Mehta

Panelists

Eriol Fox | Victory Brown

Show Notes

In this episode of Sustain, host Eriol Fox and co-host Victory Brown are at Devconnect Conference in Buenos Aires with Devansh Mehta from the Ethereum Foundation, to unpack one of the hardest problems in open source: how to fund the public good infrastructure that everything else depends on fairly, ethically, and at scale. They dig into quadratic funding, “credit assignment,” dependency graphs, Goodhart’s Law, and how AI can help, without taking over. Also, why open networks still struggle to compete with corporations and what new funding mechanisms like Deep Funding are trying to change. Hit download now to hear more!

[00:00:22] Eriol introduces Devansh, and he tells us about the work he does at Ethereum Foundation.

[00:01:32] He explains two core problems: Funding loop and Credit assignment.

[00:03:57] He identifies two failure modes: Popularity contests and lobbying & favoritism and shares why he found quadratic funding very liberating.

[00:05:48] Devansh uses Bitcoin as a simple model: miners get all the credit for a block and the new BTC is the funding loop.

[00:06:51] He defines public goods as value created minus value captured and argues the real challenge is linking revenue centers to cost centers.

[00:09:19] Devansh proposes a 3-step model for connecting revenue and OSS dependencies: Build an accurate dependency graph, weight the edges, capturing “how much value I get from you, and send money into one address and let it flow through the graph by weights.

[00:11:28] Goodhart’s law is explained, and Devansh warns metrics like stars/downloads break once tied directly to money and he gives some solutions to use non-deterministic AI and human judgement.

[00:16:04] Victory wonders how we can make this more ethical. Devansh notes that experts have the biggest conflict of interest, and he introduces cryptographic ideas: Confusion and Diffusion.

[00:18:27] Devansh analogizes funding mechanisms are like recommendation algorithms and critiques the current RFP/grant system common in non-Web3 open source.

[00:21:01] Find out where you can follow Devansh on the internet and he shares the Ethereum Foundation believes in the “policy of subtraction” and highlights some key partners in deep funding: Seer, Pond, and Drips.