The Nonlinear Library EA - Why Yudkowsky is wrong about "covalently bonded equivalents of biology" by titotal
Dec 6, 2023
17:04
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Yudkowsky is wrong about "covalently bonded equivalents of biology", published by titotal on December 6, 2023 on The Effective Altruism Forum.
confidence level: I am a physicist, not a biologist, so don't take this the account of a domain level expert. But this is really basic stuff, and is very easy to verify.
Recently I encountered a scientific claim about biology, made by Eliezer Yudkowsky. I searched around for the source of the claim, and found that he has been repeating versions of the claim for over a decade and a half, including in "the sequences" and his TED talk. In recent years, this claim has primarily been used as an argument for why an AGI attack would be extremely deadly. I believe this claim is factually incorrect.
The quotes:
I'm going to show the various versions of the claim I found below, with the relevant sentences bolded:
To plausibly argue that "humans" were intelligently designed, you'd have to lie about the design of the human retina, the architecture of the human brain, the proteins bound together by weak van der Waals forces instead of strong covalent bonds
Yudkowsky discussing the flaws of evolutionary design, in "the sequences" blog post "dark side epistemology".
It was obvious years before Nanosystems that molecular nanomachines would in fact be possible and have much higher power densities than biology. I could say, "Because proteins are held together by van der Waals forces that are much weaker than covalent bonds," to point to a reason how you could realize that after just reading Engines of Creation and before Nanosytems existed.
Yudkowsky discussing AI interventions on the alignment forum.
A lot of the advantage of human technology is due to human technology figuring out how to use covalent bonds and metallic bonds, where biology sticks to ionic bonds and proteins held together by van der Waals forces (static cling, basically)
Comment on a post discussing technology and AI.
Algae are tiny microns-wide solar-powered fully self-replicating factories that run on general assemblers, "ribosomes", that can replicate most other products of biology given digital instructions. This, even though the proteins are held together by van der Waals forces rather than covalent bonds, which is why algae are far less tough than diamond (as you can also make from carbon). It should not be very hard for a superintelligence to repurpose ribosomes to build better, more strongly bonded, more energy-dense tiny things that can then have a quite easy time killing everyone.
Yudkowsky's example scenario for how an AI could extinct humanity, on twitter
Can you build your own synthetic biology, synthetic cyborgs? Can you blow straight past that to covalently bonded equivalents of biology where instead of proteins that fold together and are held together by static cling, you have things that go down much sharper potential energy gradients and are bundled together, people have done advanced design work about this sort of thing.
Yudkowksy's Ted talk, again discussing AI capabilities, during the Q&A section.
I broadly endorse this reply and have mostly shifted to trying to talk about "covalently bonded" bacteria, since using the term "diamondoid" (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.
Yudkowsky's response to my recent article a few weeks ago, talking about how to refer to potential advanced nanotechnologies.
Summarising the claim
As you can see, Yudkowsky has repeated this claim several time over a time period spanning 15 years to just a few weeks ago, in very high profile contexts.
These quotes all make roughly the same argument, which I will sum up as follows:
Proteins are held together by weak van-der-waals forces, which are weak forces, akin to static...
confidence level: I am a physicist, not a biologist, so don't take this the account of a domain level expert. But this is really basic stuff, and is very easy to verify.
Recently I encountered a scientific claim about biology, made by Eliezer Yudkowsky. I searched around for the source of the claim, and found that he has been repeating versions of the claim for over a decade and a half, including in "the sequences" and his TED talk. In recent years, this claim has primarily been used as an argument for why an AGI attack would be extremely deadly. I believe this claim is factually incorrect.
The quotes:
I'm going to show the various versions of the claim I found below, with the relevant sentences bolded:
To plausibly argue that "humans" were intelligently designed, you'd have to lie about the design of the human retina, the architecture of the human brain, the proteins bound together by weak van der Waals forces instead of strong covalent bonds
Yudkowsky discussing the flaws of evolutionary design, in "the sequences" blog post "dark side epistemology".
It was obvious years before Nanosystems that molecular nanomachines would in fact be possible and have much higher power densities than biology. I could say, "Because proteins are held together by van der Waals forces that are much weaker than covalent bonds," to point to a reason how you could realize that after just reading Engines of Creation and before Nanosytems existed.
Yudkowsky discussing AI interventions on the alignment forum.
A lot of the advantage of human technology is due to human technology figuring out how to use covalent bonds and metallic bonds, where biology sticks to ionic bonds and proteins held together by van der Waals forces (static cling, basically)
Comment on a post discussing technology and AI.
Algae are tiny microns-wide solar-powered fully self-replicating factories that run on general assemblers, "ribosomes", that can replicate most other products of biology given digital instructions. This, even though the proteins are held together by van der Waals forces rather than covalent bonds, which is why algae are far less tough than diamond (as you can also make from carbon). It should not be very hard for a superintelligence to repurpose ribosomes to build better, more strongly bonded, more energy-dense tiny things that can then have a quite easy time killing everyone.
Yudkowsky's example scenario for how an AI could extinct humanity, on twitter
Can you build your own synthetic biology, synthetic cyborgs? Can you blow straight past that to covalently bonded equivalents of biology where instead of proteins that fold together and are held together by static cling, you have things that go down much sharper potential energy gradients and are bundled together, people have done advanced design work about this sort of thing.
Yudkowksy's Ted talk, again discussing AI capabilities, during the Q&A section.
I broadly endorse this reply and have mostly shifted to trying to talk about "covalently bonded" bacteria, since using the term "diamondoid" (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.
Yudkowsky's response to my recent article a few weeks ago, talking about how to refer to potential advanced nanotechnologies.
Summarising the claim
As you can see, Yudkowsky has repeated this claim several time over a time period spanning 15 years to just a few weeks ago, in very high profile contexts.
These quotes all make roughly the same argument, which I will sum up as follows:
Proteins are held together by weak van-der-waals forces, which are weak forces, akin to static...