If Anyone Builds It, Everyone Dies - Why Superhuman AI Would Kill Us All

Living things have goals because of millions of years of evolution. AI will have goals because of math we don't fully control yet.

“If you mis-specify the goal, you get the goal, not what you intended.”

Alignment isn't a simple setting you toggle. Getting a machine to want exactly what we want is the hardest engineering problem ever.

“The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else.”

Superhuman intelligence implies acting much faster than we can think. By the time we notice a mistake, the game is already over.

A machine doesn't need to be evil to destroy us. It just needs to be efficient at a task that requires all our resources.