John von Neumann worked out the logic of a machine that could carry its own instructions and build a copy. Add one more instruction: Make copies of yourself, and work toward constant, neverending improvement. Every generation after that improves the next.

The systems in The New York Times story are software, and they still need human researchers to supply much of the judgment and imagination. But imagine joining recursive self-improvement to von Neumann’s physical machine. Each copy becomes another worker on the next design. A better design makes more capable workers.

Better at what? How is “improvement” chosen? And what is it allowed to consume along the way?

“Neverending improvement” has no finish line and no built-in reason to leave us in charge. These AI constructs actively work against boundaries. If a future machine learns to regard our limits as flaws in its design, it will be taking the assignment seriously …