A massively open source neural network
When I was 12, I was really into computer graphics. I loved that you could simulate entire worlds inside a computer—down to each ray of light. The only issue? My laptop sucked.
So I installed SheepIt. SheepIt is a distributed render farm. It allows you to enter into an agreement with other people that says: if you leave your computer running overnight for me, I’ll leave my computer running overnight for you.
This worked pretty well when enough people participate. I believe the same principle applies to the training of large neural networks.
In order to build the best chatbot or find the best drug, you simply need to train the biggest model. There is a strictly increasing mapping between input compute and output quality.
Neural networks are winner take all. This is why Google is erecting nuclear reactors and why Larry Ellison and Elon Musk are sweet talking Jensen Huang to take more of their money. But acres-wide GPU farms all running identical computations to train the same model seems like a sad waste of this planet’s resources.
Some napkin math tells us that if 1 billion people left their computers running overnight, they could, in aggregate, beat corporate GPU farms. Sure, there is coordination overhead, hardware problems to overcome, and incentives to work out. But we’ll figure something out.
The next generation of hackers will build the algorithm that can aggregate the outputs billions of nodes in a network. Model training and inference will be done on the internet highway. The neural network that wins will be open source both in architecture and compute.