guy recently linked this essay, its old, but i don't think its significantly wrong (despite gpt evangelists) also read weizenbaum, libs, for the other side of the coin
guy recently linked this essay, its old, but i don't think its significantly wrong (despite gpt evangelists) also read weizenbaum, libs, for the other side of the coin
relative to what and in what sense do you mean this?
I mean for the most extreme example, it takes approximately 1 bazillion operations to solve 1+1
It doesn't actually. Bees can be trained to do simple arithmetic and they have relatively few neurons.
I hypothesize that it takes fewer watts for a bee brain to do arithmetic than it does for my gpu to simulate an incredibly simple and highly reduced model of a biological neural network to do the same thing.
No I mean the human brain does that, and adding 1 and 1 can be done with like a few wires, or technically two rocks if you wanna be silly about it
Sorry for being thick. You mean the human brain does which?
The human brain does that many operations to add 1 and 1
How do you know?
Don't you think imagining 1, imagining another 1, briefly remembering the concept of addition, thinking the word "plus", remembering that 1+1=2 is a true thing you know... that involves quite a few neurons firing right? And each neuron is unimaginably more complex than a piece of digital hardware that adds 1 and 1, which again is like 40 or 50 pieces of metal and plastic
I think it's far fewer than you think. Here's my logic:
First, let's make this a fair comparison. If we're comparing neural computation of 1+1 to a 4 bit adder, then our signals are already downstream of the networks that do the initial perceptual processing and convert those into neural correlates of the number 1 and the operation addition. If they aren't, then we need to somehow compare power consumed for data input to the 4 bit adder and generally make the problem much more difficult.
My quick hypothesis is that for people who have practiced the arithmetic enough to not need to work it out manually, the recall is accomplished with fewer than 10,000 neurons spiking (and orders of magnitude more neurons not spiking, which consumes a negligible amount of energy). Even if this many neurons are involved, they're not all constantly spiking or the neural network is having a seizure. Typically only a small fraction of neurons that are involved in a task are spiking at high rate simultaneously, but even if it's all 10,000 spiking constantly for the duration of the calculation, a typical human brain consumes about 20 watts and has 10^11 neurons total, so our computation requires only 20 microwatts.
Now, if we were to make the comparison truly level, we'd discard all the memory circuitry and we'd compare the power consumption of the 4 bit adder to a reasonably sized biological neural network that has learned to accomplish only the task of 4 bit addition.
I'm trying not to get nerd sniped here, but if you're curious and handy with python, you could put together an experiment to see how many leaky integrate and fire neurons you need to do 4 bit addition with this library
This group found a way to do 16 bit addition with a 4-neuron spiking neural network in silico. If indeed four LIF neurons are all you need to do 16 bit addition, then the calculation can probably be accomplished with only 8E-10 watts (compared to an 8 gate circuit that likely consumes (when switching) on the order of 8 nanoWatts, more than 10x our estimate for the biological neural network)
and that's why I hypothesize the computational efficiency ie bits computed per watt expended is higher for biological neural networks doing addition than silicon doing the same thing. Thanks for coming to my
talk