I know guitar strings are also measured in gauges with the bigger number being a smaller diameter. I don't know how that fits into the fraction of pound logic though.

found this quora post about it - this is the simplest explanation I found in the comments

https://www.quora.com/Why-is-the-gauge-number-of-say-wire-needles-and-shotgun-shells-inversely-proportional-to-their-sizes-For-ex-12-gauge-wire-is-thicker-than-20-gauge-wire?share=1

With wire, it is simply how many pieces laid side by side make an inch. This also applies to needles. With shotguns, it is how many lead balls of the bore diameter it takes to make a pound. So, one is a measurement of size and the other of weight.

more contex:

Think about how you might measure these . Take wire. You might not have access to a Vernier caliper or a micrometer, but you will have a standard ruler. So you wrap the wire around tightly and count the number of turns,to cover the inch spacing..The same trick would have been in use by weavers for hundreds of years.

And there's some other explanation on the wire related to how many times the metal has to be pulled through smaller funnel-like things to make it smaller without breaking. Not sure what is the most historically accurate answer, but I like the measure on a ruler explanation as it would be plausibly preferable to craft folks.