Conservation of mass?
Do they understand that producing energy and fertilizer using the bodies of animals is less efficient than producing the same number of calories or mass of nitrates from plants+less-energy-than-is-required-to-raise-the-animal-in-question?
There's land that isn't good enough to grow crops, but is good enough for wild plants to grow. You can, as you said, 'manage' it - give enough fertiliser and water to make it suitable for agriculture. But that is often unaffordable for the people living in such places, so using animals to gather and concentrate the available nutrients is the best option available to them.
That's an orthogonal injustice though, no? Collectively, our species massively overproduce food, so I would think the fact that there is a prior reason to be trying to cultivate land like this, which ought to be managed for native flora/fauna is a separate and solvable large scale land allocation problem, the solution to which frees whatever livestock use the argument excuses.