Conservation of mass?
Do they understand that producing energy and fertilizer using the bodies of animals is less efficient than producing the same number of calories or mass of nitrates from plants+less-energy-than-is-required-to-raise-the-animal-in-question?
The carbon, nitrogen, etc. contained in that grass, waste, and straw should be buried in/returned to the soil to grow plants instead of being farted into our atmosphere. Assuming the land in question is arable in the first place (which I think is valid if it's producing enough plant matter for grazing to be viable), if managed at all would produce more calories of human-compatible nutrition per calorie invested than harvesting of grazing animals on said land would.
There's land that isn't good enough to grow crops, but is good enough for wild plants to grow. You can, as you said, 'manage' it - give enough fertiliser and water to make it suitable for agriculture. But that is often unaffordable for the people living in such places, so using animals to gather and concentrate the available nutrients is the best option available to them.
That's an orthogonal injustice though, no? Collectively, our species massively overproduce food, so I would think the fact that there is a prior reason to be trying to cultivate land like this, which ought to be managed for native flora/fauna is a separate and solvable large scale land allocation problem, the solution to which frees whatever livestock use the argument excuses.