It's an ego thing, in my experience. Lots of physicists--especially those who aren't really all that philosophically sensitive--think that they do the most robust or "real" version of science, and that everyone else is working off approximations or "softer" versions of what they do. They think that any science that doesn't look like what they're used to (i.e. precise mathematical models that describe the behavior of constituent parts of a system) is at best a soft science and at worst totally illegitimate. Because they're good at math, they think they can barge into any other field, immediately understand it, and be better at it than people who have spent their whole life in that field.