Do organizations like Apache mitigate that? I would assume it helps to have standard ways of contributing, bylaws, documentation, appointed positions, etc. so that there's always a process.

For example, the OpenJDK project (Java) has instructions on how to become a maintainer (someone who can commit changes and so on). The instructions are basically to do some small code changes and work your way up. And they don't really create any illusion that just anyone can waltz in with a big change (you can't even create a bug or feature in the bug tracker, you need to submit your first changes via the mailing list).