I get what you're saying, but I do think that it's a good idea to just default to the oxford comma, since it doesn't detract from clarity even in cases when it maybe wasn't necessary, and there are definitely situations where its absence impedes communication. Especially since it can be hard to tell when you're not communicating well sometimes, since you're in your own head and you know what you mean.