No need to be sorry, these are a lot of my thoughts too. It really breaks the immersion of being in a peaceful town when he enters. I find it uncomfortable that he's around and his dialogue doesn't make him sympathetic.

According the the wiki, there's even a book in the museum titled "Is War With The Gotoro Empire Necessary? A Unique Perspectiv" so clearly not everyone in this world thinks this is a justified war