It's more implied than explicit. He gets his kingdom but wants to extend it to the whole world. Guts' refusal to submit continuously gets under his skin even if it's just a mild annoyance. He doesn't seem to enjoy his minions as much as he did the Band.

It doesn't undermine him as a villain because he's constantly doing more stuff, but it does sort of build a sense of a person drinking ever more seawater to try and cure their thirst.