The best part of the fediverse is that anyone can run their own server. The downside of this is that anyone can easily create hordes of fake accounts, as I will now demonstrate.
Fighting fake accounts is hard and most implementations do not currently have an effective way of filtering out fake accounts. I'm sure that the developers will step in if this becomes a bigger problem. Until then, remember that votes are just a number.
People may not like it but a reputation system could solve this. Yes, it's not the ultimate weapon and can surely be abused itself.
But it could help to prevent something like this.
How could it work? Well, each server could retain a reputation score for each user it knows. Every up- or downvote is then modified by this value.
This will not solve the issue entirely, but will make it less easy to abuse.