This isn't a moral argument. It's a tactical one.
If you are a white vegan arguing with a carnist and that carnist brings up indigenous meat eating to associate veganism with colonialism, you may appear to be in a double bind. It is extremely difficult if not impossible to go on the attack without associating the vegan movement with colonialism and thus damaging it. But if you back down without fighting, then the carnist's point goes unchallenged, strengthening their position and weakening that of veganism.
That doesn't mean you don't have options, though. Think of the carnist's argument as a heavily fortified military strongpoint. You don't attack such a position head-on. You infiltrate. You hit it from its blind spots. You attack the weaker flanks and encircle it.
Instead of attacking indigenous people who eat meat, point out how cattle ranchers drove bison to the verge of extinction to force Native Americans to become dependent on their product. Talk about how commercial overfishing threatens the food supplies of coastal indigenous communities. Ask them about the vast portions of the Amazon being cleared for cattle grazing. Remind them of the exploited immigrants getting PTSD from their work in slaughterhouses. In short, confront them with the fact that carnism does far more to harm indigenous communities than veganism ever has or (owing to carnists having vastly more political power than we do) presently could.
All of this, of course, assumes that the person you're talking to is not indigenous. If they are indigenous, there's a good chance they'll already be sympathetic to some of your views. In fact, indigenous communities have actually been at the forefront of fighting some of the worst excesses of western carnism, such as when the Inuit got Canada and four other countries to ban commercial fishing in the Arctic. It's important to recognize that they're probably closer to our position than an average westerner, and that they're doing meaningful work to advance goals that align with our movement. No, most indigenous cultures aren't 100% vegan as we define the concept, but as Lenin said, you can't make a revolution in white gloves.
The points you mention as possible responses are correct of course. But I feel it would make me sound like I'm trying to avoid answering an actual question. Assuming the argument was chiefly a moral one, the question could be, "are indigenous hunting practices morally wrong?". How would you answer this, without deflecting?
I don't necessarily see how it would hurt my position to say that killing animals is wrong, no matter who does it. If the pro-murder side accuses me of being a colonialist/settler whatever, it's easy to point out, that colonialist interference to stop indigenous hunting practices by force is also morally wrong, and the two positions are not mutually exclusive. As of yet, no vegan militia has encircled a native village anywhere in the world.
I can't speak much about US politics, but if we move to an argument about policy, I could simply agree that lobbying efforts should be focused on the curtailing or abolishment of commercial murder.