Sure, but Peyote takes 10-30 years to grow in a very specific climate, and is endangered due to overharvesting. Growing your own is one thing, but exploiting an endangered plant and telling the groups that use it to "Grow it on their own land" is blatant colonizer behavior.

is pandanas an endangered species? the cactus is.

If whitey wants peyote, whitey should grow it on his own land, it's all stolen anyway.

This isn't something I'm super knowledgeable about, but I think Peyote cactus is mostly wild, relatively rare, and black market access to it cuts into the supply available for native religious groups:

Dawn Davis, 43, an expert in peyote conservation and a member of the Shoshone-Bannock tribes of Fort Hall, Idaho, worries that decriminalization efforts will renew the kind of fascination with psychedelic experiences that moved a generation of seekers to buy peyote from black market sources in the 1960s.

https://www.latimes.com/environment/story/2020-03-29/native-americans-want-mind-bending-peyote-cactus-removed-from-efforts-to-decriminalize-psychedelic-plants

Edit: apparently I'm 20 minutes late and 4 other people said the same stuff. Oh well.

I think there is more going on than just "it's ours, you shouldn't have it" though. Peyote in the US is designated legal for religious use by indigenous people as part of traditional practices only. It's also extremely slow growing and requires very careful harvesting to keep the plant alive, it's listed as an endangered species in the wild, explicitly due to over-harvesting.

Basically people using it as a recreational drug can have a very real impact on the legality and availability of the plant for indigenous people.

The people in the OP are not engaging with any of this stuff and treating it simply as culture war, with an attitude of extreme contempt and hostility to the people who they see as potentially stopping them from getting high, which I find pretty gross - regardless of whether you think it's fair to gatekeep the plant.