Relevant Boots Riley explanation: https://youtu.be/JmyWvjszBOw

Also, Bacon's Rebellion is an interesting topic related to this, since this seems to be where the racial hierarchy in the US really began to solidify:

Bacon’s “army” consisted of many poor whites (both freed and bonded) and poor Blacks (freed, bonded, and enslaved). Poor white and Black workers and farmers joined together in a revolt against the colonial regime and big landowners responsible for their exploitation and impoverishment. Even when Bacon died, they persisted in their rebellion, forcing English ships to threaten them with bombardment before their final surrender. The last group of one hundred rebel holdouts consisted of eighty Black people and twenty English.

Their cause undoubtedly denied indigenous people’s rights. But this unity between poor white and Black workers and farmers terrified the planters and the colonial government. If white and Black workers could unite again in the future, they could easily overthrow the government and planter aristocracy, which was of course only a small minority of the population. They had to find a way to make sure it never happened again.

They quickly hit on a cunning, cruel, and divisive way to do so: give white workers certain rights and advantages over Black workers, while at the same time enshrining Black slavery into law. In the next twenty-five years, the colonial legislature passed a series of laws designed specifically to privilege white workers over Black, and to divide the two groups. In the words of one historian, “by a series of acts, the assembly deliberately did what it could to foster the contempt of whites for Blacks and Indians.”

Once the legislature was done, Blacks were not allowed to own slaves; Blacks were not allowed to own weapons; Blacks were not allowed to “lift their hand” against Christians; Blacks could be punished by dismemberment, but white indentured servants could no longer even be whipped; slaves were deprived of property, which was then turned directly over to whites; harsh punishments were implemented for miscegenation; people with any African heritage were defined as Black; release from slavery was forbidden.

By 1705, with these laws, Virginia’s system of slavery based on skin color was in place. White workers were given freedom from servitude, the ability to own property, and other privileges that Blacks were specifically denied. And the mixing of the two groups was more restricted than ever before. These factors all made it less and less likely that poor white workers and farmers would risk their small privileges to help more oppressed Black workers. And, once white-only slave patrols were instituted in 1727, white workers and farmers actually were tasked with directly oppressing now enslaved Blacks, and were usually paid to do so.

In other words, the rulers – the tobacco planters and colonial government – had successfully divided white workers and farmers from Black workers, who were now legally enslaved.