Unless magic is a taught skill in which case maybe the only reason the upper class is magic is because they train from the age of 7. Maybe the plot could be about peasants teaching each other magic sort of like how martial arts started as a peasant thing