That's a fair point, but they did not yet have the capacity to make the entire population literate and educated, which I think is necessary to give people lots of free time. I'm not trying to say anything like dumb workers need to be kept busy or evil ensues, I just mean that people need something to keep them engaged, and they'd get bored of everything in their little village after a while and probably start breaking things or rioting. If they have an education, they can read and write and put on plays or skits and travel and do all sorts of other things to keep themselves engaged.