I expect it had something to do with the Highland Clearances. The landlord class was undertaking a gradual process of enclosure and land commoditization, imposing changes in agriculture to maximize rental incomes which involved evictions and eventually deportations of large numbers of highlanders.

Large scale projects of displacement and proletarianization are often accompanied by a narrative of racial inferiority to explain and justify them.