A landlord renting out a back yard cottage isn't driving up housing costs relative to their homeowning neighbor that has fewer people on the same plot of land. They're still a leech looking to make money from monopoly land rights, but as an individual they didn't make rent go up nor did they do so collectively.

Where they do drive up rent is in pure investment properties re: land use. Even a simple argument holds this: after putting in the capital to build the housing, all the landlord does is extract money from ownership. But unlike a capitalist, where this drives the production of goods and services, the landlord seeks rent using natural monopolies: they provide nothing that a centralized authority couldn't do better and without rent-seeking. The landlord's cut is pure overhead on housing costs.

Because of the land use monopolization aspect, landlords also fight against other forms of housing. They fight against public housing and they fight against programs to house the homeless anywhere near themselves. They entrench the existing forms of land use, themselves beneficial to existing property owners and detrimental for anyone trying to buy housing for themselves - or to rent. Want to build medium density public housing in a single family home neighborhood? Sorry kiddo, that would require zoning changes and every property owner in the area (with 25-50% of the housing owned by landlords) raise a fuss.

The real elephants in the room are artificial scarcity of housing and the financialization of housing, though, which are highly interrelated and not really small landlords' fault.