I also don’t understand how such a system would have arisen if at every point each private capitalist was seeking to maximise their profit (and thus looking to remove the excess workers rather than hiring people who added no value).

It's been a while since I read the book, but he does spend some time talking about this. He doesn't have an exact answer but he points out how curious it is that this phenomenon is occurring under a system that prides itself on maximizing profit.