It does not say they can't handle women making more, it says people quit when their wages fall relative to others, including college educated men

Younger white men in particular were more likely to leave when their expected wages fell relative to their more educated peers, according to the Fed study.

It mentions women once to say that they have seen a pay increase regardless of education status.

and they’re leaving in part because of their perceived social status relative to better-educated men of similar age according to a new study from the Federal Reserve Bank of Boston.

Non-college-educated men have seen their pay shrink by more than 30% since 1980 compared to the average earnings of all other prime-age workers. Their weekly earnings have declined 17%, while those of college-educated men rose by 20%, adjusting for inflation. That earnings loss has caused a decline in their social status, prompting them to walk away from work entirely, Pinghui Wu, the author of the study, wrote.

I did add the back breaking work part since you were talking about hard physical labor. The article does not say what jobs the non-college educated men were doing but some of those jobs with declining wages would be like that