Slavery was abolished because the risk of slave rebellions, including the risk of the slave master's throat being slit and his head impaled on a stake, and cost of suppressing slave rebellions in order to prevent said slave master's head from being separated from his neck, was too great. At the end of the day, the cost/benefit was not in slavery's favor. It's like how "becoming a drug lord selling illegal drugs" isn't a career path most people would choose if the profit margin is extremely high for the commodity.

You can see this clearly with the way the British empire abolished slavery. There were various slave rebellions, including two Maroon wars where runaway slaves forming autonomous communities outside of British rule in Jamaica fought against their British oppressors. The triumph of the Haitian Revolution, where massa's collective throats were slit and massa's collective heads were impaled onto stakes, struck fear in the slaveowning class. And when Jamaica again had another slave rebellion where one fifth of Jamaica's slaves revolted against their British oppressors, the British empire saw the writing on the wall. After the rebellion was crushed in late 1832, Parliament passed the Slavery Abolition Act of 1833, almost 6 months later.

As for why the US South stuck with slavery, there's a major demographic difference between Caribbean islands like Haiti and Jamaica and the antebellum South. With this demographic advantage and the settler colony's strategy of self-deputizing settlers as a means of protecting theft of stolen Indigenous land, I believe the slaveowning class simply felt that the risk of slave rebellions was low enough that slavery should continue and with the UK's booming textile industry and their need for cotton, the status quo of slaves toiling in cotton fields can be sustained and a potential Nat Turner or John Brown wouldn't be enough to overthrow slavery.