Black holes are not infinitely dense because physical infinities do no exist, you are correct that it doesn't make sense and anyone claiming this is wrong.

The foundation of quantum mechanics is that the universe is quantum, which is to say, it is made up of a finite quantity of discrete "bits." An area of one Planck Length by one Planck Length contains one bit of information, either it contains mass or it doesn't. The concept of infinite density violates this by claiming that an infinite amount of information can be stored in an area smaller than that, which contradicts our understanding of quantum mechanics on a very fundamental level. The highest density something can have is to contain one bit of mass in every Planck area.

The reality is that black holes come from stars, with finite mass, and they occur when the force of gravity becomes so great that it overpowers the nuclear forces holding an atom together, causing electrons to collapse into the nucleus, and even for the subatomic particles to break down. However, the mass still occupies a discrete area, incredibly small but not infintismally so.

Now having said that, black holes are currently a controversial and unresolved subject because no consensus has formed around a satisfying resolution to what's called the Black Hole Information Paradox, which pits several fundamental principles against each other.

Information, in the context of physics, cannot be destroyed, because the destruction of information would mean reducing the complexity of the universe, reducing entropy and violating the second law of thermodynamics. What this means is that it's hypothetically possible, given sufficient knowledge of the physical world, to reconstruct everything that has ever happened. Every event that occurs leaves behind evidence, and that evidence can be jumbled up and garbled beyond recognition, but never completely erased from history.

For this to be true in the context of black holes, it is necessary for there to be some method of extracting information from a black hole - but there isn't. Or at least, nobody has been able to explain how that would work. Some have speculated that the information is emitted in Hawking radiation, but this doesn't really make any sense because of what Hawking radiation is. Others have speculated that the information could be stored on the event horizon, but this has been rejected as the event horizon is just a mathematical abstraction (this is sometimes referred to as a question of whether black holes have "hair," with most physicists agreeing that "black holes have no hair"). Various other hypotheses have been proposed over the decades, but all of them seem to contradict some principle that we have very good reasons for believing in, and we can't overturn any of those principles without solid evidence, which we don't have.

To restate the problem: if black holes can consume physical things that have high entropy, and reduce them to a state of low physical complexity, then they are capable of reducing entropy in a closed system, and our understanding thermodynamics is fundamentally wrong. If they are capable of storing that entropy in an infintesmally small area, then our understanding of quantum mechanics is fundamentally wrong. If it's possible to retrieve information from a black hole, then our understanding of relativity is fundamentally wrong. So either some fundamental principle in physics is wrong, or someone will eventually come up with some very clever solution that reconciles these competing principles.