I believe if life is common than there must of been at least one group of aliens that at least attempted to colonize it self everywhere in space?
Unless humans are a extreme anomaly and that most other aliens don’t really care about space exploration, and just focus on their home planet.
This fails to really reckon with deep time. Yes the Universe is unimaginably large, but time is also unimaginably deep. Von Neumann probes with modest replication rates and interstellar cruise speeds could colonize the Milky Way in a few million years, which is exceptionally quick.
One of the things that's always bothered me is the "when you look out at the night sky, most of the stars you're seeing are already dead!" line because it's just flat wrong. You can't really see stars more than a few thousand light years away, and even 10,000 years is quite short on stellar lifetimes. The only stars that are likely in our sky but already gone are supergiants already on the verge of collapse like Betelgeuse.