Your point only applys to other galaxies.

Even with current drive tech we could colonize the galaxy in under 2 million years. So if there are aliens in our galaxy they should be younger than that.

Given the timescales involved it seems too much of a coincidence that we apered so close to each other. So its posible we are first.