Our peripheral vision is much better than it looks, and we're highly attuned to the kind of situational changes in sound and lighting that might accompany being stared at. Also, there are a lot of false positives and we only remember the true positives. This has been highly studied