I feel for you, it must suck. The first... year or two of driving, I was extremely anxious about the whole thing. Not just navigation, but the driving itself, especially when I had to get around in unfamiliar vehicles. That really changed with practice over the years tho, I'm pretty relaxed behind the wheel now.

I also get nervous driving somewhere far away for the very first time, since the navigation part is a bit of an issue for me too. But when I remind myself that I have all the time I need and can, at any point, just pull over and look for the way, I can generally deal with it.