I don't have much experience with dogs, much less untrained off leash ones. One is pretty big (a lab maybe?) and always runs right up to me and my wife barking territorially whenever I'm out in my garden, and doesn't stop until one of my neighbor's kids comes and gets them. I've had dogs try to fight me before and they make me really nervous. I just want to touch grass :(
How should I be responding to the dog when this happens?
Depends, if you maintain proximity long enough and offer treats (sneaky or with neighbors permission) they can become habituated to you. They just have to see you as part of the normal fabric of their reality, they don't really understand property boundaries and all that.
If they seem too aggressive to acclimate to you, you might just have to talk to the neighbor directly. Alternately put up a visual barrier on the fence (they sell opaque removable slats for chainlink), install a wooden fence, or plant a hedge on your side.