Yeah they are mostly irrelevant. There are a few interesting details in the first movie about why Thor came to Earth and why Loki was a villain, but all the movies are designed to be watched by people who've never seen a Marvel movie before anyway.