If someone made a "x points to gryffindor" joke and you felt the need to chime in with "what the fucks a gryffindor, why would I know that, you know that story isn't real right" you would be treated much the same way you're being treated now

You are generally expected, for better or worse, to have some level of familiarity with things that exist within the culture of the people you are interacting with