If Brexit contributed to a general economic downturn then luxury goods with cheaper and readily available substitute goods would be one of the first to be hit by people spending less.

But like you said, it's probably very hard to delineate the effects of Brexit from general economic malaise.