Avoid acrylic, cotton, or low quality polyester.

Layers are your friend. Thermal underwear under trousers, and maybe sleet or snow pants on top if super cold.

Linen underwear if you're fancy.

Key point is to avoid getting too hot as well, since sweat will make you cool down too much.