Look up slipper socks. Like this amazing combination of a house shoe and wool sock.

Also I saw you comment about house shoes making your feel sweat. I too have this issue. Wear a regular sock in combination with the slipper sock and there will be no sweaty feet as the two layers of cloth help regulate the heat and the regular socks are better at whisking away moisture as well.