XOrg is designed so a central server (mainframe) sends and receives data from smaller terminals, and that not only includes a heap of devices that haven't been in use since the 90s it also has a ton of features that nobody uses. (See: X native fonts, X native widgets, X driver model...)

X's way of handling events and sending draws to clients as such is somewhat convoluted. Once you start to really dig into it, it's amazing how much people managed to stack on top of it until today.

Besides, modern day X over Network is a somewhat niche and possibly broken function