I'd like to keep XCFE for VERY occasional use, so when I'm not using it I'd like to quit it. Also I don't want it to start at boot. I've found a bunch of ways of uninstalling it but that's not what I'm after, I just want to make sure it is not running unless I start it.

  • sudo systemctl disable lightdm

    You'll boot into a text tty login prompt. Log in, and run startx to start your X session. You don't need lightdm at all.

    Or, if you do want to run the session manager, log in and run sudo systemctl start lightdm. You'll still be logged into tty1, so you might want to switch back to that and log out if you're concerned about physical security at your device, but it doesn't otherwise hurt anything.