If you want a clean install, go for a carefully curated set of packages, rather than trying to mix and match to create your own selection - that's bound to result in a Frankenstein installation.

I'm partial to Fedora Silverblue, which is essentially just a single package containing the things you need to have a usable desktop. You can install what you need separately on top of that, but on updates, the whole base gets replaced wholesale - including, which is most relevant to your concerns, removing stuff that is no longer used/needed, rather than having that clog up over time.

Okay, good :) You can try for yourself. Install bare-boned Linux, and then install for example fvwm, a light weight Window Manager. Fair chance that for example on Debian you will have to figure out which more packages (e.g. xinit and xorg and fonts) to install to make the GUI work. I like Debian for reasons, but the software package dependencies are not always on par with the package installations is my experience.

There are options to start at any level you feel comfortable with. In the far end of not installing anything is Linux From Scratch. As the name implies, you start from nothing. I don't recommend this unless you have specific reasons. If you want a small command line only system to start from there's plenty of those, many distros offer this choice. Arch, Debian, etc.