No... Not in practice anyway, maybe in theory. I know on ARM SoCs there's lack of auto-configuration (like you have on PCs with e.g. PCI), and the kernel has no way of knowing what hardware is available. So there's a file that lists all the devices, and how to talk to them, called (I think) a "device tree". This file gets appended to the kernel image, and so the bootloader just loads that together with the kernel. The kernel doesn't do any auto-configuration and rather just reads this file and loads the relevant drivers based on that. I guess it might be (in theory) possible to do this on PC, but I've never heard of such a thing. I also don't expect that to make any noticeable difference for boot times. Pretty sure boot times are dominated by user space, and not the kernel anyway.

Sidenote (don't do this): You can compile your own kernel (this used to be pretty common back in the day). You can select only the drivers you need, and can also select whether they should be compiled directly into the kernel or as modules that can be loaded later if needed. Pretty sure the auto-detection happens regardless for most hardware, since the driver needs to be initialized and told where the hardware is to be found. Compiling a driver right into the kernel just means the driver code is in memory right from the very start. I don't recommend doing this btw, the only difference you will notice is shit not working due to you screwing up, and you're going to waste a bunch of time and electricity compiling your kernel with every update. You sometimes needed to do this to get all your hardware working, but I haven't done this in ages.