We probably still have at least two big performance leaps left before we have to either abandon silicon based transistors or the FET based logic gate. The next big thing Intel, TSMC, Samsung, and IBM are all working on is replacing the FinFET transistor we currently use for logic with Gate All Around Field Effect Transistors (GaaFET). Everyone will stay on gate all around for maybe 5 to 7 years and after we have squeezed out all possible performance optimizations, most roadmaps point to something like non-planar designs with multiple logic gates vertically stacked atop each other (CFET) being the next evolution. After that, maybe FET based logic gates on silicon will have finally hit their limit and a new material like germanium will be adopted... or we might just replace FETs all together with new forms of logic gates based on a novel mechanism.

but we're also nearly into the territory that the last mile of improvements require the entire computer (e.g. RAM/wifi/GPU/disk) must all be embedded into the final chip.

We already went there, all modern CPUs (Intel, AMD, ARM) are true SOCs (systems on a chip) where the components that used to be discreet (south bridge, north bridge, memory controller, clock generators, io/storage/network/usb controllers) are now integrated on the same silicon as the CPU cores themselves. The latest generation chips from both Intel and AMD are even leveraging 3D integration (vertically stacking modules on top of each other) to squeeze out that last bit of extra performance while maintaining the same physical footprint. It's at the point where they are 3D stacking up to a gigabyte of SRAM based L3 cache directly on top of CPU cores or putting up to 128GB of HBM ram directly on the CPU package to act as an L4 cache.

Some people assume only CPUs built for mobile devices (phones, laptops) are full SOCs but desktop/server CPUs that get socketed into a motherboard at this point are also true SOCs. Modern desktop and server motherboards tend to be nothing more than power delivery components and a physical jig that makes it easy to plug peripherals into the CPU SOC or connect multiple CPU SOCs together on one board. Other than increasing the bandwidth of interconnects, there is still very real performance that can be gained by lowering the latency between CPUs and plug in peripherals like memory DIMMs, discrete GPUs, network adapters, or other CPUs via replacing electrical traces with photonics. Intel's lab division has been working on maturing silicon photonics to the point it can be directly integrated into a CPU.