and kind of an interesting fractal when you think about how most generative ML models are trained by adjusting their parameters to maximize the likelihood that they fool a so-called discriminator model

This isn't as common anymore, most modern image models are diffusion models which do not rely on a discriminator but which transform noise into an image using an iterative refinement process. GANs are annoying to train and don't work quite as well for image synthesis but they are still somewhat used as components (like as an encoder to transform an image into a latent image so it is easier to process and decode it back at the end, e.g. Stable Diffusion's VAE) or as extra models for other processing (like ESRGAN and its derivatives which is fairly old at this point, often used for image upscaling or sometimes for removing compression noise). The main force that pushes AI model output to be less detectable is that AI models are built to represent the distribution of the dataset they are trained on, and over time better designed models and training regimes will fit that distribution better, which by definition includes outputs becoming more indistinguishable from the dataset.

As far as I have seen, the AI classifier arms race is already very far behind on the classifier side. I have seen far more cases of things like ZeroGPT returning false positives than I have seen true positives that don't include "As a large language model...". I have seen plenty of instances of photos of the current conflict in Israel where people fed a photo to an AI classifier site and confidently said it was 97% chance of being AI when visually the photo doesn't even show any signs of being fake, and it's more likely that the photo is just a real photo that doesn't actually show what is claimed (which shows that people need to learn more about propaganda in general -- the base unit of propaganda is not lies, it is emphasis, because of this you need to be more wary of context than whether information is factual in most cases). The fact that people blindly trust AI classifiers is arguably somewhat more damaging right now than generative AI models.