I forgot where I learned this, but I remember hearing that after The Simpsons were renewed for their 7th season, most of the writers assumed that would be the last so they went a bit more outlandish and less grounded.

This is because The Flintstones only lasted 6 seasons and they didn't imagine their show would get that much more than the cultural cornerstone of US TV animation at the time.

Also, as seasons 8 onwards were being produced, fewer and fewer of the original writing staff remained, and the new blood included people who watched the Simpsons growing up or as young adults. As you may know, the vast majority of fans of any franchise, be it video games or sports or fiction have absolute dogshit ideas and should not be in charge of the media they're a fan of. It slowly devolved into "episode concepts I thought would have made a good episode when I was 16" like what if there was a Harry Potter parody or what if there was a celebrity cameo that was cringe?

For example, the baseball episode of the Simpsons was a good use of celebrity cameos as themselves because they were can still be replaced with a fictional in universe parallel without much issue. Another example is Leonard Nimoy's cameo. If it was a weird one off character saying the things he said, it would still work as a gag (outside of being beamed up Star Trek style)