Fun reminder: The "Ivy League" has nothing to do with academia. It is the league within which these various universities' sports teams compete.

So the reason for the vaunted "Ivy League" as a euphemism for these elite institutions has more to do with somebody's failson holding a tennis racket than it does with them holding a pen.