For sure, the ability of these models to produce very convincing looking output creates an illusion of intelligence and understanding that isn't there. It's very easy for people to become convinced that the model really knows what it's talking about. In that regard, they're not really that different from CEOs. :)