Because no formal declaration had been made after the Trent affair, and the English textile industry had always worked off warehouse stock for cotton so some people thought the crown was going to declare war, and sold (driving down the stock), but the people in the know expected that any formal declaration of war would be held off at least for a year in hopes that the blockade would be lifted in time for the cotton crop for the next year to be shipped on time.

Over time the markets did in fact shift very heavily, but it's still telling that even a civil war in the United States had less of an immediate effect on the stock prices than this.