The server is used for hole punching, to open up a P2P connection thorugh NATs and Firewalls. If it doesn't work the server also relays the traffic between the clients.

Getting an end to end connection through todays internet is unfortunately not easy for an average user.

Theoretically, without the server, every time you want to connect to a peer you would have to figure out what's their public IP address is, which can change. The server acts as a middleman between the peers so you dont have to do this manually, all peers only need to know the server's IP address to connect to each other. The server is really only used for this initial linking up of peers, afterwards the connection is P2P (if possible, they fall back to a relay server if P2P fails).