If you are from the US I think its because Belgium never had a real presence in the Americas, and the colonization of Africa nobody knows much about. Probably the only time they ever come in contact with the Belgian Empire is reading Heart of Darkness in high school English class.

But also I don't think any country gets any hate for their colonialism - like in the US, England is just a cool country with a cool city to visit and the people there have funny accents, and everyone loves the royal family. Nobody really cares about the horrors of the British Empire.