I guess it isn't, is it?
First you should ask why one is vegan? There are different reasons someone might be vegan that will inform you better really. Is using an ox to plow a field vegan? If you're looking at it from a labor perspective than it is not vegan as you're breeding worms to make compost.
Another perspective is by providing a luxurious environment for the worms you are improving quality of life. Purchasing worms bred explicitly for this purpose could be a conflict. I think about animal sanctuaries, are those vegan?
Worms live to eat decaying organic stuff. You provide food and a safe environment for them and they naturally produce a useful product for us as waste.
I think it's a bit different from something like honey because bees make honey to store their food for later use. Worms aren't going to use their waste so taking it isn't exploitative, least in my eyes.