First you should ask why one is vegan? There are different reasons someone might be vegan that will inform you better really. Is using an ox to plow a field vegan? If you're looking at it from a labor perspective than it is not vegan as you're breeding worms to make compost.
Another perspective is by providing a luxurious environment for the worms you are improving quality of life. Purchasing worms bred explicitly for this purpose could be a conflict. I think about animal sanctuaries, are those vegan?
This is a great point. In tandem I will often hear, unionize your workplace! Well what if it is then what? Neoliberal unions aren't exactly utopian. Then someone says, well help organize other workplaces. Well been there too and I have to ask. To what end?