I think it would be fine. We are talking compost, not fresh manure. Compost has lots of nutrients but in a stable, water insoluble form, available over a long period. In my lettuce trial of different composts/manures in pots (somewhere in the forum!) the homemade compost sustained growth even longer than composted cow manure and proved it contained lots of goodness. But because its nutrients are stable and insoluble, they don’t/didn’t flood the lettuce with easily accessible nutrients at first, so growth was always measured and steady.
Containers need concentrated goodness, because the root run is restricted compared with open ground, so I expect that you can grow wonderful salads with worm compost in pots, boxes or whatever.