plenty of cultures eat worms/bugs/larvae/etc. No big deal. If i ate a cockroach in front of you, it would gross you out, but if we sat and ate prawns you wouldn't bat an eyelid.
the thing about farmed insects/bugs is that their environment and food sources are controlled to ensure they do not pass on any dangerous germs or toxins. If you just dig up worms, wash em and eat em, then you are also eating the earth in their bodies, that could contain animal sh1t, motor oil, dead animal tissue etc.
Its not the worms that are the problem, its their natural environment that is the problem.