Unwanted Pests

I have an old boiler out front that I plant annuals in every year. Last year a tree sprouted in there but I left it there. I figured it wasn't hurting anything. This year it was much bigger and definitely a poplar tree. Still no biggy until it was overtaken by worms. I didn't get a picture of them but they would line each leaf and eat it entirely. Well, that was enough of that, I got the big pruner and cut it down. I'll have to dig out the roots this fall when the flowers die back. I don't know what kind of worms they are but they are small and, apparently, hungry.

I had entertained the idea of relocating it somewhere else, maybe out by the pond - pre worm infestation. Oh well, what do you do? Sure looks bare in that planter with it gone though.