Just because you are not "working in that field" does not mean you aren't using your education. Knowledge and education should be lifelong pursuits, just for their own sake.
I have two bachelor degrees (Law Enforcement Administration and Geology) that are separated by almost ten years. Altogether they add up to about 212 college credits. And it turned out that my major career (for the last 20 years) is unrelated to either one.
Furthermore... as soon as I retire (1-2 more years) one of the top things on my list is to go back and start auditing some courses on subjects I just want to study more: art, music, history, geography, languages.
I'd relax about all this. If you are lucky enough to enjoy a long life, it will take more twists and turns than you could imagine. Education is just the lightposts along the way. Our lives are a little darker without it.