Figuring out property lines and lot sizes is trickier than I thought

We moved into our new place a few months ago, and I figured the property lines were pretty straightforward. You know, fences on both sides, seemed obvious enough. But then last weekend, our neighbor casually mentioned that the fence on one side might actually be a foot or two off from the actual property line. Um, what? I never even thought to double-check that stuff when we bought the house.

Now I'm wondering if I should get someone out here to officially measure it or something. I mean, it's not like we're planning to build anything major, but it'd be nice to know exactly what's ours. Has anyone else run into something like this? Did you end up getting a survey done or just let it slide?