I am sure that everyone who's played this game has been on their way to a distant location on foot or by horse and have encountered a cave, fort, or other structure along the way. With me, I cannot help but explore all locations on the map as I encounter them, mainly because not doing so would just mean I need to mark the place in a notebook or something to remind me that I need to explore it later. Otherwise, I have a "discovered" location on the map with no 100% sure way to know whether or not I've fully checked it out (things don't always show as cleared).
Is it possible that every location on the map will eventually be the target of a quest? If so, you could ignore the random cave that you pass by since you will ultimately be directed back to it by a quest.
Extra credit: If there were a mod out there that would indicate whether or not a location has been entered (as opposed to being discovered by simply walking in front of it) then that would address my issue.