An adult man holding the hand of a little girl in public: A cause for concern?

When you see this how do you react? A lot of times I suspect it's likely they are a kidnapper/molester and will follow them for a bit. I've never intervened and feel bad knowing that I possibly allowed something awful to happen. How do you react when you see this?