Intruder Alert - Tracking down a rogue connection
The hosts I administer look through their logfiles each hour, looking for things that are out of the ordinary, and mailing them to me. This is where my uncanny ability to remember leap-seconds comes from; it's not because I actually care about things, but because I'll get an e-mail saying that a host suddenly found itself a whole second out of sync with some incredibly accurate clock somewhere. Lucky me.
Most things don't end up in my log digests, because most things are boring. The things I do see are things that I either care about, or have never seen before.
A few days ago I got an e-mail that one machine was trying to contact a particular address, 172.16.45.35. What made this noteworthy is that address is unroutable. It's a reserved, private address space. It doesn't go anywhere on the Internet, and it's not used by us internally, so there should be no reason to try and contact it. The connection indicated it would have been an outgoing web request, and since I was busy working on other things, I assumed that some other fool had set up their system incorrectly, and thought nothing of it. People leave references to their own internal sites in documents all the time.
A few days later I got another e-mail, same result. And then another, the next day, and another. Each time I looked a little closer. About the same time each day, a few attempts to contact this address, and then nothing.
Today, this bothered me. What if we're seeing these packets because there's something running on this machine that shouldn't be? So I go to my proxy logs, and do a search for the address. Nothing matches.
Hmm, that's odd. Let's see what's in our name-server cache, since the address is probably the result of a name lookup. kill -INT on your named will let you see its memory cache, a great trick to remember. Nothing in here, either, but it's now been hours since I got the mail, so the record may well have expired.
What's odd about this connection is that it seems to happen around lunchtime, but not every day, and not always exactly the same time, and sometimes it misses days, so I don't really know if or when I'll ever see it again. Rather than trying to futilely trying to find it minutes after it occurs, I figure that I'll set something running to catch it in the act: