Read your logs. You’ll be amazed what you find.

Oh, I know. You’re not like those other admins. You always read your logs. You set up Splunk and OSSEC and you have all kinds of alerts and custom event types set up and you read every log message you haven’t explicitly whitelisted from your reports, right?

Even if you’re a compulsive syslog reader, there’s a chance you’re still not reading all the right logs, because there’s a lot you can figure out about tiny little configuration problems with a little bit of network analytics, as I found this week.

I’m in the middle of a major external DNS cleanup. For reasons I’m not going to go into on this blog, my organization has an absolutely huge number of entries in external DNS that have no right to be there. As a result, I’ve turned on DNS query logging to monitor, for a couple of weeks or months, what names are actually getting queried, so we can work out any misconfigurations with our firewall team. While I was scripting some small tools to help me with this, I figured I’d analyze the data in a couple of other ways too.

I found:

• DMZ servers misconfigured to query Active Directory domain controllers (!) that had ceased to exist years ago
• Samba servers in the DMZ attempting to connect to internal WINS
• Hosts with nscd turned off making 30,000 DNS queries per day for the same hostname
• Internal client machines inexplicably configured to use external DNS
• Servers trying to resolve every domain name as a subdomain of our organization’s domain

The idea was to keep query logging running for just as long as I needed to do this project, but I’m getting way too much useful information out of it to turn it off.