[Population: One] <A HREF="http://popone.innocence.com/ar

[bryant]

My Monday LISA tutorial was on system log aggregation, analysis, and statistics. mjr taught it, and he's as good a public speaker as ever. Also the topic was pretty damned fascinating. I'll be dumping a pile of links into del.icio.us sometime soonish now.

Highlights, some of which are significant and some of which are just cool:

You can set up an invisible loghost. What you do is you specify a non-existent host as the loghost on all your DMZ servers. You're gonna need to manually stuff an entry into the arp table so that your DMZ servers will blithely send syslog packets off into thin air. Then you hook the real loghost up to the DMZ with no IP address in promiscuous mode. Run tcpdump on it to capture all the packets, and write some cheap perl to strip syslog payloads out of the captured packets.

Or use mjr's plog instead of tcpdump, since it'll automate all that complex stuff for you. Neat.

Artifical ignorance. Cute term. It's basically the same rule of thumb as "block everything, then permit what you want" but reversed. "It's interesting unless I've explicitly said it's boring." At a very basic level, it looks like this: grep -v -f patternfile. As you figure out what you don't care about, stick a regexp to match into patternfile and you won't see it again. The process speeds up over time, obviously. This calls out for a slick web front end.

First seen anomaly detection. It's sort of like artificial ignorance, but different. You alert every time something completely new appears in the logs. There is a tool for this, also written by mjr, called NBS (Never Before Seen). It uses Berkeley DB and is very fast. You feed it input for a specified dataset and it tells you if it's seen that particular chunk of input before. It can report on its database in a bunch of useful ways.

Example: record DHCP servers giving out IP addresses. (Sample string after a bit of log parsing: "10.0.0.10 gives IP 10.0.1.1 to MAC 0:2:2d:10:10:10".) If a new MAC address shows up, it'll be flagged by NBS as a new chunk of input, because that string is guaranteed to differ in that case. If an old MAC address gets a different IP address, that'll show up too, but only the first time it gets that particular IP. As a bonus, you'll find out if any new DHCP servers show up. Pure gold.

Another example, which happens to be the first use I thought of: turn it loose on my HTTPD log files. Filter said log files for referrer and URL pairs; report the first time a new referrer/URL pair is seen. I have something like this in place now but it's written in perl and it's fairly fragile; this will be better.

Or just dump URLs into the database. "Hm, someone just tried to load /cgi/foobar.exe for the first time; looks like a new exploit."

So yeah, a very cool tutorial. I'm all jazzed about the possibilities. Check out his web site on the topic.

Comments

[jeregenest.livejournal.com]

Bryant's speaking in tongues again. Quic, someone talk about politics or gaming to distract him.

[bryant]

Sniff. Kyle appreciates me.

[kniedzw.livejournal.com]

I do! I do!

Or rather, I hate you for working for a company that will send you to LISA. We of the going-bankrupt-airline sysadmin set must make do with shoestring budgets and limping, eight-year-old hardware.

I most emphatically would not ever suggest making an ISO image of the tutorial notes in PDF format and pointing me in the direction of it to placate my rage. It would, after all, be illegal.

In the meanwhile, I do have to admit that the idea of an invisible loghost is quite interesting. I have a hard enough time keeping my servers' interfaces convinced that they should talk to the routers to actually make certain the ARP caches were maintained in a state which would make this possible, and I'd need to convince my masters to actually spend the money on it, but it is certainly cool.

Oddly enough, I'm actually enjoying my current task, which is documentation of my projects and then plowing through our knowledgebase as a preparation for cruft excision.

[bryant]

Ah, so here's the cool thing about invisible loghosts: the configuration necessary for that is also the configuration necessary to set up a snort-based IDS. You can use the same box for both purposes.

My laptop's CD drive is broken, which annoys me massively. But I will continue to refrain from illegalities when I get home.

In the meantime, it's kind of interesting to know that Sarbanes-Oxley has implications for this stuff. mjr was definitely of the opinion that you need strong internal security in order to maintain "proper internal controls," which is required by Sarbanes-Oxley.

[kniedzw.livejournal.com]

If only Sarbanes-Oxley required training budgets....

Nifty re: the IDS. I hadn't really thought about that. ...and it doesn't entirely make sense to me yet, but I haven't listened to a tutorial on it today, so I can be forgiven. ...at least until I read those links you have in del.icio.us.

[bryant]

Let's see...

Ah, here we go. Nice little article on the idea.

There is a serious dearth of people using del.icio.us to share lisa04 links. Sad.

[jeffwik.livejournal.com]

I have no idea what any of that was, but it sounded impressive.

Today I used an ultraviolet-visible spectrophotometer to measure the psuedo-first-order kinetics of protonation of a tridentate chelating ligand, 2,2'-dipicolylamine, by noncoordinating perchloric acid. The metal center was octahedral nickel (II) with perchlorate counterions, forming a tetrameric complex containing a distorted cubane at the center with tert-mu-hydroxo ligands at the open sites -- this is a novel structure, and the cubane core has appeared only once or twice before. I hope to publish it soon.

[bryant]

That kicks my ass.

[head58.livejournal.com]

I had jell-o today...

[multiplexer.livejournal.com]

Interestingly, mjr was actively working on these tools while at NFR, and they ran him out because they didn't think they would be commercially viable to their customers. Why bother working on something new when they had the NID to shop around?

Nonetheless, mjr is doing a hell of alot better than the idiots from NFR.

[that-cad.livejournal.com]

So there I was, standing in my closet, trying to decide whether to wear a black shirt under my black polo shirt or whether I should wear a tan shirt under my polo shirt — or whether I should even wear a polo shirt at all!

[bryant]

5,15,25,35,45,55 * * * * /usr/local/bin/retail -T /home/durrell/lib/retail \
/var/log/httpd/httpd-access_log | /home/durrell/bin/ref-filter.sh | /usr/local/bin/nbs \ 
-d /home/durrell/lib/httpd-ref | awk '$2 !~ /^-$/'