In this article, I discuss network security monitoring (NSM) and introduce one specific form of NSM data -- transaction data.
In my books
- Alert data: Judgments made by tools that inspect network traffic.
- Statistical data: Overall summaries or profiles of network traffic.
- Session data: Conversations or flows generated from network traffic.
- Full content data: Actual packets collected by storing network traffic.
These four forms of NSM data are extremely useful, but I'd like to use this article to introduce a fifth form that I have begun to apply to my daily defensive operations: transaction data. I define transaction data as application-specific records generated from network traffic. Let me demonstrate an example before I try to explain the reasons to generate this form of NSM data.
HTTP records as transaction data
I'll demonstrate the creation of NSM transaction data for HTTP using Jason Bittel's Httpry program. Here I run Httpry on a live sensor interface and tell it to generate records using the HTTP fields you see on the command line.
# httpry -i bge0 -o /tmp/httpry_31mar09.txt -q -u richard -s timestamp,source-ip,x-forwarded-for,direction,dest-ip,method,host,request-uri,user-agent,referer,status-code,http-version,reason-phrase
A selection from the log file appears below.
# httpry version 0.1.3
# Fields: timestamp,source-ip,x-forwarded-for,direction,dest-ip,method,host,request-uri,user-agent,referer,status-code,http-version,reason-phrase
03/31/2009 20:52:41 126.96.36.199 - < 188.8.131.52 - - - - - 200 HTTP/1.0 OK
03/31/2009 20:52:45 184.108.40.206 192.168.2.106 > 220.127.116.11 GET mv.vatican.va /3_EN/pages/x-Schede/SDRs/SDRs_03_02_020.html Mozilla/5.0 (X11; U; Linux i686; en-US; rv:18.104.22.168) Gecko/2009032711 Ubuntu/8.04 (hardy) Firefox/3.0.8 http://www.bejtlich.net/lab.html - HTTP/1.0 -
03/31/2009 20:52:45 22.214.171.124 - < 126.96.36.199 - - - - - 200 HTTP/1.1 OK
For comparison's sake, here is the same record one might retrieve from a Squid proxy log:
1238547165.722 1357 192.168.2.106 TCP_MISS/200 6623 GET http://mv.vatican.va/3_EN/pages/x-Schede/SDRs/SDRs_03_02_020.html - DIRECT/188.8.131.52 text/html "http://www.bejtlich.net/lab.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:184.108.40.206) Gecko/2009032711 Ubuntu/8.04 (hardy) Firefox/3.0.8"
So, why generate the HTTP record using Httpry when you have access to Squid proxy logs? Well, imagine that you do not have access to Squid proxy logs. Maybe you don't have proxies in place. Maybe you do have proxies but don't have logging enabled, for performance reasons. Maybe you do have logging enabled, but you do not have easy access to the logs. Thanks to an application-specific traffic inspection tool like Httpry, you can collect NSM transaction records for HTTP.
Why not generate these records from the full content data when you need them? You could do that, but you will probably have better luck generating, compressing and storing text records like those created by Httpry. You can also specify exactly which fields you need.
You could easily extend this sort of transaction data approach to a variety of protocols. Another powerful example involves DNS requests and replies. Records of hosts that try to resolve various names could be invaluable for incident detection and response. This month's Conficker malware shows the prominence of DNS records, if you can collect them. Using something as simple as Tcpdump with a filter for port 53 can do the trick.
If you're wondering whether there are other programs to generate NSM transaction data, you're correct. In future articles we will examine a few, including Bro, a tool that can also generate the HTTP records seen in this article.
Richard Bejtlich is director of incident response at General Electric and author of the TaoSecurity Blog.
This was first published in April 2009