Subject: Re: FlowScan disk requirements?
From: Dave Plonka (plonka@doit.wisc.edu)
Date: Fri Mar 30 2001 - 08:13:00 CST
On Thu, Mar 22, 2001 at 12:07:31PM -0600, John Roman wrote: > We are in the process of configuring a system to collect and process > network flow data using FlowScan. We trying to determine how to guess > how much disk space we will need. > > We have read the INSTALL document which provides valuable information. > We have used argus on our main border interface for several months and it > implies that we average about 100,000 flows per 5minutes. We will collect > information with FlowScan on several other interfaces, but none will be as > busy as that one. The cflowd raw flow files (produced by the patched cflowd) contain one 55 byte record for each version 5 NetFlow record. So, 100,000 flows would produce raw flow files only ~5.5MB in size. Files that small can be processed in near-real-time, so theoretically FlowScan only needs room for a few files at a time *unless* you want to preserve raw flow files for audit, security/abuse investigations. Personally, at least half the reason for using flow export, cflowd, Cflow.pm, etc. is for that purpose so I retain them. FlowScan preserves the files if you create a "saved" directory. It's your responsibility to compress them if you wish. See the sample crontab entries in the distribution. > Could others share information about their disk space needs as related to > the number of interfaces, the sizes of flows, and how long they retain their > compressed flow files? The interfaces to disk space thing will vary wildly depending upon link capacity and, more importantly, the number of concurrent users (and therfore the number of flows) on your network. It's best to just try it out. That said, I would recommend a large disk just for FlowScan and flow file archives. I have a 35GB file system for FlowScan. I had been retaining for 14 days, but had to bump that back to 12 when things were getting tight. (DoS floods, if not caught, can produce very large five minute flow files - like up 150-200MB uncompressed.) > Or if there are other issues to consider, it would be > helpful to know that as well. This information would provide other data points > and help us know what we have to learn to improve our guess. If you're spec'ing out a machine, my recommendation is a multi-processor Intel box, like a fast PIII with good disk SCSI sub-system and multiple disks. I'm not necessarily advocating them, but we've been buying Dells. Today, on a GNU/Linux 2.2 kernel I think the best performance you'll see is with the "pset" patch described here: http://net.doit.wisc.edu/~plonka/FlowScan/INSTALL.html#Performance_Problems_ Lastly, I don't recommend using an NFS server for your RRD files! That's been trouble for a few folks. Dave -- plonka@doit.wisc.edu http://net.doit.wisc.edu/~plonka ARS:N9HZF Madison, WI -- Help mailto:majordomo@net.doit.wisc.edu and say "help" in message body Unsubscribe mailto:majordomo@net.doit.wisc.edu and say "unsubscribe flowscan" in message body Archive http://net.doit.wisc.edu/~plonka/list/flowscan/archive/
This archive was generated by hypermail 2b25 : Fri Mar 30 2001 - 08:17:46 CST