Subject: RE: [locker, gzip'ping and] Purging old flows files
From: Martinez, Samson (SMartinez@LocaleSystems.com)
Date: Mon Nov 13 2000 - 11:56:04 CST
Dave,
Didn't want to reply with this to the group...
Worked like a champ! Sincerest thanks.
Samson
Samson Martinez
Locale Systems Corporation
1613 Capital of TX Hwy S.
Suite 201
Austin, TX 78746
Email: smartinez@localesystems.com
Voice: 512-329-9805 X109
FAX : 512-329-9903
WEB: www.localesystems.com
-----Original Message-----
From: Dave Plonka [mailto:plonka@doit.wisc.edu]
Sent: Monday, November 13, 2000 11:16 AM
To: flowscan@net.doit.wisc.edu
Cc: Martinez, Samson
Subject: Re: [locker, gzip'ping and] Purging old flows files
On Mon, Nov 13, 2000 at 04:56:47PM -0000, Martinez, Samson wrote:
> I am preparing to setup the automatic removal of old flow files from the
> @PREFIX@/saved directory. However, I'm a bit confused about the example
> crontab showing the compressing and removal of these files, specifically
> this entry:
>
> # gzip the saved flow files:
> 2,7,12,17,22,27,32,37,42,47,52,57 * * * * test -d @prefix@/saved && cd
> @prefix@/saved && @prefix@/bin/locker -ne .gzip_lock "@KSH_PATH@ -c
> '@LS_PATH@ flows.[0-9]!(*.gz) 2>/dev/null | @XARGS_PATH@ -n1 @GZIP_PATH@'"
> #
>
> What is the ".gzip_lock" entry specify?
It is a file on which the locker utility does file-locking, to be sure
that you only have one command running at a time. That is its sole
purpose in life.
Try out locker on its own to learn how it works, e.g.:
$ touch /tmp/.lock
$ locker -e /tmp/.lock sleep 60 & # get an exclusive lock, and run
sleep
$ locker -ne /tmp/.lock date # non-blocking, exclusive lock
flock failed: Resource temporarily unavailable
$ locker -e /tmp/.lock date # blocking, exclusive lock
With locker, as a nice side-effect, you can say:
$ fuser .gzip_lock
- or -
$ lsof .gzip_lock
To see if any process is running in that directory, currently gzip'ping
the files.
> Is this a file that is supposed to
> exist somewhere?
Yes, you must create it yourself, as described in the FLowScan INSTALL
doc:
http://net.doit.wisc.edu/~plonka/FlowScan/INSTALL.html#Preserving_Old_Flow_F
iles
preferable prior to scheduling the cron job do this:
$ cd saved
$ touch .gzip_lock
Note that you will get an email message from the cron job saying
"Resource Temporarily Unavailable" or something to that effect,
whenever the previous gzip job is still running.
If it helps, here's the two cron jobs I run under Linux to gzip raw
flow files and remove old ones after `n' hours (e.g. 336):
#
# gzip the saved flow files:
2,7,12,17,22,27,32,37,42,47,52,57 * * * * test -d /var/local/flows/saved
&& cd /var/local/flows/saved && /var/local/flows/bin/locker -ne .gzip_lock
"/usr/bin/ksh -c '/bin/ls flows.[0-9]!(*.gz) 2>/dev/null | /usr/bin/xargs
-n1 --no-run-if-empty /bin/gzip'"
#
# Purge the flow files:
# find(1) -mtime +1 was insufficient - I want to delete them as soon as
they're
# `n' hours old:
0 * * * * /usr/bin/find /var/flows/saved -type f -name 'flows.*' -print
|/usr/bin/perl -e '$now = time; $seconds = 336*60*60; while (<>) { chomp;
(@_ = stat $_) && ($now - $_[9] > $seconds) && print $_, "\n" }'
|/usr/bin/xargs /bin/rm -f
Dave
--
plonka@doit.wisc.edu http://net.doit.wisc.edu/~plonka ARS:N9HZF Madison,
WI
--
Help mailto:majordomo@net.doit.wisc.edu and say "help" in message
body
Unsubscribe mailto:majordomo@net.doit.wisc.edu and say
"unsubscribe flowscan" in message body
Archive http://net.doit.wisc.edu/~plonka/list/flowscan/archive/
--
Help mailto:majordomo@net.doit.wisc.edu and say "help" in message body
Unsubscribe mailto:majordomo@net.doit.wisc.edu and say
"unsubscribe flowscan" in message body
Archive http://net.doit.wisc.edu/~plonka/list/flowscan/archive/
This archive was generated by hypermail 2b25 : Mon Nov 13 2000 - 12:02:16 CST