2008-05-02

My custom MRTG collectors

The idea of servers statistics in general is to communicate a collector (a server which gathers and stores values and possibly makes some kind of data mining) with "clients". In pure MRTG environment collector asks every client for some values using SNMP (Simple Network Management Protocol). In the opposite the client can send a value by SNMP or can have a script and responses on request sent by collector in much more customisable way. This is the case. Collector in mrtg.cfg invokes commands like

ssh client /path/to/script.sh


It means, that there has to be an account with authentication based on certificates to avoid asking for password. Its enough to pass id_rsa.pub from collector to all clients and cat it to ~/.ssh/authorized_keys

Disk reads



#!/bin/sh
iostat | grep sda | awk '{ print $5; print $6; }'
uptime | awk -F, '{ print $1, $2; }'
hostname


HTTP requests



#!/bin/sh
LINES=`wc -l /var/log/apache2/access.log | awk '{ print $1; }'`
echo $LINES
echo $LINES
uptime | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
hostname


Load (not CPU utilization)



UPTIME=`uptime`
CURRENT=`echo "$UPTIME" | awk '{ a=NF-1; print $a*100; }'`
echo "$CURRENT"
echo "$CURRENT"
echo "$UPTIME" | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
hostname


Note: there are some theoretical articles stating, that the best load indicator is average load for 15 minutes. I use 5 minutes, but you can change it regardless of MRTG update interval. You use absolute/gauge/idontremember keyword to not calculate difference (as is on network interface)

HTTP requests



#!/bin/bash

if [[ ! $1 ]]
then
echo "No arguments. Please provide server name"
exit 0
fi

server=$1

if [[ $server == both ]]
then
SERVER1=`echo "SHOW STATUS LIKE 'connections%'" | \
mysql --host=server1 --password=dd13nr-314:-) | \
grep Connections | awk '{ print $2; }'`
SERVER2=`echo "SHOW STATUS LIKE 'connections%'" | \
mysql --host=server2 --password=dd13nr-314:-)
--password=smladmin | \
grep Connections | awk '{ print $2; }'`
echo "$SERVER1"
echo "$SERVER2"
uptime | awk '{ gsub(/,/, ""); print $3, $4; }'
echo "Server1 and Server2"
else
CURRENT=`echo "SHOW STATUS LIKE 'connections%'" | \
mysql --host=$server --password=dd13nr-314:-) | \
grep Connections | awk '{ print $2; }'`
echo "$CURRENT"
echo "$CURRENT"
uptime | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
echo $server
fi


Mail and SPAM stats


Slightly modified Craig Sanders' mailstat:

mrtg-mailstats.pl


This script is called by MRTG:

Target[postfix]: `ssh mail_server /usr/local/bin/mrtg-mailstats.pl` / 5
Options[postfix]: gauge, growright
. . .




and


Target[spam]: `ssh mail_server /usr/local/bin/mrtg-spamstats.pl` / 5
Options[spam]: gauge, growright
. . .




The result is divided by 5 [minutes], because script returns absolute values. The option gauge stops MRTG calculating a value per second, so we have to calculate it on our own. The result is in mail messages per minute.


#!/usr/bin/perl

$source = `hostname`;
chomp $source;

#$uptime = `uptime`;
#$uptime =~ s/^.*\s+up\s+//;
#$uptime =~ s/,\s+\d+\s+users,.*//;

$uptime = `uptime`;
chomp $uptime;
$uptime =~ s/.*up //;
$uptime =~ s/(\d),.*/$1/;

$datafile = "/var/lib/mrtg/mailstat.old";

open(OLD,"<$datafile");
while() {
chomp;
($key,$val) = split /=/;
$old{$key} = $val;
}
close(OLD);

$mailstats = "/usr/local/bin/mailstats.pl";
open(STAT,"$mailstats|") || die "couldn't open pipe to $mailstats: $!";
while() {
chomp;
($type,$count) = split;
($what,undef) = split(/:/, $type);
$new{$what} += $count;
}
close(STAT);

print $new{RECEIVED}-$old{RECEIVED},"\n",$new{SENT}-$old{SENT},"\n","$uptime\n$source\n" if ($old{RECEIVED});

# save old stats
open(OLD,">$datafile");
foreach (keys %new) {
print OLD "$_=$new{$_}\n";
}
close(OLD);


Daemon reading mail log file



This script assumes that there is a spam-killer on 10.0.0.1 and all mail messages are forwarded to internal company's mail server on 100.100.1.1:




#!/usr/bin/perl

# update-mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;
use File::Tail;
$debug = 0;

$mail_log = '/var/log/mail.log';
$stats_file = '/tmp/stats.db';

$db = tie(%stats, "DB_File", "$stats_file", O_CREAT|O_RDWR, 0666, $DB_HASH)
|| die ("Cannot open $stats_file");

#my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,tail=>-1,debug=>$debug));
my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,debug=>$debug));

while () {
if (/status=sent/) {
# 10.0.0.1 is Amavisd with SpamAssassin
next unless /relay=10.0.0.1/;
# 100.100.1.1 is relay host where only legitimate mail messages are passed.
if (/relay=100.100.1.1/) {
# count received smtp messages
$stats{"RECEIVED:smtp"} += 1;
} else {
# count sent messages
if (/relay=([^,]+)/o) {
$relay = $1;
$stats{"SENT:$relay"} += 1;
} else {
$stats{"SENT:smtp"} +=1;
}
}
$db->sync;
}
}

untie $logref;
untie %stats;


DB viewer



#!/usr/bin/perl

# mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;

$|=1;

$stats_file = '/var/lib/mrtg/stats.db';

tie(%foo, "DB_File", "$stats_file", O_RDONLY, 0666, $DB_HASH) || die ("Cannot open $stats_file");

foreach (sort keys %foo) {
print "$_ $foo{$_}\n";
}
untie %foo;


SPAM statistics



SPAM scripts are modified Mail scripts and counts all incoming and rejected connections:

#!/usr/bin/perl

# update-mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;
use File::Tail;
$debug = 0;

$mail_log = '/var/log/mail.log';
$stats_file = '/var/lib/mrtg/spam_stats.db';

$db = tie(%stats, "DB_File", "$stats_file", O_CREAT|O_RDWR, 0666, $DB_HASH)
|| die ("Cannot open $stats_file");

my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,debug=>$debug));

while () {
if (/ connect from/) {
# SMTP connections
$stats{"CONN"} += 1;
$db->sync;
} elsif (/(reject|warning)/) {
# count rejected smtp messages
$stats{"REJECT"} += 1;
$db->sync;
}
}

untie $logref;
untie %stats;


Database viewer is the same, only the database file name is changed.

MRTG collector (mrtg-spamstats.pl) has changed db name and hash keys:

15c15
< $datafile = "/var/lib/mrtg/mailstat.old";
---
> $datafile = "/var/lib/mrtg/spamstat.old";
25,26c25,26
< $mailstats = "/usr/local/bin/mailstats.pl";
< open(STAT,"$mailstats|") || die "couldn't open pipe to $mailstats: $!";
---
> $spamstats = "/usr/local/bin/spamstats.pl";
> open(STAT,"$spamstats|") || die "couldn't open pipe to $spamstats: $!";
35c35
< print $new{RECEIVED}-$old{RECEIVED},"\n",$new{SENT}-$old{SENT},"\n","$uptime\n$source\n" if ($old{RECEIVED});
---
> print $new{CONN}-$old{CONN},"\n",$new{REJECT}-$old{REJECT},"\n","$uptime\n$source\n" if ($old{CONN});


...

2 comments:

Pawel said...

To hasło to prawdziwe? :-)

Artur Kaminski said...

A jak sądzisz? Oczywiście, że po Twoim komentarzu pobiegłem do konsoli i pozmieniałem hasła na wszystkich MySQLach :D
.
A tak na serio, nie sądzisz, że ta odrobina dreszczyku u czytających (mam nadzieję, że to liczba mnoga ;-)) 'ubarwia' trochę bloga? Wszędzie widzę 'secret' albo w wersji zaawansowanej 'sseeccrreett'. Szybciej jest napisać '2fb[p231f[1' - wszystkie dziesięć palców można zaangażować.

Pozdrawiam starosłowiańskim "Thank you for watching" :-)