My custom MRTG collectors

The idea of servers statistics in general is to communicate a collector (a server which gathers and stores values and possibly makes some kind of data mining) with "clients". In pure MRTG environment collector asks every client for some values using SNMP (Simple Network Management Protocol). In the opposite the client can send a value by SNMP or can have a script and responses on request sent by collector in much more customisable way. This is the case. Collector in mrtg.cfg invokes commands like

ssh client /path/to/script.sh

It means, that there has to be an account with authentication based on certificates to avoid asking for password. Its enough to pass id_rsa.pub from collector to all clients and cat it to ~/.ssh/authorized_keys

Disk reads

#!/bin/sh
iostat | grep sda | awk '{ print $5; print $6; }'
uptime | awk -F, '{ print $1, $2; }'
hostname

HTTP requests

#!/bin/sh
LINES=`wc -l /var/log/apache2/access.log | awk '{ print $1; }'`
echo $LINES
echo $LINES
uptime | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
hostname

Load (not CPU utilization)

UPTIME=`uptime`
CURRENT=`echo "$UPTIME" | awk '{ a=NF-1; print $a*100; }'`
echo "$CURRENT"
echo "$CURRENT"
echo "$UPTIME" | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
hostname

Note: there are some theoretical articles stating, that the best load indicator is average load for 15 minutes. I use 5 minutes, but you can change it regardless of MRTG update interval. You use absolute/gauge/idontremember keyword to not calculate difference (as is on network interface)

HTTP requests

#!/bin/bash

if [[ ! $1 ]]
then
  echo "No arguments. Please provide server name"
  exit 0
fi

server=$1

if [[ $server == both ]]
then
  SERVER1=`echo "SHOW STATUS LIKE 'connections%'" | \
    mysql --host=server1 --password=dd13nr-314:-) | \
    grep Connections | awk '{ print $2; }'`
  SERVER2=`echo "SHOW STATUS LIKE 'connections%'" | \
    mysql --host=server2 --password=dd13nr-314:-)
--password=smladmin | \
    grep Connections | awk '{ print $2; }'`
  echo "$SERVER1"
  echo "$SERVER2"
  uptime | awk '{ gsub(/,/, ""); print $3, $4; }'
  echo "Server1 and Server2"
else
  CURRENT=`echo "SHOW STATUS LIKE 'connections%'" | \
    mysql --host=$server --password=dd13nr-314:-) | \
    grep Connections | awk '{ print $2; }'`
  echo "$CURRENT"
  echo "$CURRENT"
  uptime | awk '{ gsub(/,/, ""); print $3, $4, $5; }'
  echo $server
fi

Mail and SPAM stats

Slightly modified Craig Sanders' mailstat:

mrtg-mailstats.pl

This script is called by MRTG:

Target[postfix]: `ssh mail_server /usr/local/bin/mrtg-mailstats.pl` / 5
Options[postfix]: gauge, growright
 . . .

and

Target[spam]: `ssh mail_server /usr/local/bin/mrtg-spamstats.pl` / 5
Options[spam]: gauge, growright
 . . .

The result is divided by 5 [minutes], because script returns absolute values. The option gauge stops MRTG calculating a value per second, so we have to calculate it on our own. The result is in mail messages per minute.

#!/usr/bin/perl

$source = `hostname`;
chomp $source;

#$uptime = `uptime`;
#$uptime =~ s/^.*\s+up\s+//;
#$uptime =~ s/,\s+\d+\s+users,.*//;

$uptime = `uptime`;
chomp $uptime;
$uptime =~ s/.*up //;
$uptime =~ s/(\d),.*/$1/;

$datafile = "/var/lib/mrtg/mailstat.old";

open(OLD,"<$datafile");
while() {
  chomp;
  ($key,$val) = split /=/;
  $old{$key} = $val;
}
close(OLD);

$mailstats = "/usr/local/bin/mailstats.pl";
open(STAT,"$mailstats|") || die "couldn't open pipe to $mailstats: $!";
while() {
  chomp;
  ($type,$count) = split;
  ($what,undef) = split(/:/, $type);
  $new{$what} += $count;
}
close(STAT);

print $new{RECEIVED}-$old{RECEIVED},"\n",$new{SENT}-$old{SENT},"\n","$uptime\n$source\n" if ($old{RECEIVED});

# save old stats
open(OLD,">$datafile");
foreach (keys %new) {
  print OLD "$_=$new{$_}\n";
}
close(OLD);

Daemon reading mail log file

This script assumes that there is a spam-killer on 10.0.0.1 and all mail messages are forwarded to internal company's mail server on 100.100.1.1:

#!/usr/bin/perl

# update-mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;
use File::Tail;
$debug = 0;

$mail_log = '/var/log/mail.log';
$stats_file = '/tmp/stats.db';

$db = tie(%stats, "DB_File", "$stats_file", O_CREAT|O_RDWR, 0666, $DB_HASH)
  || die ("Cannot open $stats_file");

#my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,tail=>-1,debug=>$debug));
my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,debug=>$debug));

while () {
  if (/status=sent/) {
    # 10.0.0.1 is Amavisd with SpamAssassin
    next unless /relay=10.0.0.1/;
    # 100.100.1.1 is relay host where only legitimate mail messages are passed.
    if (/relay=100.100.1.1/) {
      # count received smtp messages
      $stats{"RECEIVED:smtp"} += 1;
    } else {
      # count sent messages
      if (/relay=([^,]+)/o) {
        $relay = $1;
        $stats{"SENT:$relay"} += 1;
      } else {
        $stats{"SENT:smtp"} +=1;
      }
    }
    $db->sync;
  }
}

untie $logref;
untie %stats;

DB viewer

#!/usr/bin/perl

# mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;

$|=1;

$stats_file = '/var/lib/mrtg/stats.db';

tie(%foo, "DB_File", "$stats_file", O_RDONLY, 0666, $DB_HASH) || die ("Cannot open $stats_file");

foreach (sort keys %foo) {
  print "$_ $foo{$_}\n";
}
untie %foo;

SPAM statistics

SPAM scripts are modified Mail scripts and counts all incoming and rejected connections:

#!/usr/bin/perl

# update-mailstats.pl
#
# Copyright Craig Sanders 1999
#
# this script is licensed under the terms of the GNU GPL.

use DB_File;
use File::Tail;
$debug = 0;

$mail_log = '/var/log/mail.log';
$stats_file = '/var/lib/mrtg/spam_stats.db';

$db = tie(%stats, "DB_File", "$stats_file", O_CREAT|O_RDWR, 0666, $DB_HASH)
  || die ("Cannot open $stats_file");

my $logref=tie(*LOG,"File::Tail",(name=>$mail_log,debug=>$debug));

while () {
  if (/ connect from/) {
    # SMTP connections
    $stats{"CONN"} += 1;
    $db->sync;
  } elsif (/(reject|warning)/) {
    # count rejected smtp messages
    $stats{"REJECT"} += 1;
    $db->sync;
  }
}

untie $logref;
untie %stats;

Database viewer is the same, only the database file name is changed.

MRTG collector (mrtg-spamstats.pl) has changed db name and hash keys:

15c15
< $datafile = "/var/lib/mrtg/mailstat.old";
---
> $datafile = "/var/lib/mrtg/spamstat.old";
25,26c25,26
< $mailstats = "/usr/local/bin/mailstats.pl";
< open(STAT,"$mailstats|") || die "couldn't open pipe to $mailstats: $!";
---
> $spamstats = "/usr/local/bin/spamstats.pl";
> open(STAT,"$spamstats|") || die "couldn't open pipe to $spamstats: $!";
35c35
< print $new{RECEIVED}-$old{RECEIVED},"\n",$new{SENT}-$old{SENT},"\n","$uptime\n$source\n" if ($old{RECEIVED});
---
> print $new{CONN}-$old{CONN},"\n",$new{REJECT}-$old{REJECT},"\n","$uptime\n$source\n" if ($old{CONN});

...