Graph ZFS details with Cacti (Nexenta)


Today, as promised a long time ago, and therefore seriously overdue, a short writeup on how to graph ZFS details from Nexenta on Cacti.

On the Nexenta host

Firstly, add some extends to you SNMPD.CONF (here is some background on how to do this ). To do this log in (ssh) as admin to your Nexenta box, then obtain root privileges (su).

Now start  the nmc and run the command setup network service snmp-agent edit-settings . this allows you to edit the snmpd.conf file on a Nexenta host. Do not go about editing these files directly in /etc/snmp because it will not work that way

Now add your extends to the snmpd.conf file, i googled a bit to get some commands to return ZFS details, for our servers we added the following. The snmp-agent edit-settings  will open the snmpd.conf file in VI, so remember: a is to append, 😡 is to save, dd is to delete a line.

We added the following extends to our Nexenta server

extend .1.3.6.1.4.1.2012.88 zpool_name /bin/bash -c “zpool list -H -o name”
extend .1.3.6.1.4.1.2021.88 zpool_snap /bin/bash -c “zpool list -Ho name|for zpool in `xargs`;do zfs get -rHp -o value usedbysnapshots $zpool|awk -F: ‘{sum+=$1} END{print sum}’;done”
extend .1.3.6.1.4.1.2021.88 zpool_used /bin/bash -c “zpool list -Ho name|xargs zfs get -Hp -o value used”
extend .1.3.6.1.4.1.2021.88 zpool_data_used /bin/bash -c “zpool list -Ho name|for zpool in `xargs`;do snap=`zfs get -rHp -o value usedbysnapshots $zpool|awk -F: ‘{sum+=$1} END{print sum}’`;pool=`zfs get -Hp -o value used $zpool`; echo $pool $snap|awk ‘{print (\$1-\$2);}’;done”
extend .1.3.6.1.4.1.2021.88 zpool_available /bin/bash -c “zpool list -Ho name|xargs zfs get -Hp -o value available”
extend .1.3.6.1.4.1.2021.88 zpool_capacity /bin/bash -c “zpool list -H -o capacity”
extend .1.3.6.1.4.1.2021.85 arc_meta_max /bin/bash -c “echo ::arc | mdb -k| grep arc_meta_max|tr -cd ‘[:digit:]'”
extend .1.3.6.1.4.1.2021.85 arc_meta_used /bin/bash -c “echo ::arc | mdb -k| grep arc_meta_used|tr -cd ‘[:digit:]'”
extend .1.3.6.1.4.1.2021.85 arc_size /bin/bash -c “echo ::arc | mdb -k| grep -w size|tr -cd ‘[:digit:]'”
extend .1.3.6.1.4.1.2021.85 arc_meta_limit /bin/bash -c “echo ::arc | mdb -k| grep arc_meta_limit|tr -cd ‘[:digit:]'”
extend .1.3.6.1.4.1.2021.85 arc_meta_c_max /bin/bash -c “echo ::arc | mdb -k| grep c_max|tr -cd ‘[:digit:]'”
extend .1.3.6.1.4.1.2021.89 arc_hits /bin/bash -c “kstat -p ::arcstats:hits| cut -s -f 2”
extend .1.3.6.1.4.1.2021.89 arc_misses /bin/bash -c “kstat -p ::arcstats:misses| cut -s -f 2”
extend .1.3.6.1.4.1.2021.89 arc_l2_hits /bin/bash -c “kstat -p ::arcstats:l2_hits| cut -s -f 2”
extend .1.3.6.1.4.1.2021.89 arc_l2_misses /bin/bash -c “kstat -p ::arcstats:l2_misses| cut -s -f 2”
extend .1.3.6.1.4.1.2021.90 vopstats_zfs_nread /bin/bash -c “kstat -p ::vopstats_zfs:nread | cut -s -f 2”
extend .1.3.6.1.4.1.2021.90 vopstats_zfs_nwrite /bin/bash -c “kstat -p ::vopstats_zfs:nwrite | cut -s -f 2”
extend .1.3.6.1.4.1.2021.90 vopstats_zfs_read_bytes /bin/bash -c “kstat -p ::vopstats_zfs:read_bytes | cut -s -f 2”
extend .1.3.6.1.4.1.2021.90 vopstats_zfs_write_bytes /bin/bash -c “kstat -p ::vopstats_zfs:write_bytes | cut -s -f 2”

I know on Extends you normally would not have to provide an OID, but I like to provide them anyway so I know where to look for the SNMP OID.

After adding these, save the file and say yes to the question to reload the file after the save. Now check the configuration with:

setup network service snmp-agent confcheck

and then restart the snmpd with:

setup network service snmp-agent restart

Your work is now done on the Nexenta host, you can check the settings with an snmpwalk command to see if it actually works

On CACTI

I assume you have an SNMP enabled device set up to point to your Nexenta server, if not this would be a good time to do so. SNMP V2c works for me.

Now import the following XML graph templates on the end of this post to your Cacti server (I’ve got these from this forum but had to modify them quite a bit for them to see get the data form the correct SNMP OID’s.

Now add these templates to your device, and create the graphs. If you are lucky you will get some pretty pictures with some usefull information, especially the one on L2ARC cache turned out to be quite useful to us.

Good luck, and if you have any questions, post  them.

 

Cheers

Continue reading “Graph ZFS details with Cacti (Nexenta)”

Cacti script to get temperatures from an ILO device.


We wanted to experiment with different airflow scenarios in our DC and did not want to get in to expensive data center power management tools, so we decided to use a simple cacti server and the ILO temperature measurements from servers in certain spots in the racks to get a historic graph of inflow air temperatures, you get the idea.

But man!, Cacti can be a pain in the butt!, and so can be the HP/ILO SNMP implementation. As you might have noticed the SNMP/OID does not push out measured temperature sensors, sigh.. To get around this hurdle we had to hack up a quick script, and since it took me quite some time to google all components together i’d thought why not put them here for others to enjoy, saving the world some time here.

Our setup, a simple vanilla Ubuntu 12.04 LTS VM,  with Cacti from the Ubuntu Repo (0.8.7i), I assume I don’t need to explain how to set this up. So to get stuff working, first we get the HP Lights-Out XML PERL Scripting Sample (I fetched this one) for Linux, untar and install.

Copy out the files Get_EmHealth.xml and locfg.pl to the /usr/share/cacti/site/scripts folder, chown them to root (as Cacti runs its scripts as root)

Add your ILO credentials to the EmHealth.xml, I could go as far as parameterizing them for Cacti to provide, but since we use the same across all ILO’s in a separate management network, i did not bother

<RIBCL VERSION=”2.21″>
<LOGIN USER_LOGIN=”your user name” PASSWORD=”your password“>
<SERVER_INFO MODE=”read”>
<GET_EMBEDDED_HEALTH />
</SERVER_INFO>
</LOGIN>
</RIBCL>

Next, I created this little PERL script

$command = “/usr/bin/perl -X /usr/share/cacti/site/scripts/locfg.pl -s $ARGV[0] -f /usr/share/cacti/site/scripts/Get_EmHealth.xml”;

#$command = “/usr/bin/perl -X /usr/share/cacti/site/scripts/locfg.pl -s 192.168.3.104 -f /usr/share/cacti/site/scripts/Get_EmHealth.xml”;

$output = `$command`;

@lines = split(/\n/,$output);

foreach $line(@lines){
if (index ($line ,”LOCATION VALUE”) != -1) {
$line =~ s/\s+/_/g;
if ($line =~ /”(.+?)”/) { print”$1:”;}}
if (index ($line ,”CURRENTREADING VALUE”) != -1) {
if ($line =~ /”(.+?)”/) { print”$1 “;}}
}

What this basically does is parse the XML (ish) output from locfg.pl, find the Location Value and CurrentReading Value (which are sensors and values) and write them out as a valid Cacti script return string for by DL385g5 this is:

CPU:62 CPU_1:50 CPU_2:44 Memory_a:57 Memory_b:46 System:62 Ambient:21

As you can see I also trip out all spaces and replace them by underscores to keep the output valid for Cacti. So now we can run this script in a Cacti Data Input Method, see this link on how to do that, and as long as we provide the EXACT output fields, we will get values back to push in to RDD. I am only interested in the Ambient sensors, but maybe you like to graph more..

Hope this helps someone save some time

 

Cheers

 

-Fault