Monitor EMC Avamar server and backups with Zabbix

De Wiki de Romain RUDIGER
Aller à : navigation, rechercher

Introduction

This page show you how-to monitor an Avamar backup appliance (physical or virtual).

Zabbix version: 2.0.5 and 2.2.2
Avamar type: single node physical appliance
Avamar OS: RHEL 5.8
Avamar version: 6.0.1.66 and 6.1.2-47

Monitored elements

You will monitor these Avamar metrics:

  • Active sessions
  • Backup Deduplication Ratio
  • Bytes protected
  • Capacity
  • Capacity used
  • Last GC age
  • Last GC chunks deleted
  • Last GC data recovered
  • Last GC duration
  • Last GC passes
  • Last GC result
  • Last valid checkpoint age
  • Last valid checkpoint duration
  • Load
  • Utilization

And for each client+dataset these values:

  • Backup - <client> (<dataset>): age
  • Backup - <client> (<dataset>): bytes scanned
  • Backup - <client> (<dataset>): duration
  • Backup - <client> (<dataset>): number of files

The template include these triggers for the server:

  • Last checkpoint is too old on {HOST.NAME}
  • Last garbage collector is too old on {HOST.NAME}
  • Last garbage collector recovered less than $1 on {HOST.NAME}
  • Low free space on {HOST.NAME}
  • No data protected on {HOST.NAME}

And these triggers for each backup-dataset:

  • No backup since $1 for <client> (<dataset>) by {HOST.NAME}
  • No files backup for <client> (<dataset>) by {HOST.NAME}

The Zabbix agent configuration

The zabbix agent have this UserParameter in its configuration file:

UserParameter=avamar.get[*],sudo -H /etc/zabbix/avamar.get.sh $1 $2 $3

To enable sudo without password, add this line in the sudoers file:

zabbix ALL = (root) NOPASSWD: /etc/zabbix/avamar.get.sh

The data collection script (avamar.get.sh)

The file is located in the zabbix folder but is owned by root for security reason:

-rwxr-xr-x  1 root root 10195 Feb 26 17:45 /etc/zabbix/avamar.get.sh

Content of the avamar.get.sh script:

#!/bin/bash

export PGUSER=viewuser
export PGPASSWORD=viewuser1
export PGPORT=5555
export PGDATABASE=mcdb
export PGHOST=127.0.0.1

function usage
{
	echo "Valid options are:
	server-sessions
	server-capacity
	server-capacity-used
	server-utilization
	server-bytes-protected
	server-load
	checkpoint-age
	checkpoint-duration
	garbage-passes
	garbage-chunks-deleted
	garbage-bytes-recovered
	garbage-result
	garbage-age
	garbage-duration
	backup-last-duration client_name [dataset]
	backup-last-age client_name [dataset]
	backup-last-bytes_scanned client_name [dataset]
	backup-last-num_of_files client_name [dataset]" >&2
	exit 1
}

case "$1" in
	server-dedup-rate-bckgrd)
		[[ -r $serverDedupRate ]] && cat $serverDedupRate || echo 0
	;;
	server-dedup-rate)
		# Get client with domain informations list
		enabledClients="$(/usr/local/avamar/bin/mccli client show --recursive=true --verbose=true --normalize=true | grep -Ev "^(.*CLI command completed successfully\.$|Client[[:space:]]+Domain[[:space:]]+|-+([[:space:]]+-+)+|$)" | awk '$3="Yes" && $4="Yes" {print $2"|"$1}')"
		echo $enabledClients | head -1 | grep -Eq "^/[[:alnum:]]*\|[[:alnum:]]+"
		[[ $? -ne 0 ]] && echo 0 && exit
		# Compute the total backup size for all these clients
		ORIGIFS=$IFS
		IFS=$(echo -en "\n\b")
		totalBackupSize=0
		for domainClient in $enabledClients; do
			client=$(echo $domainClient | awk -F"|" '{print $2}')
			domain=$(echo $domainClient | awk -F"|" '{print $1}')
			backupSize=$(/usr/local/avamar/bin/mccli backup show --name=$client --domain=$domain --normalize=true | grep -Ev "^(.*CLI command completed successfully\.$|Created[[:space:]]+LabelNum[[:space:]]+|-+([[:space:]]+-+)+|$)" | awk '{print $3}' | tr '\n' '+')
			echo $backupSize | grep -Eq "^([[:digit:]]+\+)+$"
			if [[ $? -ne 0 ]]; then
				backupSize=0
			else
				backupSize=$(echo $backupSize | sed -r "s/\+$//" | bc)
				echo $backupSize | grep -Eq "^[[:digit:]]+$"
				[[ $? -ne 0 ]] && backupSize=0
			fi
			totalBackupSize=$(($totalBackupSize + $backupSize))
		done
		# Get the capacity used
		capacityUsed=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Capacity used" | awk '{print $3}')
                echo $capacityUsed | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
		# Compute the deduplication ratio
		echo $(echo "scale=2;$totalBackupSize / $capacityUsed" | bc)
	;;
	server-sessions)
		sessions=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Active sessions" | awk '{print $3}')
		echo $sessions | grep -Eq "^[[:digit:]]+$"
		[[ $? -ne 0 ]] && echo 0 && exit
		echo $sessions
	;;
	server-capacity)
		capacity=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Total capacity" | awk '{print $3}')
		echo $capacity | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
		echo $capacity
	;;
	server-capacity-used)
		capacityUsed=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Capacity used" | awk '{print $3}')
		echo $capacityUsed | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
                echo $capacityUsed
	;;
	server-utilization)
		capacity=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Total capacity" | awk '{print $3}')
		echo $capacity | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
		capacityUsed=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep "Capacity used" | awk '{print $3}')
		echo $capacityUsed | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
		echo $( echo "scale=2; $capacityUsed * 100 / $capacity " | bc )
	;;
	server-bytes-protected)
		bp=$(/usr/local/avamar/bin/mccli server show-prop --normalize=true | grep -E "Bytes protected[[:space:]]+[[:digit:]]+" | awk '{print $3}')
		echo $bp | grep -Eq "^[[:digit:]]+$"
		[[ $? -ne 0 ]] && echo 0 && exit
		echo $bp
	;;
	server-load)
		load=$(/usr/local/avamar/bin/mccli server show-services | grep "Load Average" | awk '{print $3}')
		echo $load | grep -Eq "^[[:digit:]]+(\.[[:digit:]]+|)$"
		[[ $? -ne 0 ]] && echo 0 && exit
		echo $load
	;;
	checkpoint-age)
		checkpoint=$(/usr/local/avamar/bin/mccli checkpoint show --normalize=true --verbose=true | grep -E -m 1 "[[:digit:]]+[[:space:]]+Validated[[:space:]]+[[:alpha:]]+")
                cpStartTime=$(echo $checkpoint | awk '{print $7}')
                echo $cpStartTime | grep -Eq "^[[:digit:]]+$"
                [[ $? -ne 0 ]] && echo 0 && exit
		#checkpoint=$(avmain lscp --xmlperline=15 | grep -E "validcheck=\"true\"" | grep -E "errors=\"0\"" | grep -E "type=\"(full|rolling)\"")
		#cpStartTime=$(echo $checkpoint | sed -e "s/.*starttime=//" | awk -F\" '{print $2}')
		echo $(( $(date +%s) - ${cpStartTime:0:10} ))
	;;
	checkpoint-duration)
		checkpoint=$(/usr/local/avamar/bin/mccli checkpoint show --normalize=true --verbose=true | grep -E -m 1 "[[:digit:]]+[[:space:]]+Validated[[:space:]]+[[:alpha:]]+")
		cpStartTime=$(echo $checkpoint | awk '{print $7}')
		echo $cpStartTime | grep -Eq "^[[:digit:]]+$"
		[[ $? -ne 0 ]] && echo 0 && exit
		cpEndTime=$(echo $checkpoint | awk '{print $8}')
		echo $cpEndTime | grep -Eq "^[[:digit:]]+$"
		[[ $? -ne 0 ]] && echo 0 && exit
		#checkpoint=$(avmain lscp --xmlperline=15 | grep -E "validcheck=\"true\"" | grep -E "errors=\"0\"" | grep -E "type=\"(full|rolling)\"")
		#cpStartTime=$(echo $checkpoint | sed -e "s/.*nodestarttime=//" | awk -F\" '{print $2}')
		#cpEndTime=$(echo $checkpoint | sed -e "s/.*nodefinishedtime=//" | awk -F\" '{print $2}')
		echo $(( ${cpEndTime:0:10} - ${cpStartTime:0:10} )) 
	;;
	garbage-passes)
		query="SELECT passes FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
                result=$(echo $query | psql -t -A)
                echo $result | grep -Eq "^[[:digit:]]+$"
                if [[ $? -eq 0 ]]; then
                        echo $result
                else
                        echo 0
                fi
	;;
	garbage-chunks-deleted)
		query="SELECT chunks_deleted FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
                result=$(echo $query | psql -t -A)
                echo $result | grep -Eq "^[[:digit:]]+$"
                if [[ $? -eq 0 ]]; then
                        echo $result
                else
                        echo 0
                fi
	;;
	garbage-bytes-recovered)
		query="SELECT bytes_recovered FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
                if [[ $? -eq 0 ]]; then
                        echo $result
                else
                        echo 0
                fi
		#echo $(avmain gcstatus | grep -E "megabytes-recovered=" | sed -e 's/\"//g' -e 's/\ //g' | awk -F= '{print $2}')	
	;;
	garbage-result)
		query="SELECT result FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		[[ "$result" == "OK" ]] && echo 0 || echo 1
		#echo $(avmain gcstatus | grep -E "result=" | sed -e 's/\"//g' -e 's/\ //g' | awk -F= '{print $2}')
	;;
	garbage-age)
		query="SELECT EXTRACT(EPOCH FROM now()::timestamp(0) ) - EXTRACT(EPOCH FROM start_time::timestamp(0) )FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
                if [[ $? -eq 0 ]]; then
                        echo $result
                else
                        echo 0
                fi
		#gcStartTime=$(avmain gcstatus | grep -E "start-time=" | sed -e 's/\"//g' -e 's/\ //g' | awk -F= '{print $2}')
		#echo $(( $(date +%s) - $gcStartTime ))
	;;
	garbage-duration)
		query="SELECT EXTRACT(EPOCH FROM end_time::timestamp(0) ) - EXTRACT(EPOCH FROM start_time::timestamp(0) )FROM v_gcstatus ORDER BY start_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
		if [[ $? -eq 0 ]]; then
                        echo $result
                else
                        echo 0
                fi
		#echo $(avmain gcstatus | grep -E "elapsed-time=" | sed -e 's/\"//g' -e 's/\ //g' | awk -F= '{print $2}')
	;;
	backup-last-duration)
		[[ -z $2 ]] && usage
		[[ ! -z $3 ]] && dataset="AND dataset = '$3' " || dataset=""
		query="SELECT EXTRACT(EPOCH FROM completed_ts::timestamp(0)) - EXTRACT(EPOCH FROM started_ts::timestamp(0)) FROM v_activities_2       
			WHERE client_name = '$2' $dataset
			AND status_code = 30000
			AND type = 'Scheduled Backup' ORDER BY recorded_date_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
		if [[ $? -eq 0 ]]; then
			echo $result
		else
			echo 0
		fi
	;;
	backup-last-age)
		[[ -z $2 ]] && usage
		[[ ! -z $3 ]] && dataset="AND dataset = '$3' " || dataset=""
		set -f
		query="SELECT EXTRACT(EPOCH FROM ( now()::timestamp(0) - ( started_ts::timestamp(0) + EXTRACT (timezone from now()) * interval '1s' ) ) ) FROM v_activities_2
			WHERE client_name = '$2' $dataset
			AND status_code = 30000
			AND type = 'Scheduled Backup' ORDER BY recorded_date_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
		if [[ $? -eq 0 ]]; then 
			echo $result
		else
			echo 0
		fi
		set +f
	;;
	backup-last-bytes_scanned)
		[[ -z $2 ]] && usage
		[[ ! -z $3 ]] && dataset="AND dataset = '$3' " || dataset=""
		query="SELECT bytes_scanned FROM v_activities_2
			WHERE client_name = '$2' $dataset
			AND status_code = 30000
			AND type = 'Scheduled Backup' ORDER BY recorded_date_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
		if [[ $? -eq 0 ]]; then
			echo $result
		else
			echo 0
		fi
	;;
	backup-last-num_of_files)
		[[ -z $2 ]] && usage
		[[ ! -z $3 ]] && dataset="AND dataset = '$3' " || dataset=""
		query="SELECT num_of_files FROM v_activities_2
			WHERE client_name = '$2' $dataset
			AND status_code = 30000
			AND type = 'Scheduled Backup' ORDER BY recorded_date_time DESC LIMIT 1;"
		result=$(echo $query | psql -t -A)
		echo $result | grep -Eq "^[[:digit:]]+$"
		if [[ $? -eq 0 ]]; then
			echo $result
		else
			echo 0
		fi
	;;
	*)
	usage
	;;
	esac
exit 0

The Avamar Server Template

This template must be associated with each Avamar utility node.

Media:Monitor_EMC_Avamar_server_template.xml

Monitor a Backup

Introduction

The idea is to be able to script the configuration of the items and triggers to add for each client-dataset entry in Zabbix.

The solution use the Zabbix API and a php library written by Domi Barton (PhpZabbixApi - Archive copy under GNU GPL General Public License)).

Install the PhpZabbixApi library

wget http://zabbixapi.confirm.ch/download.php?file=library
tar -xvf PhpZabbixApi_Library.tar
cd PhpZabbixApi_Library

Create the php script

The script is quite simple but it's a good way to put your hands on Zabbix configuration automation.

The following script only manage items creation. Feel free to add the remove and update functions !

<?php

// load ZabbixApi
require 'ZabbixApiAbstract.class.php';
require 'ZabbixApi.class.php';

// check CLI arguments
if ($argc != 6 || in_array($argv[1], array('--help', '-help', '-h', '-?'))) {
	echo "Script usage:\n";
	echo "php ".$argv[0]." <username> <password> <avamar server name> <backup client name> <backup client dataset>\n";
} else {
	try {
		//start Zabbix API function
		$api = new ZabbixApi('https://127.0.0.1/zabbix/api_jsonrpc.php', $argv[1], $argv[2]);
		// use extended output for all further requests
		$api->setDefaultParams(array(
			'output' => 'extend'
		));
		//get the hostid
		$host = $api->hostGet(array('filter'=>array('host'=>$argv[3])));
		if ( ! isset($host) || count($host) != 1 ) {
			echo "Can't get the host object with the name ".$argv[3].".\n";
			exit;
		}
		echo "Host ".$argv[3]." found (hostid=".$host[0]->hostid.").\n";
		//Check if the items already exist
		echo "Check if the items doesn't already exist...";
		$items = $api->itemGet(array('filter'=>array('key_'=>'avamar.get[backup-last-age,'.$argv[4].','.$argv[5].']'),'hostids'=>$host[0]->hostid));
		if ( isset($items) && count($items) == 1 ) {
			echo "error item '".$items[0]->key_."' already exist!\n";
			exit;
		}
		$items = $api->itemGet(array('filter'=>array('key_'=>'avamar.get[backup-last-bytes_scanned,'.$argv[4].','.$argv[5].']'),'hostids'=>$host[0]->hostid));
		if ( isset($items) && count($items) == 1 ) {
			echo "error item '".$items[0]->key_."' already exist!\n";
			exit;
		}
		$items = $api->itemGet(array('filter'=>array('key_'=>'avamar.get[backup-last-duration,'.$argv[4].','.$argv[5].']'),'hostids'=>$host[0]->hostid));
		if ( isset($items) && count($items) == 1 ) {
			echo "error item '".$items[0]->key_."' already exist!\n";
			exit;
		}
		$items = $api->itemGet(array('filter'=>array('key_'=>'avamar.get[backup-last-num_of_files,'.$argv[4].','.$argv[5].']'),'hostids'=>$host[0]->hostid));
		if ( isset($items) && count($items) == 1 ) {
			echo "error item '".$items[0]->key_."' already exist!\n";
			exit;
		}
		//get the first interfaceid of type agent
		echo "\nGet the first interfaces of type agent...";
		$interfaces = $api->hostinterfaceGet(array('filter'=>array('type'=>'1'),'hostids'=>$host[0]->hostid));
		if ( ! isset($interfaces) || count($interfaces) == 0 ) {
			echo "error can't find an interface of type agent.\n";
			exit;
		}
		echo " ".$interfaces[0]->interfaceid."\n";
		//get the application id named 'Avamar_Backup'
		echo "Get the application id named 'Avamar_Backup' for this host...";
		$applications = $api->applicationGet(array('filter'=>array('name'=>'Avamar_Backup','hostid'=>$host[0]->hostid)));
		if ( ! isset($applications) || count($applications) != 1 ) {
			echo "the application doesn't existe, create...";
			$result = $api->applicationCreate(array('name'=>'Avamar_Backup', 'hostid'=>$host[0]->hostid));
			$applicationID=$result->applicationids[0];
		} else {
			$applicationID=$applications[0]->applicationid;
		}
		echo "$applicationID\n";
		//get the number of items before (for information)
		echo "Get the number of each items by type on this host:\n";
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-age,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupAgeCount=count($items);
		} else {
			$backupAgeCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-bytes_scanned,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupBytesCount=count($items);
		} else {
			$backupBytesCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-duration,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupDurationCount=count($items);
		} else {
			$backupDurationCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-num_of_files,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupNumOfFilesCount=count($items);
		} else {
			$backupNumOfFilesCount=0;
		}
		//get the number of triggers
		$triggers = $api->triggerGet(array('search'=>array('description'=>'No backup since $1 for ','hostid'=>$host[0]->hostid)));
		if ( isset($triggers) ) {
			$backupAgeTriggerCount=count($triggers);
                } else {
			$backupAgeTriggerCount=0;
		}
		$triggers = $api->triggerGet(array('search'=>array('description'=>'No files backup for ','hostid'=>$host[0]->hostid)));
		if ( isset($triggers) ) {
			$backupNumOfFilesTriggerCount=count($triggers);
                } else {
			$backupNumOfFilesTriggerCount=0;
		}
		echo " -number of item with the key avamar.get[backup-last-age,*,*]: ".$backupAgeCount."\n";
		echo " -number of item with the key avamar.get[backup-last-bytes_scanned,*,*]: ".$backupBytesCount."\n";
		echo " -number of item with the key avamar.get[backup-last-duration,*,*]: ".$backupDurationCount."\n";
		echo " -number of item with the key avamar.get[backup-last-num_of_files,*,*]: ".$backupNumOfFilesCount."\n";
		echo " -number of trigger with the name 'No backup since $1 for ': ".$backupAgeTriggerCount."\n";
		echo " -number of trigger with the name 'No files backup for ': ".$backupNumOfFilesTriggerCount."\n";
		//add the items
		echo "Create the item 'backup-last-age'...";
		$result = $api->itemCreate(array('name'=>'Backup - $2 ($3): age', 'key_'=>'avamar.get[backup-last-age,'.$argv[4].','.$argv[5].']', 'hostid'=>$host[0]->hostid, 'type'=>'0', 'value_type'=>'3', 'interfaceid'=>$interfaces[0]->interfaceid, 'applications'=>array($applicationID), 'delay'=>'120', 'units'=>'s'));
		echo $result->itemids[0]."\n";
		echo "Create the item 'backup-last-bytes_scanned'...";
		$result = $api->itemCreate(array('name'=>'Backup - $2 ($3): bytes scanned', 'key_'=>'avamar.get[backup-last-bytes_scanned,'.$argv[4].','.$argv[5].']', 'hostid'=>$host[0]->hostid, 'type'=>'0', 'value_type'=>'3', 'interfaceid'=>$interfaces[0]->interfaceid, 'applications'=>array($applicationID), 'delay'=>'120', 'units'=>'B'));
		echo $result->itemids[0]."\n";
		echo "Create the item 'backup-last-duration'...";
		$result = $api->itemCreate(array('name'=>'Backup - $2 ($3): duration', 'key_'=>'avamar.get[backup-last-duration,'.$argv[4].','.$argv[5].']', 'hostid'=>$host[0]->hostid, 'type'=>'0', 'value_type'=>'3', 'interfaceid'=>$interfaces[0]->interfaceid, 'applications'=>array($applicationID), 'delay'=>'120', 'units'=>'s'));
		echo $result->itemids[0]."\n";
		echo "Create the item 'backup-last-num_of_files'...";
		$result = $api->itemCreate(array('name'=>'Backup - $2 ($3): number of files', 'key_'=>'avamar.get[backup-last-num_of_files,'.$argv[4].','.$argv[5].']', 'hostid'=>$host[0]->hostid, 'type'=>'0', 'value_type'=>'3', 'interfaceid'=>$interfaces[0]->interfaceid, 'applications'=>array($applicationID), 'delay'=>'120'));
		echo $result->itemids[0]."\n";
		//add the triggers
		echo "Create the trigger on 'backup-last-age'...";
		$result = $api->triggerCreate(array('description'=>'No backup since $1 for '.$argv[4].' ('.$argv[5].') by {HOST.NAME}','expression'=>'{'.$argv[3].':avamar.get[backup-last-age,'.$argv[4].','.$argv[5].'].last(0)}>3d','priority'=>'2'));
		echo $result->triggerids[0]."\n";
		echo "Create the trigger on 'backup-last-num_of_files'...";
		$result = $api->triggerCreate(array('description'=>'No files backup for '.$argv[4].' ('.$argv[5].') by {HOST.NAME}','expression'=>'{'.$argv[3].':avamar.get[backup-last-num_of_files,'.$argv[4].','.$argv[5].'].last(0)}=0','priority'=>'2'));
		echo $result->triggerids[0]."\n";
		//get the number of items
		echo "Get the number of each items by type on this host:\n";
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-age,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupAgeCount=count($items);
		} else {
			$backupAgeCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-bytes_scanned,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupBytesCount=count($items);
		} else {
			$backupBytesCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-duration,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupDurationCount=count($items);
		} else {
			$backupDurationCount=0;
		}
		$items = $api->itemGet(array('search'=>array('key_'=>'avamar.get[backup-last-num_of_files,'),'hostids'=>$host[0]->hostid));
		if ( isset($items) ) {
			$backupNumOfFilesCount=count($items);
		} else {
			$backupNumOfFilesCount=0;
		}
		//get the number of triggers
		$triggers = $api->triggerGet(array('search'=>array('description'=>'No backup since $1 for ','hostid'=>$host[0]->hostid)));
		if ( isset($triggers) ) {
			$backupAgeTriggerCount=count($triggers);
		} else {
			$backupAgeTriggerCount=0;
		}
		$triggers = $api->triggerGet(array('search'=>array('description'=>'No files backup for ','hostid'=>$host[0]->hostid)));
		if ( isset($triggers) ) {
			$backupNumOfFilesTriggerCount=count($triggers);
                } else {
			$backupNumOfFilesTriggerCount=0;
		}
		echo " -number of item with the key avamar.get[backup-last-age,*,*]: ".$backupAgeCount."\n";
		echo " -number of item with the key avamar.get[backup-last-bytes_scanned,*,*]: ".$backupBytesCount."\n";
		echo " -number of item with the key avamar.get[backup-last-duration,*,*]: ".$backupDurationCount."\n";
		echo " -number of item with the key avamar.get[backup-last-num_of_files,*,*]: ".$backupNumOfFilesCount."\n";
		echo " -number of trigger with the name 'No backup since $1 for ': ".$backupAgeTriggerCount."\n";
		echo " -number of trigger with the name 'No files backup for ': ".$backupNumOfFilesTriggerCount."\n";
		echo "End of the script.";
	} catch(Exception $e) {
		// Exception in ZabbixApi catched
		echo $e->getMessage();
	}
}
?>

Usage and example

The command line usage:

Script usage:
php Zabbix_manage_avamar_backup_client.php <username> <password> <avamar server name> <backup client name> <backup client dataset>

Example to add the Avamar client astra with the dataset /Astra to the Avamar server backup.novalan.priv:

php Zabbix_manage_avamar_backup_client.php "romain.rudiger" "passwd" "backup.novalan.priv" "astra" "/Astra"
Host backup.novalan.priv found (hostid=10065).
Check if the items doesn't already exist...
Get the first interfaces of type agent... 30195
Get the application id named 'Avamar_Backup' for this host...the application doesn't existe, create...380
Get the number of each items by type on this host:
 -number of item with the key avamar.get[backup-last-age,*,*]: 0
 -number of item with the key avamar.get[backup-last-bytes_scanned,*,*]: 0
 -number of item with the key avamar.get[backup-last-duration,*,*]: 0
 -number of item with the key avamar.get[backup-last-num_of_files,*,*]: 0
 -number of trigger with the name 'No backup since $1 for ': 0
 -number of trigger with the name 'No files backup for ': 0
Create the item 'backup-last-age'...24308
Create the item 'backup-last-bytes_scanned'...24309
Create the item 'backup-last-duration'...24310
Create the item 'backup-last-num_of_files'...24311
Create the trigger on 'backup-last-age'...13804
Create the trigger on 'backup-last-num_of_files'...13805
Get the number of each items by type on this host:
 -number of item with the key avamar.get[backup-last-age,*,*]: 1
 -number of item with the key avamar.get[backup-last-bytes_scanned,*,*]: 1
 -number of item with the key avamar.get[backup-last-duration,*,*]: 1
 -number of item with the key avamar.get[backup-last-num_of_files,*,*]: 1
 -number of trigger with the name 'No backup since $1 for ': 1
 -number of trigger with the name 'No files backup for ': 1
End of the script.

For the virtual machines, the client name displayed in Avamar Administrator will differ from the one used in the MCS database. To get the one to use with the php script:

export PGUSER=viewuser
export PGPASSWORD=viewuser1
export PGPORT=5555
export PGDATABASE=mcdb
export PGHOST=127.0.0.1
echo "select full_domain_name, client_name from v_clients;" | psql   
                    full_domain_name                    |            client_name            
--------------------------------------------------------+-----------------------------------
...
 /sphere.novalan.priv/Supervisor_UAJ0ROWDWnHgcgvlLM1bwA | Supervisor_UAJ0ROWDWnHgcgvlLM1bwA
 /sphere.novalan.priv/Remi7_UqNn6Q6zfuBI75YMbY8jnw      | Remi7_UqNn6Q6zfuBI75YMbY8jnw
...

Screenshots

Avamar server items configuration
Avamar server items last data
Avamar client items configuration
Avamar client items last data
Avamar client triggers