Project

General

Profile

Task #8950

Task #8949: Migrate dCache store nodes monitoring from zabbix to check_mk

Planning and figuraing out the scope

Added by Chih-Hao Huang over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Start date:
06/12/2015
Due date:
06/17/2015
% Done:

100%

Estimated time:
4.00 h
Spent time:
Duration: 6

Description

Looking at what need to be done.


Related issues

Precedes (1 day) CMS dCache - Task #8952: Prototyping one monitoring migration from zabbix to check_mkResolved07/23/201507/30/2015

History

#1 Updated by Chih-Hao Huang over 4 years ago

  • Status changed from New to Accepted
  • % Done changed from 0 to 10

It is easier to hack into zabbix database to get the information out.

#2 Updated by Chih-Hao Huang over 4 years ago

There are:
227 nodes under "cmssl6Template dcachepools",
8 nodes under "cmssl6template dcachesrv", and
36 nodes under "dcachepool_tape" (these are also in cmssl6Template dcachepools).

In "cmssl6Template dcachepools", there are 77 items and 63 triggers.
In "cmssl6template dcachesrv", there are 66 items and 69 triggers.
In "dcachepool_tape", there are no additional items nor triggers.

#4 Updated by Chih-Hao Huang over 4 years ago

  • Related to Task #8952: Prototyping one monitoring migration from zabbix to check_mk added

#5 Updated by Chih-Hao Huang over 4 years ago

  • Related to deleted (Task #8952: Prototyping one monitoring migration from zabbix to check_mk)

#6 Updated by Chih-Hao Huang over 4 years ago

  • Precedes Task #8952: Prototyping one monitoring migration from zabbix to check_mk added

#7 Updated by Chih-Hao Huang over 4 years ago

  • Estimated time changed from 8.00 h to 4.00 h

#8 Updated by Chih-Hao Huang over 4 years ago

  • % Done changed from 10 to 60
[root@cmszabbix1 HUANGCH]# cat triggers.sql 
select
    h.host,
    i.name as "item",
    f.function,
    f.parameter,
    t.expression,
    t.description as "trigger" 
from
    triggers t
    join functions f on f.triggerid = t.triggerid
    join items i on i.itemid = f.itemid
    join hosts h on h.hostid = i.hostid
where
    h.host in ('cmsTemplate-phedex', 'dcachepool_tape', 'cmssl6template dcachesrv', 'cmssl6Template dcachepools')
order by
    h.host, item, trigger
;
[root@cmszabbix1 HUANGCH]# psql -U zabbix zabbix -f triggers.sql 
            host            |                         item                         | function | parameter |   expression   |                                        trigger                                        
----------------------------+------------------------------------------------------+----------+-----------+----------------+---------------------------------------------------------------------------------------
 cmssl6Template dcachepools | Check host certificate                               | last     | 0         | {16875}=102    | Host certificate on {HOSTNAME} will expire within 28 days
 cmssl6Template dcachepools | Check host certificate                               | last     | 0         | {16858}=101    | {HOSTNAME} host certificate has expired.
 cmssl6Template dcachepools | check iptables stopped                               | last     | 0         | {16846}>0      | iptables stopped on  {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | diff     | 0         | {16876}>0      | /etc/services has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | diff     | 0         | {16848}>0      | /usr/bin/ssh has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | diff     | 0         | {16892}>0      | /usr/sbin/sshd has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | clock skew verify                                    | last     | 0         | {16898}=2      | clock skewed on {HOSTNAME}
 cmssl6Template dcachepools | dcache service verification                          | last     | 0         | {16899}>0      | failure of dcache service
 cmssl6Template dcachepools | Free disk space on $1 in %                           | last     | 0         | {16883}<10     | Low free disk space on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free disk space on $1 in %                           | last     | 0         | {16900}<5      | Low free disk space on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free disk space on $1 in %                           | last     | 0         | {16896}<10     | Low free disk space on {HOSTNAME} volume /var/lib/pqsql-puppet
 cmssl6Template dcachepools | Free memory                                          | last     | 0         | {16897}<10000  | Lack of free memory on server {HOSTNAME}
 cmssl6Template dcachepools | Free number of inodes on $1 in %                     | last     | 0         | {16852}<10     | Low number of free inodes on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free swap space                                      | last     | 0         | {16843}<100000 | Lack of free swap space on {HOSTNAME}
 cmssl6Template dcachepools | Host information                                     | diff     | 0         | {16845}>0      | Host information was changed on {HOSTNAME}
 cmssl6Template dcachepools | Host name                                            | diff     | 0         | {16884}>0      | Hostname was changed on {HOSTNAME}
 cmssl6Template dcachepools | Host uptime (in sec)                                 | last     | 0         | {16853}<600    | {HOSTNAME} has just been restarted
 cmssl6Template dcachepools | ICMP loss                                            | min      | 5m        | {89989}>10     | Ping loss is too high on {HOST.NAME}
 cmssl6Template dcachepools | ICMP ping                                            | max      | #3        | {89258}=0      | {HOST.NAME} is unavailable
 cmssl6Template dcachepools | ICMP response time                                   | avg      | 5m        | {89475}>0.15   | Response time is too high on {HOST.NAME}
 cmssl6Template dcachepools | Maximum number of processes                          | last     | 0         | {16862}<256    | Configured max number of processes is too low on {HOSTNAME}
 cmssl6Template dcachepools | Number of processes                                  | last     | 0         | {16878}>2000   | Too many processes on {HOSTNAME}
 cmssl6Template dcachepools | Number of running crond                              | last     | 0         | {16844}=0      | crond is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running gmond                              | last     | 0         | {16894}=0      | gmond is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running ntpd                               | last     | 0         | {16857}=0      | ntpd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes $1                       | last     | 0         | {16867}<1      | Zabbix_agentd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes sshd                     | last     | 0         | {16864}<1      | Sshd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes syslogd                  | last     | 0         | {16890}<1      | Syslogd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running puupet agent                       | last     | 0         | {16863}<1      | puppet agent is not running on {HOSTNAME}
 cmssl6Template dcachepools | OCS inventory check                                  | last     | 0         | {16885}=101    | No OCS inventory file found.  Possibly not running/configured/installed on {HOSTNAME}
 cmssl6Template dcachepools | OCS inventory check                                  | last     | 0         | {16889}=102    | The OCS inventory is out of date on {HOSTNAME}
 cmssl6Template dcachepools | OSG certificates                                     | last     | 0         | {16861}=101    | Expired OSG certificates
 cmssl6Template dcachepools | Processor load                                       | last     | 0         | {16851}>200    | Processor load is too high on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | last     |           | {109165}=4     | puppet:  Could not parse the YAML from the summary report for {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | last     |           | {108729}=2     | puppet: Could not read the summary report on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | last     |           | {108947}=3     | puppet:  job failures listed in the summary report on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | last     |           | {109383}=5     | puppet:   Summary report has not updated in at least 2 hours for {HOSTNAME}
 cmssl6Template dcachepools | SSH server is running                                | last     | 0         | {16854}=0      | SSH server is down on {HOSTNAME}
 cmssl6Template dcachepools | storage data1 mount                                  | last     | 0         | {16901}=101    | storage data1 mount failure
 cmssl6Template dcachepools | storage data2 mount                                  | last     | 0         | {16902}=101    | storage data2  mount failure
 cmssl6Template dcachepools | storage data3  mount                                 | last     | 0         | {16903}=101    | storage data3  mount failure
 cmssl6Template dcachepools | Swatch: bh:   1 [0 1]                                | last     | 0         | {16868}=101    | Swatch: BH alert
 cmssl6Template dcachepools | Swatch: Dcache mem error                             | last     | 0         | {16873}=101    | Swatch: Dcache memory Error reported
 cmssl6Template dcachepools | Swatch: DMA disabled                                 | last     | 0         | {16872}=101    | Swatch: DMA is disabled
 cmssl6Template dcachepools | Swatch: dma_intr: error=0x84                         | last     | 0         | {16870}=101    | Swatch: dma_intr: error=0x84
 cmssl6Template dcachepools | Swatch: dma_intr: status=0x40                        | last     | 0         | {16850}=101    | Swatch: dma_intr: status=0x40
 cmssl6Template dcachepools | Swatch: dma_intr: status=0x51                        | last     | 0         | {16869}=101    | Swatch: dma_intr: status=0x51
 cmssl6Template dcachepools | Swatch: drive not ready for command                  | last     | 0         | {16879}=101    | Swatch: drive not ready for command
 cmssl6Template dcachepools | Swatch: end_request: I/O error                       | last     | 0         | {16887}=101    | Swatch: end_request: I/O error
 cmssl6Template dcachepools | Swatch: ext2_write_inode: unable to read inode block | last     | 0         | {16891}=101    | Swatch: ext2_write_inode: unable to read inode block
 cmssl6Template dcachepools | Swatch: Hardware Error                               | last     | 0         | {16893}=101    | Swatch: Hardware Error reported
 cmssl6Template dcachepools | Swatch: irq:  0 [0 0]                                | last     | 0         | {16866}=101    | Swatch: irq: 0 [0 0]
 cmssl6Template dcachepools | Swatch: IRQ Timeout                                  | last     | 0         | {16874}=101    | Swatch: IRQ Timeout
 cmssl6Template dcachepools | Swatch: kernel: EXT3-fs error                        | last     | 0         | {16859}=101    | Swatch: kernel: EXT3-fs error
 cmssl6Template dcachepools | Swatch: kernel: I/O error                            | last     | 0         | {16860}=101    | Swatch:kernel: I/O error
 cmssl6Template dcachepools | Swatch: Message queue overflow                       | last     | 0         | {101994}=101   | Swatch: Dcache logs "Message queue overflow" reported
 cmssl6Template dcachepools | Swatch: reset success                                | last     | 0         | {16865}=101    | Swatch: reset success
 cmssl6Template dcachepools | Swatch: reset timed-out                              | last     | 0         | {16888}=101    | Swatch:reset timed-out
 cmssl6Template dcachepools | Swatch: status timeout                               | last     | 0         | {16856}=101    | Swatch: status timeout
 cmssl6Template dcachepools | Swatch: timeout waiting for DMA                      | last     | 0         | {16886}=101    | Swatch: timeout waiting for DMA
 cmssl6Template dcachepools | Swatch: wait_on_bh, CPU 0                            | last     | 0         | {16877}=101    | Swatch: wait_on_bh, CPU 0
 cmssl6Template dcachepools | Swatch: XFS error seen                               | last     | 0         | {16880}=101    | Swatch: XFS filesystem error
 cmssl6Template dcachepools | verify postgresql running                            | last     | 0         | {16847}>0      | postgresql not running on {HOSTNAME}
 cmssl6template dcachesrv   | Certificate expiration check                         | last     | 0         | {16837}=101    | Expired certificates
 cmssl6template dcachesrv   | Certificate expiration check                         | last     | 0         | {16839}=103    | Expired certificates
 cmssl6template dcachesrv   | Certificate expiration check                         | last     | 0         | {16838}=102    | Expired certificates
 cmssl6template dcachesrv   | Certificate Monitoring                               | last     | 0         | {16836}=101    | Certificate Monitoring
 cmssl6template dcachesrv   | Certificate Monitoring                               | last     | 0         | {16835}=103    | Certificate Monitoring
 cmssl6template dcachesrv   | Certificate Monitoring                               | last     | 0         | {16834}=102    | Certificate Monitoring
 cmssl6template dcachesrv   | Check for read-only filesystems                      | last     | 0         | {15894}#0      | problem with filesytem / on {HOSTNAME} - possibly read only
 cmssl6template dcachesrv   | Check host certificate                               | last     | 0         | {15870}=102    | Host certificate on {HOSTNAME} will expire within 28 days
 cmssl6template dcachesrv   | Check host certificate                               | last     | 0         | {15869}=101    | {HOSTNAME} host certificate has expired.
 cmssl6template dcachesrv   | check iptables stopped                               | last     | 0         | {15856}>0      | iptables stopped on  {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | diff     | 0         | {15902}>0      | /etc/services has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | diff     | 0         | {15903}>0      | /usr/bin/ssh has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | diff     | 0         | {15904}>0      | /usr/sbin/sshd has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | clock skew verify                                    | last     | 0         | {15859}=2      | clock skewed on {HOSTNAME}
 cmssl6template dcachesrv   | Crls Monitoring                                      | last     | 0         | {16840}=101    | Crls Monitoring
 cmssl6template dcachesrv   | Crls Monitoring                                      | last     | 0         | {16842}=102    | Crls Monitoring
 cmssl6template dcachesrv   | Crls Monitoring                                      | last     | 0         | {16841}=103    | Crls Monitoring
 cmssl6template dcachesrv   | dcache service verification                          | last     | 0         | {15860}>0      | failure of dcache service
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | last     | 0         | {15907}<5      | Low free disk space on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | last     | 0         | {15906}<10     | Low free disk space on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | last     | 0         | {15908}<10     | Low free disk space on {HOSTNAME} volume /var/lib/pqsql-puppet
 cmssl6template dcachesrv   | Free memory                                          | last     | 0         | {15909}<10000  | Lack of free memory on server {HOSTNAME}
 cmssl6template dcachesrv   | Free number of inodes on $1 in %                     | last     | 0         | {15905}<10     | Low number of free inodes on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free swap space                                      | last     | 0         | {15899}<100000 | Lack of free swap space on {HOSTNAME}
 cmssl6template dcachesrv   | Host information                                     | diff     | 0         | {15900}>0      | Host information was changed on {HOSTNAME}
 cmssl6template dcachesrv   | Host name                                            | diff     | 0         | {15898}>0      | Hostname was changed on {HOSTNAME}
 cmssl6template dcachesrv   | Host uptime (in sec)                                 | last     | 0         | {15901}<600    | {HOSTNAME} has just been restarted
 cmssl6template dcachesrv   | ICMP loss                                            | min      | 5m        | {90251}>10     | Ping loss is too high on {HOST.NAME}
 cmssl6template dcachesrv   | ICMP ping                                            | max      | #3        | {89231}=0      | {HOST.NAME} is unavailable
 cmssl6template dcachesrv   | ICMP response time                                   | avg      | 5m        | {89240}>0.15   | Response time is too high on {HOST.NAME}
 cmssl6template dcachesrv   | Maximum number of processes                          | last     | 0         | {15875}<256    | Configured max number of processes is too low on {HOSTNAME}
 cmssl6template dcachesrv   | Number of processes                                  | last     | 0         | {15888}>2000   | Too many processes on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running crond                              | last     | 0         | {15880}=0      | crond is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running gmond                              | last     | 0         | {15881}=0      | gmond is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running ntpd                               | last     | 0         | {15882}=0      | ntpd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes $1                       | last     | 0         | {15887}<1      | Zabbix_agentd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes sshd                     | last     | 0         | {15885}<1      | Sshd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes syslogd                  | last     | 0         | {15884}<1      | Syslogd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running puupet agent                       | last     | 0         | {15883}<1      | puppet agent is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running xinetd                             | last     | 0         | {15886}=0      | xinetd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | OCS inventory check                                  | last     | 0         | {15858}=101    | No OCS inventory file found.  Possibly not running/configured/installed on {HOSTNAME}
 cmssl6template dcachesrv   | OCS inventory check                                  | last     | 0         | {15857}=102    | The OCS inventory is out of date on {HOSTNAME}
 cmssl6template dcachesrv   | Processor load                                       | last     | 0         | {15897}>80     | Processor load is too high on {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | last     |           | {109623}=4     | puppet:  Could not parse the YAML from the summary report for {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | last     |           | {109614}=3     | puppet:  job failures listed in the summary report on {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | last     |           | {109632}=5     | puppet:   Summary report has not updated in at least 2 hours for {HOSTNAME}
 cmssl6template dcachesrv   | SSH server is running                                | last     | 0         | {15877}=0      | SSH server is down on {HOSTNAME}
 cmssl6template dcachesrv   | Swatch: bh:   1 [0 1]                                | last     | 0         | {15855}=101    | Swatch: BH alert
 cmssl6template dcachesrv   | Swatch:  dcache log reported a "CLUMPING_ISSUE"      | last     |           | {103803}=101   | Swatch: Dcache logs reported an "CLUMPING_ISSUE" 
 cmssl6template dcachesrv   | Swatch: DMA disabled                                 | last     | 0         | {15861}=101    | Swatch: DMA is disabled
 cmssl6template dcachesrv   | Swatch: dma_intr: error=0x84                         | last     | 0         | {15864}=101    | Swatch: dma_intr: error=0x84
 cmssl6template dcachesrv   | Swatch: dma_intr: status=0x40                        | last     | 0         | {15862}=101    | Swatch: dma_intr: status=0x40
 cmssl6template dcachesrv   | Swatch: dma_intr: status=0x51                        | last     | 0         | {15863}=101    | Swatch: dma_intr: status=0x51
 cmssl6template dcachesrv   | Swatch: drive not ready for command                  | last     | 0         | {15866}=101    | Swatch: drive not ready for command
 cmssl6template dcachesrv   | Swatch: end_request: I/O error                       | last     | 0         | {15867}=101    | Swatch: end_request: I/O error
 cmssl6template dcachesrv   | Swatch: ext2_write_inode: unable to read inode block | last     | 0         | {15872}=101    | Swatch: ext2_write_inode: unable to read inode block
 cmssl6template dcachesrv   | Swatch: Hardware Error                               | last     | 0         | {15871}=101    | Swatch: Hardware Error reported
 cmssl6template dcachesrv   | Swatch: irq:  0 [0 0]                                | last     | 0         | {15873}=101    | Swatch: irq: 0 [0 0]
 cmssl6template dcachesrv   | Swatch: IRQ Timeout                                  | last     | 0         | {15874}=101    | Swatch: IRQ Timeout
 cmssl6template dcachesrv   | Swatch: kernel: EXT3-fs error                        | last     | 0         | {15868}=101    | Swatch: kernel: EXT3-fs error
 cmssl6template dcachesrv   | Swatch: kernel: I/O error                            | last     | 0         | {15876}=101    | Swatch:kernel: I/O error
 cmssl6template dcachesrv   | Swatch: Message queue overflow                       | last     | 0         | {102764}=101   | Swatch: Dcache logs "Message queue overflow" reported
 cmssl6template dcachesrv   | Swatch: reset success                                | last     | 0         | {15892}=101    | Swatch: reset success
 cmssl6template dcachesrv   | Swatch: reset timed-out                              | last     | 0         | {15893}=101    | Swatch:reset timed-out
 cmssl6template dcachesrv   | Swatch: status timeout                               | last     | 0         | {15896}=101    | Swatch: status timeout
 cmssl6template dcachesrv   | Swatch: timeout waiting for DMA                      | last     | 0         | {15865}=101    | Swatch: timeout waiting for DMA
 cmssl6template dcachesrv   | Swatch: wait_on_bh, CPU 0                            | last     | 0         | {15910}=101    | Swatch: wait_on_bh, CPU 0
 cmssl6template dcachesrv   | Swatch: XFS error seen                               | last     | 0         | {15911}=101    | Swatch: XFS filesystem error
 cmssl6template dcachesrv   | verify postgresql running                            | last     | 0         | {15879}>0      | postgresql not running on {HOSTNAME}
 cmsTemplate-phedex         | check proxy lifetime                                 | last     | 0         | {86183}=101    | proxy expire in 1 day
 cmsTemplate-phedex         | check proxy lifetime                                 | last     | 0         | {86182}=103    | proxy expire in 3 days
 cmsTemplate-phedex         | monitor phedex agents                                | last     | 0         | {86179}=1      | probem with phedex agents on {HOSTNAME}
 cmsTemplate-phedex         | monitor phedex debug agents                          | last     | 0         | {86178}=1      | probem with debug phedex agents on {HOSTNAME}
 cmsTemplate-phedex         | monitor srmcp-eos agents                             | last     |           | {109601}=1     | srmcp-eos test failed
 cmsTemplate-phedex         | monitor xrootd disk                                  | last     |           | {112581}#0     | Problem with xrootd: not able to write to disk
 cmsTemplate-phedex         | monitor xrootd eos                                   | last     |           | {112589}#0     | Problem with xrootd: not able to write to eos
 cmsTemplate-phedex         | monitor xrootd tape                                  | last     |           | {112585}#0     | Problem with xrootd: not able to write to tape
 cmsTemplate-phedex         | Number of running debug                              | last     | 0         | {86174}=0      | Debug instance not running on {HOSTNAME}
 cmsTemplate-phedex         | Number of running debug watchdog                     | last     | 0         | {86175}=0      | watchdog for Debug instance not running on {HOSTNAME}
 cmsTemplate-phedex         | Number of running prod                               | last     | 0         | {86181}=0      | watchdog for Prod instance not running on {HOSTNAME}
 cmsTemplate-phedex         | PhEDEx agents                                        | last     | 0         | {86177}=0      | Prod instance not running on {HOSTNAME}
 cmsTemplate-phedex         | verify dcache mounted                                | last     | 0         | {86173}#0      | Problem with dcache mount on {HOSTNAME}
 cmsTemplate-phedex         | verify dcache mounted - test                         | last     | 0         | {86176}#0      | Problem with dcache mount on {HOSTNAME} -"TEST" 
 cmsTemplate-phedex         | verify pnfs mounted                                  | last     | 0         | {86180}#0      | Problem with pnfs mount on {HOSTNAME}
(147 rows)

[root@cmszabbix1 HUANGCH]#  

#9 Updated by Chih-Hao Huang over 4 years ago

  • % Done changed from 60 to 90

All triggers in 'cmsTemplate-phedex', 'dcachepool_tape', 'cmssl6template dcachesrv', and 'cmssl6Template dcachepools' host-groups.

[root@cmszabbix1 HUANGCH]# cat triggers.sql 
create temp table item_type
(
    id    integer,
    name    varchar
);

insert into item_type (id, name) values (0, 'Zabbix agent');
insert into item_type (id, name) values (1, 'SNMPv1 agent');
insert into item_type (id, name) values (2, 'Zabbix trapper');
insert into item_type (id, name) values (3, 'simple check');
insert into item_type (id, name) values (4, 'SNMPv2 agent');
insert into item_type (id, name) values (5, 'Zabbix internal');
insert into item_type (id, name) values (6, 'SNMPv3 agent');
insert into item_type (id, name) values (7, 'Zabbix agent (active)');
insert into item_type (id, name) values (8, 'Zabbix aggregate');
insert into item_type (id, name) values (9, 'web item');
insert into item_type (id, name) values (10, 'external check');
insert into item_type (id, name) values (11, 'database monitor');
insert into item_type (id, name) values (12, 'IPMI agent');
insert into item_type (id, name) values (13, 'SSH agent');
insert into item_type (id, name) values (14, 'TELNET agent');
insert into item_type (id, name) values (15, 'calculated');
insert into item_type (id, name) values (16, 'JMX agent');

select
    h.host as "host-group",
    i.name as "item",
    i.key_ as "key",
    it.name as "type",
    f.function,
    f.parameter,
    t.expression,
    t.description as "trigger" 
from
    triggers t
    join functions f on f.triggerid = t.triggerid
    join items i on i.itemid = f.itemid
    join hosts h on h.hostid = i.hostid
    join item_type it on it.id = i.type
where
    h.host in ('cmsTemplate-phedex', 'dcachepool_tape', 'cmssl6template dcachesrv', 'cmssl6Template dcachepools')
order by
    h.host, i.type, item, trigger
;
[root@cmszabbix1 HUANGCH]# psql -U zabbix zabbix -f triggers.sql | grep -v INSERT | grep -v CREATE
         host-group         |                         item                         |                    key                    |      type      | function | parameter |   expression   |                                        trigger                                        
----------------------------+------------------------------------------------------+-------------------------------------------+----------------+----------+-----------+----------------+---------------------------------------------------------------------------------------
 cmssl6Template dcachepools | Check host certificate                               | grid_cert                                 | Zabbix agent   | last     | 0         | {16875}=102    | Host certificate on {HOSTNAME} will expire within 28 days
 cmssl6Template dcachepools | Check host certificate                               | grid_cert                                 | Zabbix agent   | last     | 0         | {16858}=101    | {HOSTNAME} host certificate has expired.
 cmssl6Template dcachepools | check iptables stopped                               | check_iptablesStopped                     | Zabbix agent   | last     | 0         | {16846}>0      | iptables stopped on  {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | vfs.file.cksum[/etc/services]             | Zabbix agent   | diff     | 0         | {16876}>0      | /etc/services has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | vfs.file.cksum[/usr/bin/ssh]              | Zabbix agent   | diff     | 0         | {16848}>0      | /usr/bin/ssh has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | Checksum of $1                                       | vfs.file.cksum[/usr/sbin/sshd]            | Zabbix agent   | diff     | 0         | {16892}>0      | /usr/sbin/sshd has been changed on server {HOSTNAME}
 cmssl6Template dcachepools | clock skew verify                                    | clockskew                                 | Zabbix agent   | last     | 0         | {16898}=2      | clock skewed on {HOSTNAME}
 cmssl6Template dcachepools | dcache service verification                          | dcache_status                             | Zabbix agent   | last     | 0         | {16899}>0      | failure of dcache service
 cmssl6Template dcachepools | Free disk space on $1 in %                           | vfs.fs.size[/,pfree]                      | Zabbix agent   | last     | 0         | {16883}<10     | Low free disk space on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free disk space on $1 in %                           | vfs.fs.size[/,pfree]                      | Zabbix agent   | last     | 0         | {16900}<5      | Low free disk space on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free disk space on $1 in %                           | vfs.fs.size[/var/lib/pqsql-puppet ,pfree] | Zabbix agent   | last     | 0         | {16896}<10     | Low free disk space on {HOSTNAME} volume /var/lib/pqsql-puppet
 cmssl6Template dcachepools | Free memory                                          | vm.memory.size[free]                      | Zabbix agent   | last     | 0         | {16897}<10000  | Lack of free memory on server {HOSTNAME}
 cmssl6Template dcachepools | Free number of inodes on $1 in %                     | vfs.fs.inode[/,pfree]                     | Zabbix agent   | last     | 0         | {16852}<10     | Low number of free inodes on {HOSTNAME} volume /
 cmssl6Template dcachepools | Free swap space                                      | system.swap.size[,free]                   | Zabbix agent   | last     | 0         | {16843}<100000 | Lack of free swap space on {HOSTNAME}
 cmssl6Template dcachepools | Host information                                     | system.uname                              | Zabbix agent   | diff     | 0         | {16845}>0      | Host information was changed on {HOSTNAME}
 cmssl6Template dcachepools | Host name                                            | system.hostname                           | Zabbix agent   | diff     | 0         | {16884}>0      | Hostname was changed on {HOSTNAME}
 cmssl6Template dcachepools | Host uptime (in sec)                                 | system.uptime                             | Zabbix agent   | last     | 0         | {16853}<600    | {HOSTNAME} has just been restarted
 cmssl6Template dcachepools | Maximum number of processes                          | kernel.maxproc                            | Zabbix agent   | last     | 0         | {16862}<256    | Configured max number of processes is too low on {HOSTNAME}
 cmssl6Template dcachepools | Number of processes                                  | proc.num[]                                | Zabbix agent   | last     | 0         | {16878}>2000   | Too many processes on {HOSTNAME}
 cmssl6Template dcachepools | Number of running crond                              | proc.num[crond]                           | Zabbix agent   | last     | 0         | {16844}=0      | crond is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running gmond                              | proc.num[gmond]                           | Zabbix agent   | last     | 0         | {16894}=0      | gmond is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running ntpd                               | proc.num[ntpd]                            | Zabbix agent   | last     | 0         | {16857}=0      | ntpd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes $1                       | proc.num[zabbix_agentd]                   | Zabbix agent   | last     | 0         | {16867}<1      | Zabbix_agentd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes sshd                     | proc.num[sshd]                            | Zabbix agent   | last     | 0         | {16864}<1      | Sshd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running processes syslogd                  | proc.num[rsyslogd]                        | Zabbix agent   | last     | 0         | {16890}<1      | Syslogd is not running on {HOSTNAME}
 cmssl6Template dcachepools | Number of running puupet agent                       | proc.num[puppet]                          | Zabbix agent   | last     | 0         | {16863}<1      | puppet agent is not running on {HOSTNAME}
 cmssl6Template dcachepools | OSG certificates                                     | osg_certs                                 | Zabbix agent   | last     | 0         | {16861}=101    | Expired OSG certificates
 cmssl6Template dcachepools | Processor load                                       | system.cpu.load[,avg1]                    | Zabbix agent   | last     | 0         | {16851}>200    | Processor load is too high on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {109165}=4     | puppet:  Could not parse the YAML from the summary report for {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {108729}=2     | puppet: Could not read the summary report on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {108947}=3     | puppet:  job failures listed in the summary report on {HOSTNAME}
 cmssl6Template dcachepools | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {109383}=5     | puppet:   Summary report has not updated in at least 2 hours for {HOSTNAME}
 cmssl6Template dcachepools | SSH server is running                                | net.tcp.service[ssh]                      | Zabbix agent   | last     | 0         | {16854}=0      | SSH server is down on {HOSTNAME}
 cmssl6Template dcachepools | storage data1 mount                                  | data1                                     | Zabbix agent   | last     | 0         | {16901}=101    | storage data1 mount failure
 cmssl6Template dcachepools | storage data2 mount                                  | data2                                     | Zabbix agent   | last     | 0         | {16902}=101    | storage data2  mount failure
 cmssl6Template dcachepools | storage data3  mount                                 | data3                                     | Zabbix agent   | last     | 0         | {16903}=101    | storage data3  mount failure
 cmssl6Template dcachepools | verify postgresql running                            | postgresql_status                         | Zabbix agent   | last     | 0         | {16847}>0      | postgresql not running on {HOSTNAME}
 cmssl6Template dcachepools | OCS inventory check                                  | check_ocs                                 | Zabbix trapper | last     | 0         | {16885}=101    | No OCS inventory file found.  Possibly not running/configured/installed on {HOSTNAME}
 cmssl6Template dcachepools | OCS inventory check                                  | check_ocs                                 | Zabbix trapper | last     | 0         | {16889}=102    | The OCS inventory is out of date on {HOSTNAME}
 cmssl6Template dcachepools | Swatch: bh:   1 [0 1]                                | BH                                        | Zabbix trapper | last     | 0         | {16868}=101    | Swatch: BH alert
 cmssl6Template dcachepools | Swatch: Dcache mem error                             | OOMERROR                                  | Zabbix trapper | last     | 0         | {16873}=101    | Swatch: Dcache memory Error reported
 cmssl6Template dcachepools | Swatch: DMA disabled                                 | DMADISABLED                               | Zabbix trapper | last     | 0         | {16872}=101    | Swatch: DMA is disabled
 cmssl6Template dcachepools | Swatch: dma_intr: error=0x84                         | DMAINTRx84                                | Zabbix trapper | last     | 0         | {16870}=101    | Swatch: dma_intr: error=0x84
 cmssl6Template dcachepools | Swatch: dma_intr: status=0x40                        | DMAINTRx40                                | Zabbix trapper | last     | 0         | {16850}=101    | Swatch: dma_intr: status=0x40
 cmssl6Template dcachepools | Swatch: dma_intr: status=0x51                        | DMAINTRx51                                | Zabbix trapper | last     | 0         | {16869}=101    | Swatch: dma_intr: status=0x51
 cmssl6Template dcachepools | Swatch: drive not ready for command                  | DRVNOTREADY                               | Zabbix trapper | last     | 0         | {16879}=101    | Swatch: drive not ready for command
 cmssl6Template dcachepools | Swatch: end_request: I/O error                       | ENDREQUEST                                | Zabbix trapper | last     | 0         | {16887}=101    | Swatch: end_request: I/O error
 cmssl6Template dcachepools | Swatch: ext2_write_inode: unable to read inode block | INODEBLOCK                                | Zabbix trapper | last     | 0         | {16891}=101    | Swatch: ext2_write_inode: unable to read inode block
 cmssl6Template dcachepools | Swatch: Hardware Error                               | HARDWAREERROR                             | Zabbix trapper | last     | 0         | {16893}=101    | Swatch: Hardware Error reported
 cmssl6Template dcachepools | Swatch: irq:  0 [0 0]                                | IRQ                                       | Zabbix trapper | last     | 0         | {16866}=101    | Swatch: irq: 0 [0 0]
 cmssl6Template dcachepools | Swatch: IRQ Timeout                                  | IRQTIMEOUT                                | Zabbix trapper | last     | 0         | {16874}=101    | Swatch: IRQ Timeout
 cmssl6Template dcachepools | Swatch: kernel: EXT3-fs error                        | EXT3ERROR                                 | Zabbix trapper | last     | 0         | {16859}=101    | Swatch: kernel: EXT3-fs error
 cmssl6Template dcachepools | Swatch: kernel: I/O error                            | KERNELIOERROR                             | Zabbix trapper | last     | 0         | {16860}=101    | Swatch:kernel: I/O error
 cmssl6Template dcachepools | Swatch: Message queue overflow                       | MQO                                       | Zabbix trapper | last     | 0         | {101994}=101   | Swatch: Dcache logs "Message queue overflow" reported
 cmssl6Template dcachepools | Swatch: reset success                                | RESETSUCCESS                              | Zabbix trapper | last     | 0         | {16865}=101    | Swatch: reset success
 cmssl6Template dcachepools | Swatch: reset timed-out                              | RESETTIMEOUT                              | Zabbix trapper | last     | 0         | {16888}=101    | Swatch:reset timed-out
 cmssl6Template dcachepools | Swatch: status timeout                               | STATUSTIMEOUT                             | Zabbix trapper | last     | 0         | {16856}=101    | Swatch: status timeout
 cmssl6Template dcachepools | Swatch: timeout waiting for DMA                      | DMATIMEOUT                                | Zabbix trapper | last     | 0         | {16886}=101    | Swatch: timeout waiting for DMA
 cmssl6Template dcachepools | Swatch: wait_on_bh, CPU 0                            | WAITONBH                                  | Zabbix trapper | last     | 0         | {16877}=101    | Swatch: wait_on_bh, CPU 0
 cmssl6Template dcachepools | Swatch: XFS error seen                               | XFSERROR                                  | Zabbix trapper | last     | 0         | {16880}=101    | Swatch: XFS filesystem error
 cmssl6Template dcachepools | ICMP loss                                            | icmppingloss                              | simple check   | min      | 5m        | {89989}>10     | Ping loss is too high on {HOST.NAME}
 cmssl6Template dcachepools | ICMP ping                                            | icmpping                                  | simple check   | max      | #3        | {89258}=0      | {HOST.NAME} is unavailable
 cmssl6Template dcachepools | ICMP response time                                   | icmppingsec                               | simple check   | avg      | 5m        | {89475}>0.15   | Response time is too high on {HOST.NAME}
 cmssl6template dcachesrv   | Check for read-only filesystems                      | ro_filesystems                            | Zabbix agent   | last     | 0         | {15894}#0      | problem with filesytem / on {HOSTNAME} - possibly read only
 cmssl6template dcachesrv   | Check host certificate                               | grid_cert                                 | Zabbix agent   | last     | 0         | {15870}=102    | Host certificate on {HOSTNAME} will expire within 28 days
 cmssl6template dcachesrv   | Check host certificate                               | grid_cert                                 | Zabbix agent   | last     | 0         | {15869}=101    | {HOSTNAME} host certificate has expired.
 cmssl6template dcachesrv   | check iptables stopped                               | check_iptablesStopped                     | Zabbix agent   | last     | 0         | {15856}>0      | iptables stopped on  {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | vfs.file.cksum[/etc/services]             | Zabbix agent   | diff     | 0         | {15902}>0      | /etc/services has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | vfs.file.cksum[/usr/bin/ssh]              | Zabbix agent   | diff     | 0         | {15903}>0      | /usr/bin/ssh has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | Checksum of $1                                       | vfs.file.cksum[/usr/sbin/sshd]            | Zabbix agent   | diff     | 0         | {15904}>0      | /usr/sbin/sshd has been changed on server {HOSTNAME}
 cmssl6template dcachesrv   | clock skew verify                                    | clockskew                                 | Zabbix agent   | last     | 0         | {15859}=2      | clock skewed on {HOSTNAME}
 cmssl6template dcachesrv   | dcache service verification                          | dcache_status                             | Zabbix agent   | last     | 0         | {15860}>0      | failure of dcache service
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | vfs.fs.size[/,pfree]                      | Zabbix agent   | last     | 0         | {15907}<5      | Low free disk space on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | vfs.fs.size[/,pfree]                      | Zabbix agent   | last     | 0         | {15906}<10     | Low free disk space on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free disk space on $1 in %                           | vfs.fs.size[/var/lib/pqsql-puppet ,pfree] | Zabbix agent   | last     | 0         | {15908}<10     | Low free disk space on {HOSTNAME} volume /var/lib/pqsql-puppet
 cmssl6template dcachesrv   | Free memory                                          | vm.memory.size[free]                      | Zabbix agent   | last     | 0         | {15909}<10000  | Lack of free memory on server {HOSTNAME}
 cmssl6template dcachesrv   | Free number of inodes on $1 in %                     | vfs.fs.inode[/,pfree]                     | Zabbix agent   | last     | 0         | {15905}<10     | Low number of free inodes on {HOSTNAME} volume /
 cmssl6template dcachesrv   | Free swap space                                      | system.swap.size[,free]                   | Zabbix agent   | last     | 0         | {15899}<100000 | Lack of free swap space on {HOSTNAME}
 cmssl6template dcachesrv   | Host information                                     | system.uname                              | Zabbix agent   | diff     | 0         | {15900}>0      | Host information was changed on {HOSTNAME}
 cmssl6template dcachesrv   | Host name                                            | system.hostname                           | Zabbix agent   | diff     | 0         | {15898}>0      | Hostname was changed on {HOSTNAME}
 cmssl6template dcachesrv   | Host uptime (in sec)                                 | system.uptime                             | Zabbix agent   | last     | 0         | {15901}<600    | {HOSTNAME} has just been restarted
 cmssl6template dcachesrv   | Maximum number of processes                          | kernel.maxproc                            | Zabbix agent   | last     | 0         | {15875}<256    | Configured max number of processes is too low on {HOSTNAME}
 cmssl6template dcachesrv   | Number of processes                                  | proc.num[]                                | Zabbix agent   | last     | 0         | {15888}>2000   | Too many processes on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running crond                              | proc.num[crond]                           | Zabbix agent   | last     | 0         | {15880}=0      | crond is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running gmond                              | proc.num[gmond]                           | Zabbix agent   | last     | 0         | {15881}=0      | gmond is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running ntpd                               | proc.num[ntpd]                            | Zabbix agent   | last     | 0         | {15882}=0      | ntpd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes $1                       | proc.num[zabbix_agentd]                   | Zabbix agent   | last     | 0         | {15887}<1      | Zabbix_agentd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes sshd                     | proc.num[sshd]                            | Zabbix agent   | last     | 0         | {15885}<1      | Sshd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running processes syslogd                  | proc.num[rsyslogd]                        | Zabbix agent   | last     | 0         | {15884}<1      | Syslogd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running puupet agent                       | proc.num[puppet]                          | Zabbix agent   | last     | 0         | {15883}<1      | puppet agent is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Number of running xinetd                             | proc.num[xinetd]                          | Zabbix agent   | last     | 0         | {15886}=0      | xinetd is not running on {HOSTNAME}
 cmssl6template dcachesrv   | Processor load                                       | system.cpu.load[,avg1]                    | Zabbix agent   | last     | 0         | {15897}>80     | Processor load is too high on {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {109623}=4     | puppet:  Could not parse the YAML from the summary report for {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {109614}=3     | puppet:  job failures listed in the summary report on {HOSTNAME}
 cmssl6template dcachesrv   | Puppet report - failed, too quiet or access deny     | puppetReport                              | Zabbix agent   | last     |           | {109632}=5     | puppet:   Summary report has not updated in at least 2 hours for {HOSTNAME}
 cmssl6template dcachesrv   | SSH server is running                                | net.tcp.service[ssh]                      | Zabbix agent   | last     | 0         | {15877}=0      | SSH server is down on {HOSTNAME}
 cmssl6template dcachesrv   | verify postgresql running                            | postgresql_status                         | Zabbix agent   | last     | 0         | {15879}>0      | postgresql not running on {HOSTNAME}
 cmssl6template dcachesrv   | Certificate expiration check                         | certcheck                                 | Zabbix trapper | last     | 0         | {16839}=103    | Expired certificates
 cmssl6template dcachesrv   | Certificate expiration check                         | certcheck                                 | Zabbix trapper | last     | 0         | {16838}=102    | Expired certificates
 cmssl6template dcachesrv   | Certificate expiration check                         | certcheck                                 | Zabbix trapper | last     | 0         | {16837}=101    | Expired certificates
 cmssl6template dcachesrv   | Certificate Monitoring                               | cacheck                                   | Zabbix trapper | last     | 0         | {16835}=103    | Certificate Monitoring
 cmssl6template dcachesrv   | Certificate Monitoring                               | cacheck                                   | Zabbix trapper | last     | 0         | {16836}=101    | Certificate Monitoring
 cmssl6template dcachesrv   | Certificate Monitoring                               | cacheck                                   | Zabbix trapper | last     | 0         | {16834}=102    | Certificate Monitoring
 cmssl6template dcachesrv   | Crls Monitoring                                      | crlcheck                                  | Zabbix trapper | last     | 0         | {16840}=101    | Crls Monitoring
 cmssl6template dcachesrv   | Crls Monitoring                                      | crlcheck                                  | Zabbix trapper | last     | 0         | {16841}=103    | Crls Monitoring
 cmssl6template dcachesrv   | Crls Monitoring                                      | crlcheck                                  | Zabbix trapper | last     | 0         | {16842}=102    | Crls Monitoring
 cmssl6template dcachesrv   | OCS inventory check                                  | check_ocs                                 | Zabbix trapper | last     | 0         | {15858}=101    | No OCS inventory file found.  Possibly not running/configured/installed on {HOSTNAME}
 cmssl6template dcachesrv   | OCS inventory check                                  | check_ocs                                 | Zabbix trapper | last     | 0         | {15857}=102    | The OCS inventory is out of date on {HOSTNAME}
 cmssl6template dcachesrv   | Swatch: bh:   1 [0 1]                                | BH                                        | Zabbix trapper | last     | 0         | {15855}=101    | Swatch: BH alert
 cmssl6template dcachesrv   | Swatch:  dcache log reported a "CLUMPING_ISSUE"      | CLUMPINGISSUE                             | Zabbix trapper | last     |           | {103803}=101   | Swatch: Dcache logs reported an "CLUMPING_ISSUE" 
 cmssl6template dcachesrv   | Swatch: DMA disabled                                 | DMADISABLED                               | Zabbix trapper | last     | 0         | {15861}=101    | Swatch: DMA is disabled
 cmssl6template dcachesrv   | Swatch: dma_intr: error=0x84                         | DMAINTRx84                                | Zabbix trapper | last     | 0         | {15864}=101    | Swatch: dma_intr: error=0x84
 cmssl6template dcachesrv   | Swatch: dma_intr: status=0x40                        | DMAINTRx40                                | Zabbix trapper | last     | 0         | {15862}=101    | Swatch: dma_intr: status=0x40
 cmssl6template dcachesrv   | Swatch: dma_intr: status=0x51                        | DMAINTRx51                                | Zabbix trapper | last     | 0         | {15863}=101    | Swatch: dma_intr: status=0x51
 cmssl6template dcachesrv   | Swatch: drive not ready for command                  | DRVNOTREADY                               | Zabbix trapper | last     | 0         | {15866}=101    | Swatch: drive not ready for command
 cmssl6template dcachesrv   | Swatch: end_request: I/O error                       | ENDREQUEST                                | Zabbix trapper | last     | 0         | {15867}=101    | Swatch: end_request: I/O error
 cmssl6template dcachesrv   | Swatch: ext2_write_inode: unable to read inode block | INODEBLOCK                                | Zabbix trapper | last     | 0         | {15872}=101    | Swatch: ext2_write_inode: unable to read inode block
 cmssl6template dcachesrv   | Swatch: Hardware Error                               | HARDWAREERROR                             | Zabbix trapper | last     | 0         | {15871}=101    | Swatch: Hardware Error reported
 cmssl6template dcachesrv   | Swatch: irq:  0 [0 0]                                | IRQ                                       | Zabbix trapper | last     | 0         | {15873}=101    | Swatch: irq: 0 [0 0]
 cmssl6template dcachesrv   | Swatch: IRQ Timeout                                  | IRQTIMEOUT                                | Zabbix trapper | last     | 0         | {15874}=101    | Swatch: IRQ Timeout
 cmssl6template dcachesrv   | Swatch: kernel: EXT3-fs error                        | EXT3ERROR                                 | Zabbix trapper | last     | 0         | {15868}=101    | Swatch: kernel: EXT3-fs error
 cmssl6template dcachesrv   | Swatch: kernel: I/O error                            | KERNELIOERROR                             | Zabbix trapper | last     | 0         | {15876}=101    | Swatch:kernel: I/O error
 cmssl6template dcachesrv   | Swatch: Message queue overflow                       | MQO                                       | Zabbix trapper | last     | 0         | {102764}=101   | Swatch: Dcache logs "Message queue overflow" reported
 cmssl6template dcachesrv   | Swatch: reset success                                | RESETSUCCESS                              | Zabbix trapper | last     | 0         | {15892}=101    | Swatch: reset success
 cmssl6template dcachesrv   | Swatch: reset timed-out                              | RESETTIMEOUT                              | Zabbix trapper | last     | 0         | {15893}=101    | Swatch:reset timed-out
 cmssl6template dcachesrv   | Swatch: status timeout                               | STATUSTIMEOUT                             | Zabbix trapper | last     | 0         | {15896}=101    | Swatch: status timeout
 cmssl6template dcachesrv   | Swatch: timeout waiting for DMA                      | DMATIMEOUT                                | Zabbix trapper | last     | 0         | {15865}=101    | Swatch: timeout waiting for DMA
 cmssl6template dcachesrv   | Swatch: wait_on_bh, CPU 0                            | WAITONBH                                  | Zabbix trapper | last     | 0         | {15910}=101    | Swatch: wait_on_bh, CPU 0
 cmssl6template dcachesrv   | Swatch: XFS error seen                               | XFSERROR                                  | Zabbix trapper | last     | 0         | {15911}=101    | Swatch: XFS filesystem error
 cmssl6template dcachesrv   | ICMP loss                                            | icmppingloss                              | simple check   | min      | 5m        | {90251}>10     | Ping loss is too high on {HOST.NAME}
 cmssl6template dcachesrv   | ICMP ping                                            | icmpping                                  | simple check   | max      | #3        | {89231}=0      | {HOST.NAME} is unavailable
 cmssl6template dcachesrv   | ICMP response time                                   | icmppingsec                               | simple check   | avg      | 5m        | {89240}>0.15   | Response time is too high on {HOST.NAME}
 cmsTemplate-phedex         | Number of running debug                              | Debug                                     | Zabbix agent   | last     | 0         | {86174}=0      | Debug instance not running on {HOSTNAME}
 cmsTemplate-phedex         | Number of running debug watchdog                     | Debug_watchdog                            | Zabbix agent   | last     | 0         | {86175}=0      | watchdog for Debug instance not running on {HOSTNAME}
 cmsTemplate-phedex         | Number of running prod                               | Prod                                      | Zabbix agent   | last     | 0         | {86181}=0      | watchdog for Prod instance not running on {HOSTNAME}
 cmsTemplate-phedex         | verify pnfs mounted                                  | pnfs                                      | Zabbix agent   | last     | 0         | {86180}#0      | Problem with pnfs mount on {HOSTNAME}
 cmsTemplate-phedex         | check proxy lifetime                                 | proxyExp                                  | Zabbix trapper | last     | 0         | {86183}=101    | proxy expire in 1 day
 cmsTemplate-phedex         | check proxy lifetime                                 | proxyExp                                  | Zabbix trapper | last     | 0         | {86182}=103    | proxy expire in 3 days
 cmsTemplate-phedex         | monitor phedex agents                                | phedex_agents-prod                        | Zabbix trapper | last     | 0         | {86179}=1      | probem with phedex agents on {HOSTNAME}
 cmsTemplate-phedex         | monitor phedex debug agents                          | phedex_agents-debug                       | Zabbix trapper | last     | 0         | {86178}=1      | probem with debug phedex agents on {HOSTNAME}
 cmsTemplate-phedex         | monitor srmcp-eos agents                             | srmcp-eos                                 | Zabbix trapper | last     |           | {109601}=1     | srmcp-eos test failed
 cmsTemplate-phedex         | monitor xrootd disk                                  | xrdcp-disk                                | Zabbix trapper | last     |           | {112581}#0     | Problem with xrootd: not able to write to disk
 cmsTemplate-phedex         | monitor xrootd eos                                   | xrdcp-eos                                 | Zabbix trapper | last     |           | {112589}#0     | Problem with xrootd: not able to write to eos
 cmsTemplate-phedex         | monitor xrootd tape                                  | xrdcp-tape                                | Zabbix trapper | last     |           | {112585}#0     | Problem with xrootd: not able to write to tape
 cmsTemplate-phedex         | PhEDEx agents                                        | phedex_agents                             | Zabbix trapper | last     | 0         | {86177}=0      | Prod instance not running on {HOSTNAME}
 cmsTemplate-phedex         | verify dcache mounted                                | dcache                                    | Zabbix trapper | last     | 0         | {86173}#0      | Problem with dcache mount on {HOSTNAME}
 cmsTemplate-phedex         | verify dcache mounted - test                         | mount-dcache                              | Zabbix trapper | last     | 0         | {86176}#0      | Problem with dcache mount on {HOSTNAME} -"TEST" 
(147 rows)

[root@cmszabbix1 HUANGCH]# 

#10 Updated by Chih-Hao Huang over 4 years ago

  • Status changed from Accepted to Resolved
  • % Done changed from 90 to 100

Most of the trigger are covered already.
We know enough to start.
The trigger:check does not have to be 1:1
Will try swatch.



Also available in: Atom PDF