Project

General

Profile

Syslog destination

In addition to the standard mf destinations, a syslog destination has been added. This option allows a user, logged onto a given node, to send messages to the syslog daemon of that node.

1 FHiCL configuration

To enable the syslog logging option via FHiCL configuration, the following destination can be added in the user's FHiCL file under the message FHiCL parameter:

message : {
  destinations : {

    # normal log destinations
    log1 : { type : cout   }   # OK - writes to stdout
    log2 : { type : cerr   }   # OK - writes to stderr
    log3 : { type : file   }   # OK - writes to file
    log4 : { type : syslog }   # OK - writes to syslog

  }
}

N.B. Cataloging statistics using syslog, however, is not supported. The following code, for example, will result in a run-time error:

message : {
  destinations : {

    # normal log destinations
    log1 : { type : cout   }   # OK
    log2 : { type : cerr   }   # OK
    log3 : { type : cerr   }   # OK
    log4 : { type : syslog }   # OK

    # include statistics destination
    statistics : {  
      dest1 : { type : cout   }  # OK
      dest2 : { type : cerr   }  # OK
      dest3 : { type : file   }  # OK
      dest4 : { type : syslog }  # ERROR!  messagefacility will throw
    }

  }
} 

2 messagefacility interface with syslog

The messagefacility interacts with syslog via the C library syslog.h. A connection to the syslog facility is opened in the constructor of an ELsyslog object:

openlog("MF",0,LOG_LOCAL0);

For now, only connections to the syslog user-defined destination local0 are allowed.

The semantics of syslog message logging are shown ELsyslog::log in messagefacility/MessageService/ELsyslog.cc, which looks like:

syslog( severity, message-string );

where severity and message-string are described in the next sections.

2.1 syslog severities

Eight severity levels are defined for syslog. The below table shows these levels, their integer representations according to syslog, and the mapping of the mf facilities (and corresponding mf severities) to the syslog severities.

syslog Severity Integer rep. mf facility mf severity
LOG_EMERG 0 - -
LOG_ALERT 1 - -
LOG_CRIT 2 LogAbsolute, LogSystem ELsev_severe
LOG_ERR 3 LogError, LogImportant, LogProblem ELsev_error
LOG_WARNING 4 LogPrint, LogWarning ELsev_warning
LOG_NOTICE 5 - -
LOG_INFO 6 LogInfo, LogVerbatim ELsev_info
LOG_DEBUG 7 LogDebug, LogTrace ELsev_success

2.2 Message output format

For a syslog destination, the message format is as follows (new lines have been inserted for clarity):

timestamp|
host_name|
host_address|
mf_severity|
category|
optional_application_name|
optional_process_ID|
optional_current_run_event_number|
software_module_name|
user_supplied_text_message

So for the following example that might be found in a user's code:

 mf::LogError("err1) << "This is an ERROR message.";

the message sent to the syslog daemon looks like:

 02-Oct-2014 14:05:37 CDT|cluck.fnal.gov|131.225.65.46|Error|err1|MessageFacility|29305|MF-online|MFTest|This is an ERROR message.

Note that if a run-event number is not specified, the default message is "MF-online".

N.B. It is permitted to use "|" symbols in the user-supplied message, even though they serve as delimiters in the message sent to syslog.

3 syslog configuration

The configuration of running the syslog daemon is specified in /etc/syslog.conf, /etc/rsyslog.conf, or /etc/syslog-ng.conf for UNIX-based systems. We specify guidance for syslogd and rsyslogd here.

3.1 syslogd configuration

For users whose systems run syslogd, the 'MF' messages can be logged to a file by inserting the following rule in the /etc/syslog.conf file:

local0.*    [TAB]    /var/log/messagesLocal0  

where a tab space must separate the local0.* string from /var/log/messagesLocal0. This configuration logs all messages coming from the syslog-facility local0 to the file /var/log/messagesLocal0. More specific rules can be used to send messages of a given severity to a given log file:

local0.info   [TAB]    /var/log/messagesLocal0_info  
local0.err    [TAB]    /var/log/messagesLocal0_err 
[ etc. ] 

Note that root privileges are required in order to modify the /etc/syslog.conf settings and for syslogd to write to the /var/log/ directory.

3.2 rsyslogd configuration

For systems with rsyslogd, an extra configuration file and routing script have been provided to insert messagefacility messages into an sqlite3 database. Both of these can be found in messagefacility package here:

messagefacility/tools/syslog/010-tosqlite3-router.conf
messagefacility/tools/syslog/tosqlite3-router.rb

To use the routing capability, enter the following on the command line (root privileges required):

cp messagefacility/tools/syslog/010-tosqlite3-router.conf /etc/rsyslog.d/
cp messagefacility/tools/syslog/tosqlite3-router.rb /var/log/

To enable the new rsyslog configuration, restart the rsyslog daemon:

service rsyslog restart

Only messages logged with the identifier 'MF' are sent to the routing script. This distinction is not enacted for the standard syslogd option described above. Details of the routing script are presented in Sec. 4.

3.3 Log rotation

It is conceivable that the sizes of the message log files and database files could grow rather quickly if a large number messages are expected. In that case, the user should consider using log rotation, such that the large number of messages can be handled in a manageable way.

4 syslog-to-sqlite3 routing script

4.1 Required software

The routing script is written in the Ruby language and has been tested using version 1.8.7 on an x86_64-linux architecture. The script depends on rubygems which can be installed on Linux systems using yum:

yum install rubygems

the rpms required for the script are ruby-sqlite3 and rubygem-sqlite3-ruby, which can be installed via:

yum --enablerepo=epel install ruby-sqlite3
yum --enablerepo=epel install rubygem-sqlite3-ruby

To verify that the installation was successful, you should see the following when executing interactive Ruby (type irb):

[user@host]$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'sqlite3'
=> true

4.2 Output directory

The script is designed to create two kinds of output located here:

/var/log/local0/messageFacilityLog.db     # the sqlite3 database file
/var/log/local0/err.log                   # err. printout for sqlite3 exceptions

The permissions of the files are adjusted in the script so that users can read them but not write to them. Only the account owner that runs the rsyslog daemon is allowed to write to the files (typically root or syslog).

4.3 sqlite3 database tables

All messages are written to at least two sqlite3 tables. The first table is called MessageHeaders and contains information that looks like:

Timestamp                 Hostname        Hostaddress    Severity    Category    AppName          ProcessId   RunEventNo  ModuleName
------------------------  --------------  -------------  ----------  ----------  ---------------  ----------  ----------  ----------
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Error       err1        MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Error       err2        MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Warning     warning     MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Error       catError    MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Warning     catWarning  MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Info        catInfo     MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Error       catError    MessageFacility  29305       MF-online   MFTest    
02-Oct-2014 14:05:37 CDT  cluck.fnal.gov  131.225.65.46  Warning     catWarning  MessageFacility  29305       MF-online   MFTest    

the user-supplied message itself is provided in a separate table called UserMessages:

HeaderId    Message                  
----------  -------------------------
1           This is an ERROR message.
2           This is an ERROR message.
3           Followed by a WARNING mes
4           Error information.       
5           Warning information.     
6           Info information.        
7           Error information.       
8           Warning information.     

where the HeaderId column provides the mapping to the rowid (not shown) in the MessageHeaders table.