Chukwa
  1. Chukwa
  2. CHUKWA-530

Syslog Adaptor for mapping Syslog facility name to Chukwa data type

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Java 6, Mac OS X

    • Hadoop Flags:
      Reviewed

      Description

      UDPAdaptor can record syslog logs but the log entry is tagged with Chukwa internal data type at adaptor start up time. I am thinking to extend UDPAdaptor to record syslog messages with small enhancements. SyslogAdaptor can extract facility name field from syslog message to tag facility name as the data type. This would provide the ability to invoke different demux parsers base on the facility name in syslog. HOSTNAME can also be mapped to source for Chukwa chunk meta data.

      My use case is to configure hadoop log4j.properties to log everything through org.apache.log4j.net.SyslogAppender to one SysLogAdaptor on the local Chukwa agent, and invokes different parsers for audit log and name node log. Facility name is some what limited, but it should be enough type left to manage hadoop log files.

      1. CHUKWA-530.patch
        7 kB
        Eric Yang
      2. CHUKWA-530-draft.patch
        5 kB
        Eric Yang

        Activity

        Hide
        Eric Yang added a comment -

        Working implementation.

        Show
        Eric Yang added a comment - Working implementation.
        Hide
        Eric Yang added a comment -

        I am thinking to add more configure parameters to Chukwa Agent config file. i.e.:

        Trigger for adding syslog adaptor

        add SyslogAdaptor SYSLOG 9095 0
        

        SyslogAdaptor would look up:

        <property>
          <name>syslog.adaptor.[port].[facility name]</name>
          <value>[chukwa data type]</value>
        </property>
        

        Example:

        <property>
          <name>syslog.adaptor.9095.LOCAL0</name>
          <value>HADOOP</value>
        </property>
        
        <property>
          <name>syslog.adaptor.9095.LOCAL1</name>
          <value>HadoopMetrics</value>
        </property>
        
        <property>
          <name>syslog.adaptor.9095.AUDIT</name>
          <value>AUDITLOG</value>
        </property>
        

        This means each udp port can host up to 24 data types. Make sense?

        Show
        Eric Yang added a comment - I am thinking to add more configure parameters to Chukwa Agent config file. i.e.: Trigger for adding syslog adaptor add SyslogAdaptor SYSLOG 9095 0 SyslogAdaptor would look up: <property> <name>syslog.adaptor.[port].[facility name]</name> <value>[chukwa data type]</value> </property> Example: <property> <name>syslog.adaptor.9095.LOCAL0</name> <value>HADOOP</value> </property> <property> <name>syslog.adaptor.9095.LOCAL1</name> <value>HadoopMetrics</value> </property> <property> <name>syslog.adaptor.9095.AUDIT</name> <value>AUDITLOG</value> </property> This means each udp port can host up to 24 data types. Make sense?
        Hide
        Eric Yang added a comment -

        Revised patch to include javadoc and test case. New feature includes mapping of facility name to Chukwa data type.

        Usage in Chukwa Agent configuration file:

        <property>
           <name>syslog.adaptor.port.9095.facility.LOCAL1</name>
           <value>HADOOP</value>
        </property>
        
        Show
        Eric Yang added a comment - Revised patch to include javadoc and test case. New feature includes mapping of facility name to Chukwa data type. Usage in Chukwa Agent configuration file: <property> <name>syslog.adaptor.port.9095.facility.LOCAL1</name> <value>HADOOP</value> </property>
        Hide
        Ari Rabkin added a comment -

        +1 looks good.

        Show
        Ari Rabkin added a comment - +1 looks good.
        Hide
        Eric Yang added a comment -

        Thanks Ari, I just committed this.

        Show
        Eric Yang added a comment - Thanks Ari, I just committed this.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development