Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      $CHUKWA_CONF_DIR/hadoop-log4j.properties file contains a list of appender to stream Hadoop logs to Chukwa. This was designed for Hadoop 0.20.x. For supporting Hadoop 2.x, we need to update the log4j.properties example, and hadoop-metrics2.properties files as examples of how to stream logs and metrics to Chukwa.

      On Chukwa side, the initial_adaptors file also need to register SocketAdaptors to stream the data to Chukwa.

      The define ports to stream data are:

      Port Description
      9095 Hadoop Metrics
      9096 Hadoop NN Log
      9097 Hadoop SNN Log
      9098 Hadoop DN Log
      9099 Hadoop RM Log
      9100 Hadoop NM Log
      9101 Hadoop HS Log
      9102 Hadoop RM Summary
      9103 Hadoop TLA

        Activity

        Hide
        eyang Eric Yang added a comment -

        In Hadoop-env.sh, user also needs to define:

        export HADOOP_NAMENODE_OPTS="-Dhadoop.log.port=9096 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
        export HADOOP_DATANODE_OPTS="-Dhadoop.log.port=9098 -Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
        
        export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.log.port=9097 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
        

        In yarn-env.sh:

        export YARN_RESOURCEMANAGER_OPTS="-Dhadoop.log.port=9099"
        export YARN_NODEMANAGER_OPTS="-Dhadoop.log.port=9100"
        

        The ports need to align with port number in initial_adaptors for this to work correctly.

        Show
        eyang Eric Yang added a comment - In Hadoop-env.sh, user also needs to define: export HADOOP_NAMENODE_OPTS= "-Dhadoop.log.port=9096 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS= "-Dhadoop.log.port=9098 -Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS= "-Dhadoop.log.port=9097 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" In yarn-env.sh: export YARN_RESOURCEMANAGER_OPTS= "-Dhadoop.log.port=9099" export YARN_NODEMANAGER_OPTS= "-Dhadoop.log.port=9100" The ports need to align with port number in initial_adaptors for this to work correctly.
        Hide
        eyang Eric Yang added a comment -

        I just committed this.

        Show
        eyang Eric Yang added a comment - I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development