Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      I did a telnet list command, it shows

      telnet localhost 9093
      Trying 127.0.0.1...
      Connected to localhost (127.0.0.1).
      Escape character is '^]'.
      list
      1) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Iostat 0 /grid/0/chukwa/var/log/Iostat.log 145600
      2) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Top 0 /grid/0/chukwa/var/log/Top.log 1742264
      3) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 9339 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      4) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 14777 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
      5) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 12449 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      6) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 12061 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      7) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Sar 0 /grid/0/chukwa/var/log/Sar.log 271400
      8) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 12817 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      9) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 19221 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
      10) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Ps 0 /grid/0/chukwa/var/log/Ps.log 3332933
      11) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 13185 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      12) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Netstat 0 /grid/0/chukwa/var/log/Netstat.log 15537
      13) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 11788 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      14) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 4767 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      15) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped Df 0 /grid/0/chukwa/var/log/Df.log 54540
      17) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 9873 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      16) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 18887 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
      19) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 0 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      18) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 12334 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      21) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 6077 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
      20) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 18553 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005
      23) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chunkQueue 20207 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chunkQueue.log 36825
      22) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 21281 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209
      25) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped CC 0 /grid/0/chukwa/var/log/collector.log 1040900
      24) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaHttpSender 31669 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaHttpSender.log 58005

      chukwAgent is listed 6 times.

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #8 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/8/ )
        Hide
        Eric Yang added a comment -

        I just committed this, thanks Cheng.

        Show
        Eric Yang added a comment - I just committed this, thanks Cheng.
        Hide
        Jerome Boulon added a comment -

        indexof is better than equal which will always fail.
        Do we still have duplicate issue with Checkpoint file?

        Show
        Jerome Boulon added a comment - indexof is better than equal which will always fail. Do we still have duplicate issue with Checkpoint file?
        Hide
        Eric Yang added a comment -

        getStreamName does not include offset. Therefore this patch would work. +1 Looks good.

        Show
        Eric Yang added a comment - getStreamName does not include offset. Therefore this patch would work. +1 Looks good.
        Hide
        Eric Yang added a comment -

        Let's assume the checkpoint file writes:

        add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 21281 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209

        And next request adds:

        add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 0 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 0

        This would not be considered as duplicated streams from the patch above. We may have to do duplicate detection at adaptor level because the parameters passed to the adaptor are non-uniform. However, be careful of the implementation. We don't want too much cost in doing the adaptor duplication detection. Add Ari, Jerome to charm in.

        Show
        Eric Yang added a comment - Let's assume the checkpoint file writes: add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 21281 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 38209 And next request adds: add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped chukwaAgent 0 /grid/0/chukwa/var/log/metrics/chukwa-gmon-chukwaAgent.log 0 This would not be considered as duplicated streams from the patch above. We may have to do duplicate detection at adaptor level because the parameters passed to the adaptor are non-uniform. However, be careful of the implementation. We don't want too much cost in doing the adaptor duplication detection. Add Ari, Jerome to charm in.
        Hide
        Cheng added a comment -

        Parameters also has additional information besides stream name.

        Show
        Cheng added a comment - Parameters also has additional information besides stream name.

          People

          • Assignee:
            Cheng
            Reporter:
            Cheng
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development