Chukwa
  1. Chukwa
  2. CHUKWA-632

Logs getting lost in hbase using TsProcessor

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When collector writes to hbase using TsProcessor, it is using timestamp as the key. If there are multiple log statements with the same timestamp, which is the key, it looses them in hbase.

      As a fix, I extended AbstractProcessor and built my key using timestamp and the AbstractProcessor.startOffset. But, for that, it has to be protected or a getter on it in AbstractProcessor class to be able to use it.

      Also, the way OutputCollector.collect method works, it might need to redo the way rowKey is created.

      1. patch.txt
        0.9 kB
        Abhijit Dhar

        Activity

        Abhijit Dhar created issue -
        Abhijit Dhar made changes -
        Field Original Value New Value
        Description When collector writes to hbase using TsProcessor, it is using timestamp as the key. If there are multiple log statements with the same timestamp, which is the key, it looses them in hbase.

        As a fix, I extended AbstractProcessor and built my key using timestamp and the AbstractProcessor.startOffset. But, for that, it has to be protected or a getter on it in AbstractProcessor class to be able to use it
        When collector writes to hbase using TsProcessor, it is using timestamp as the key. If there are multiple log statements with the same timestamp, which is the key, it looses them in hbase.

        As a fix, I extended AbstractProcessor and built my key using timestamp and the AbstractProcessor.startOffset. But, for that, it has to be protected or a getter on it in AbstractProcessor class to be able to use it.

        Also, the way OutputCollector.collect method works, it might need to redo the way rowKey is created.
        Hide
        Eric Yang added a comment -

        +1 for adding the getter method for startOffset.

        Show
        Eric Yang added a comment - +1 for adding the getter method for startOffset.
        Abhijit Dhar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.5.0 [ 12315030 ]
        Abhijit Dhar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Abhijit Dhar added a comment -

        Patch attached

        Show
        Abhijit Dhar added a comment - Patch attached
        Abhijit Dhar made changes -
        Attachment patch.txt [ 12522876 ]
        Abhijit Dhar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Eric Yang added a comment -

        The change looks good for AbstractProcessor. Do you mind to add buildGenericRecord method override in TsProcessor for generating proper rowKey? Thanks

        Show
        Eric Yang added a comment - The change looks good for AbstractProcessor. Do you mind to add buildGenericRecord method override in TsProcessor for generating proper rowKey? Thanks
        Eric Yang made changes -
        Assignee Abhijit Dhar [ abhijitdhar ]
        Hide
        Eric Yang added a comment -

        I just committed this, thanks Abhijit.

        Show
        Eric Yang added a comment - I just committed this, thanks Abhijit.
        Eric Yang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #460 (See https://builds.apache.org/job/Chukwa-trunk/460/)
        CHUKWA-632. Added getter method for startOffset in AbstractProcessor. (Abhijit Dhar via Eric Yang) (Revision 1411810)

        Result = SUCCESS
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411810
        Files :

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #460 (See https://builds.apache.org/job/Chukwa-trunk/460/ ) CHUKWA-632 . Added getter method for startOffset in AbstractProcessor. (Abhijit Dhar via Eric Yang) (Revision 1411810) Result = SUCCESS eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411810 Files : /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java

          People

          • Assignee:
            Abhijit Dhar
            Reporter:
            Abhijit Dhar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development