Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Since Meta has no timestamp and jobid, currently all metas have the same key /0/null/0 and they are put into the same dir.

        Activity

        Hide
        zhangyongjiang Cheng added a comment -

        Submitted patch. Changes:

        1. If line starts with "Meta", empty the savedLine for multiple lines exception handling

        2. If log type is "Meta", discard it

        3. Set time stamp to current if it's 0

        Show
        zhangyongjiang Cheng added a comment - Submitted patch. Changes: 1. If line starts with "Meta", empty the savedLine for multiple lines exception handling 2. If log type is "Meta", discard it 3. Set time stamp to current if it's 0
        Hide
        zhangyongjiang Cheng added a comment -

        We can get jobid from file name which is in the meta data of the chunk,

        Show
        zhangyongjiang Cheng added a comment - We can get jobid from file name which is in the meta data of the chunk,
        Hide
        zhangyongjiang Cheng added a comment -

        parse jobid from the stream name of chunk

        Show
        zhangyongjiang Cheng added a comment - parse jobid from the stream name of chunk
        Hide
        jboulon Jerome Boulon added a comment -

        Cheng,
        When the date is not available could you use the timePartition set at the Chunk level.
        You can get it using: archiveKey.getTimePartition();

        Could you also compile your pattern once and reused it.

        Show
        jboulon Jerome Boulon added a comment - Cheng, When the date is not available could you use the timePartition set at the Chunk level. You can get it using: archiveKey.getTimePartition(); Could you also compile your pattern once and reused it.
        Hide
        terencekwan Terence Kwan added a comment -

        from the syntax point of view, it is looking good. +1

        Show
        terencekwan Terence Kwan added a comment - from the syntax point of view, it is looking good. +1
        Hide
        jboulon Jerome Boulon added a comment -

        -1 see my comments

        Show
        jboulon Jerome Boulon added a comment - -1 see my comments
        Hide
        zhangyongjiang Cheng added a comment -

        new patch.

        • if no time stamp, use the time partition in the archiveKey
        • use pre-compiled patten
        Show
        zhangyongjiang Cheng added a comment - new patch. if no time stamp, use the time partition in the archiveKey use pre-compiled patten
        Hide
        zhangyongjiang Cheng added a comment -

        ignore unrecognized tag.

        Show
        zhangyongjiang Cheng added a comment - ignore unrecognized tag.
        Hide
        eyang Eric Yang added a comment -

        +1 on discard unrecognized tag, and use time partition in the archive key.

        Show
        eyang Eric Yang added a comment - +1 on discard unrecognized tag, and use time partition in the archive key.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Cheng.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Cheng.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #12 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/12/ )
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #45 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/45/ )

          People

          • Assignee:
            Unassigned
            Reporter:
            zhangyongjiang Cheng
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development