Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.1.2, 0.2.0, 0.3.0
    • Fix Version/s: 0.2.0, 0.3.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Data that arrived later than a day (day root directory: .D.evt files) should not be flagged as part of any hour

      1. CHUKWA-325.patch
        0.9 kB
        Jerome Boulon

        Activity

        Hide
        jboulon Jerome Boulon added a comment -

        DailyRolling takes file from all hours directories and merges them.
        xxx.D.evt files is not going to any hour directory, therefore it should not be flagged has part of an hour directory.

        Show
        jboulon Jerome Boulon added a comment - DailyRolling takes file from all hours directories and merges them. xxx.D.evt files is not going to any hour directory, therefore it should not be flagged has part of an hour directory.
        Hide
        jboulon Jerome Boulon added a comment -

        You may have to manually cleaned the daily rolling directory: /chukwa/rolling/daily.
        This happen only for files that are later than 1 day

        Show
        jboulon Jerome Boulon added a comment - You may have to manually cleaned the daily rolling directory: /chukwa/rolling/daily. This happen only for files that are later than 1 day
        Hide
        asrabkin Ari Rabkin added a comment -

        Requiring manual intervention is bad. Is there a clean way around it?

        Show
        asrabkin Ari Rabkin added a comment - Requiring manual intervention is bad. Is there a clean way around it?
        Hide
        jboulon Jerome Boulon added a comment -

        I'm aware of that but the only way to get into this situation is by running backfilling tools or re-running Demux on older data so this should not be too bad.
        Also, There's this Jira: CHUKWA-87 that is related to the rolling problem and I was planning in putting some time on this one (new feature) instead of building a temporary shell script.

        CHUKWA-87 is in my mind a better way of dealing with spill files and will be generic so we will be able to use to both Archive and ChukwaRecord files.

        Show
        jboulon Jerome Boulon added a comment - I'm aware of that but the only way to get into this situation is by running backfilling tools or re-running Demux on older data so this should not be too bad. Also, There's this Jira: CHUKWA-87 that is related to the rolling problem and I was planning in putting some time on this one (new feature) instead of building a temporary shell script. CHUKWA-87 is in my mind a better way of dealing with spill files and will be generic so we will be able to use to both Archive and ChukwaRecord files.
        Hide
        eyang Eric Yang added a comment -

        +1 I am ok with committing this for 0.2, and enhance the generic rolling algorithm in CHUKWA-87.

        Show
        eyang Eric Yang added a comment - +1 I am ok with committing this for 0.2, and enhance the generic rolling algorithm in CHUKWA-87 .
        Hide
        eyang Eric Yang added a comment -

        I just committed this. Thanks Jerome.

        Show
        eyang Eric Yang added a comment - I just committed this. Thanks Jerome.

          People

          • Assignee:
            jboulon Jerome Boulon
            Reporter:
            jboulon Jerome Boulon
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development