Chukwa
  1. Chukwa
  2. CHUKWA-317

cleaner support for archiving chunks

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.5.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.

        Issue Links

          Activity

          Ari Rabkin created issue -
          Ari Rabkin made changes -
          Field Original Value New Value
          Link This issue is related to CHUKWA-311 [ CHUKWA-311 ]
          Ari Rabkin made changes -
          Link This issue blocks CHUKWA-25 [ CHUKWA-25 ]
          Hide
          Ari Rabkin added a comment -

          I think there are two possible approaches.
          have a shell option, set in chukwa-env.sh, that processSinkFiles looks at to decide what to run.
          Have a conf option in chukwa-demux-conf that the demux jobs look at, and if it's set, they exit immediately.

          First option is cleaner, second option keeps all the conf on the java side.

          Show
          Ari Rabkin added a comment - I think there are two possible approaches. have a shell option, set in chukwa-env.sh, that processSinkFiles looks at to decide what to run. Have a conf option in chukwa-demux-conf that the demux jobs look at, and if it's set, they exit immediately. First option is cleaner, second option keeps all the conf on the java side.
          Hide
          Jerome Boulon added a comment -

          You should be able to run archive without Demux since the Archive is using the same files as input??
          If you don't want demux just don't start it and start only the archiveManager but you should set the configuration to look for the collector dataSink file instead of the demuxManager dataSinkFile.

          Show
          Jerome Boulon added a comment - You should be able to run archive without Demux since the Archive is using the same files as input?? If you don't want demux just don't start it and start only the archiveManager but you should set the configuration to look for the collector dataSink file instead of the demuxManager dataSinkFile.
          Hide
          Jerome Boulon added a comment -

          BTW, processSinkFiles.sh is no longer used ... at least at Yahoo!.

          Show
          Jerome Boulon added a comment - BTW, processSinkFiles.sh is no longer used ... at least at Yahoo!.
          Hide
          Ari Rabkin added a comment -

          After talking to Jerome, I see that this isn't so bad currently. Still could be better, but not critical.

          Show
          Ari Rabkin added a comment - After talking to Jerome, I see that this isn't so bad currently. Still could be better, but not critical.
          Ari Rabkin made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          Ari Rabkin made changes -
          Link This issue blocks CHUKWA-25 [ CHUKWA-25 ]
          Ari Rabkin made changes -
          Link This issue incorporates CHUKWA-25 [ CHUKWA-25 ]
          Hide
          Ari Rabkin added a comment -

          Things that I'd like to fix in archive:

          1) output dir should be configurable.
          2) ArchiveManager only works if it's downstream from demux, since it has very strong assumptions about structure of input dir. (grouped by day, with specified names).
          3) Only runs once per day.

          Show
          Ari Rabkin added a comment - Things that I'd like to fix in archive: 1) output dir should be configurable. 2) ArchiveManager only works if it's downstream from demux, since it has very strong assumptions about structure of input dir. (grouped by day, with specified names). 3) Only runs once per day.
          Hide
          Ari Rabkin added a comment -

          The first two of these are resolved, for my purposes, by CHUKWA-346. But configuring the final archive path is still not possible. This should be a fairly simple fix, but it's currently low priority.

          Show
          Ari Rabkin added a comment - The first two of these are resolved, for my purposes, by CHUKWA-346 . But configuring the final archive path is still not possible. This should be a fairly simple fix, but it's currently low priority.
          Hide
          Ari Rabkin added a comment -

          All that's left of this issue is improving conf options, so priority is low.

          Show
          Ari Rabkin added a comment - All that's left of this issue is improving conf options, so priority is low.
          Ari Rabkin made changes -
          Priority Major [ 3 ] Minor [ 4 ]
          Ari Rabkin made changes -
          Fix Version/s 0.4.0 [ 12314359 ]
          Fix Version/s 0.3.0 [ 12314032 ]
          Ari Rabkin made changes -
          Fix Version/s 0.5.0 [ 12315030 ]
          Fix Version/s 0.4.0 [ 12314359 ]
          Hide
          Eric Yang added a comment -

          Demux and archiving are two independent process in Chukwa 0.5.0. This has been fixed.

          Show
          Eric Yang added a comment - Demux and archiving are two independent process in Chukwa 0.5.0. This has been fixed.
          Eric Yang made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Ari Rabkin
              Reporter:
              Ari Rabkin
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development