Chukwa
  1. Chukwa
  2. CHUKWA-317

cleaner support for archiving chunks

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.5.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Right now, the demux and archiving processing phases are tied together. It should be possible to do archiving without running demux, by setting a single option.

        Issue Links

          Activity

          Hide
          Ari Rabkin added a comment -

          I think there are two possible approaches.
          have a shell option, set in chukwa-env.sh, that processSinkFiles looks at to decide what to run.
          Have a conf option in chukwa-demux-conf that the demux jobs look at, and if it's set, they exit immediately.

          First option is cleaner, second option keeps all the conf on the java side.

          Show
          Ari Rabkin added a comment - I think there are two possible approaches. have a shell option, set in chukwa-env.sh, that processSinkFiles looks at to decide what to run. Have a conf option in chukwa-demux-conf that the demux jobs look at, and if it's set, they exit immediately. First option is cleaner, second option keeps all the conf on the java side.
          Hide
          Jerome Boulon added a comment -

          You should be able to run archive without Demux since the Archive is using the same files as input??
          If you don't want demux just don't start it and start only the archiveManager but you should set the configuration to look for the collector dataSink file instead of the demuxManager dataSinkFile.

          Show
          Jerome Boulon added a comment - You should be able to run archive without Demux since the Archive is using the same files as input?? If you don't want demux just don't start it and start only the archiveManager but you should set the configuration to look for the collector dataSink file instead of the demuxManager dataSinkFile.
          Hide
          Jerome Boulon added a comment -

          BTW, processSinkFiles.sh is no longer used ... at least at Yahoo!.

          Show
          Jerome Boulon added a comment - BTW, processSinkFiles.sh is no longer used ... at least at Yahoo!.
          Hide
          Ari Rabkin added a comment -

          After talking to Jerome, I see that this isn't so bad currently. Still could be better, but not critical.

          Show
          Ari Rabkin added a comment - After talking to Jerome, I see that this isn't so bad currently. Still could be better, but not critical.
          Hide
          Ari Rabkin added a comment -

          Things that I'd like to fix in archive:

          1) output dir should be configurable.
          2) ArchiveManager only works if it's downstream from demux, since it has very strong assumptions about structure of input dir. (grouped by day, with specified names).
          3) Only runs once per day.

          Show
          Ari Rabkin added a comment - Things that I'd like to fix in archive: 1) output dir should be configurable. 2) ArchiveManager only works if it's downstream from demux, since it has very strong assumptions about structure of input dir. (grouped by day, with specified names). 3) Only runs once per day.
          Hide
          Ari Rabkin added a comment -

          The first two of these are resolved, for my purposes, by CHUKWA-346. But configuring the final archive path is still not possible. This should be a fairly simple fix, but it's currently low priority.

          Show
          Ari Rabkin added a comment - The first two of these are resolved, for my purposes, by CHUKWA-346 . But configuring the final archive path is still not possible. This should be a fairly simple fix, but it's currently low priority.
          Hide
          Ari Rabkin added a comment -

          All that's left of this issue is improving conf options, so priority is low.

          Show
          Ari Rabkin added a comment - All that's left of this issue is improving conf options, so priority is low.
          Hide
          Eric Yang added a comment -

          Demux and archiving are two independent process in Chukwa 0.5.0. This has been fixed.

          Show
          Eric Yang added a comment - Demux and archiving are two independent process in Chukwa 0.5.0. This has been fixed.

            People

            • Assignee:
              Ari Rabkin
              Reporter:
              Ari Rabkin
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development