Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.3.0
    • Component/s: Data Processors
    • Labels:
      None
    • Release Note:
      Simple lightweight archiver tool.

      Description

      The current demux-archive plumbing is quite complicated. At Berkeley, we need something much simpler.

        Issue Links

          Activity

          Ari Rabkin made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Ari Rabkin added a comment -

          Taking silence for consent, I just committed this.

          Show
          Ari Rabkin added a comment - Taking silence for consent, I just committed this.
          Ari Rabkin made changes -
          Attachment sinkArchiver.patch [ 12413269 ]
          Hide
          Ari Rabkin added a comment -

          Revised, fixes a few unit test problems.

          Show
          Ari Rabkin added a comment - Revised, fixes a few unit test problems.
          Ari Rabkin made changes -
          Attachment sinkArchiver.patch [ 12413262 ]
          Ari Rabkin made changes -
          Link This issue incorporates CHUKWA-338 [ CHUKWA-338 ]
          Hide
          Ari Rabkin added a comment -

          No. The archiver, by default in this patch, will group by cluster, day and datatype. Which is well suited to our use case, which is mapreduce analytics of logs.

          Show
          Ari Rabkin added a comment - No. The archiver, by default in this patch, will group by cluster, day and datatype. Which is well suited to our use case, which is mapreduce analytics of logs.
          Hide
          Jiaqi Tan added a comment -

          If there's no Demux, then the purpose of Chukwa will be just to collect logs, and store them in a single jumbled mix of all the log record types?

          Show
          Jiaqi Tan added a comment - If there's no Demux, then the purpose of Chukwa will be just to collect logs, and store them in a single jumbled mix of all the log record types?
          Hide
          Ari Rabkin added a comment -

          A future enhancement, once we have appends, is to actually merge files during promotion, and not just rename to avoid collision.

          Show
          Ari Rabkin added a comment - A future enhancement, once we have appends, is to actually merge files during promotion, and not just rename to avoid collision.
          Ari Rabkin made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ari Rabkin made changes -
          Field Original Value New Value
          Attachment sinkArchiver.patch [ 12413262 ]
          Hide
          Ari Rabkin added a comment -

          Simple sink archiver.

          Copies all the .done files out of the sink, runs an archiver MapReduce job, then merges output of that job into archive, renaming files to avoid collision.

          Intended use is to run once every day or two, to empty out sink.

          Show
          Ari Rabkin added a comment - Simple sink archiver. Copies all the .done files out of the sink, runs an archiver MapReduce job, then merges output of that job into archive, renaming files to avoid collision. Intended use is to run once every day or two, to empty out sink.
          Ari Rabkin created issue -

            People

            • Assignee:
              Ari Rabkin
              Reporter:
              Ari Rabkin
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development