Uploaded image for project: 'Chukwa (retired)'
  1. Chukwa (retired)
  2. CHUKWA-338

duplicate suppression in archiver

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • None
    • None
    • MR Data Processors
    • None

    Description

      Right now, Archiver uses an identity reducer.

      It should be straightforward to write a custom reducer that does duplicate detection and suppression if we get multiple chunks with the same key.

      Attachments

        1. archiveDupSuppress.patch
          5 kB
          Ariel Shemaiah Rabkin

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. CHUKWA-339 Archiver needs to be ported to new MapReduce API

          • Major - Major loss of function.
          • Open
          is part of

          New Feature - A new feature of the product, which has yet to be developed. CHUKWA-346 simplified sink archiver

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              asrabkin Ariel Shemaiah Rabkin
              asrabkin Ariel Shemaiah Rabkin
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: