Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-330

Use of multiple output counters can be disabled in configuration.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 0.8.3
    • Component/s: Core, IO
    • Labels:
      None

      Description

      We're having some trouble with the amount of counters that Crunch creates
      when writing to a lot of different output files (slightly more than 120).
      This wouldn't be an issue if we were able to configure the maximum number
      of allowed counters but unfortunately, because we are running an older
      version of Hadoop, doing this is not an option and we are required to patch
      Crunch locally when using a new release to leave out the counters.

      I'm not saying the counters should be removed but maybe it is an option to
      make them configurable without paying too much of a performance penalty?

      I will implement this functionality and submit a patch.

        Attachments

        1. CRUNCH-330b.patch
          5 kB
          Josh Wills
        2. 0001-working-version.patch
          16 kB
          Dominique Dierickx

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              dominiek Dominique Dierickx
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: