Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-330

Use of multiple output counters can be disabled in configuration.

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.10.0, 0.8.3
    • Core, IO
    • None

    Description

      We're having some trouble with the amount of counters that Crunch creates
      when writing to a lot of different output files (slightly more than 120).
      This wouldn't be an issue if we were able to configure the maximum number
      of allowed counters but unfortunately, because we are running an older
      version of Hadoop, doing this is not an option and we are required to patch
      Crunch locally when using a new release to leave out the counters.

      I'm not saying the counters should be removed but maybe it is an option to
      make them configurable without paying too much of a performance penalty?

      I will implement this functionality and submit a patch.

      Attachments

        1. CRUNCH-330b.patch
          5 kB
          Josh Wills
        2. 0001-working-version.patch
          16 kB
          Dominique Dierickx

        Activity

          People

            jwills Josh Wills
            dominiek Dominique Dierickx
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: