Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-658

Add a way to skip the getSize checks for Sources from object stores

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.14.0
    • None
    • Core
    • None

    Description

      Ran into a problem when using Crunch to process a lot of data from S3: the getSize checks can be very slow to run and don't materially add much to the overall processing of a pipeline when things like reducer counts are manually specified. I'd like to add a way to disable the file size checks, either globally or for specific input sources.

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. CRUNCH-683 Avoid unnecessary listStatus calls from getSize computation

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              jwills Josh Wills
              jwills Josh Wills
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: