Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-463

Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs with lots of splits

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.8.4, 0.11.0
    • Component/s: Core
    • Labels:
      None

      Description

      Trying to run a job with 11k input files and that yields about 25k splits results in OOM errors due to too many copies of the Configuration object being created when the CrunchInputSplit is initialised. I know that that's the result of CRUNCH-313, but perhaps a better way to deal with that problem should be found.

        Attachments

        1. CRUNCH-463.patch
          2 kB
          Josh Wills

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              hekoru Hector Izquierdo Seliva
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: