Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-556

Fix total sorts in Crunch-on-Spark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.13.0
    • 0.14.0
    • Spark
    • None

    Description

      From the user mailing list, trying to perform a total sort to create an HFile w/Crunch on Spark throws the following exception: https://gist.github.com/anonymous/920c000f20229eaa76d8

      The problem can be traced to not properly configuring the partitioner w/the path to the partition file that is stored in the GroupingOptions extra configuration settings. These settings get passed correctly for the MR job, but not for the Spark ones.

      Attachments

        1. CRUNCH-556b.patch
          22 kB
          Josh Wills
        2. CRUNCH-556.patch
          4 kB
          Josh Wills

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. CRUNCH-559 Crunch-Spark Integration Tests not executing

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jwills Josh Wills
              jwills Josh Wills
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: