Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-556

Fix total sorts in Crunch-on-Spark

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: Spark
    • Labels:
      None

      Description

      From the user mailing list, trying to perform a total sort to create an HFile w/Crunch on Spark throws the following exception: https://gist.github.com/anonymous/920c000f20229eaa76d8

      The problem can be traced to not properly configuring the partitioner w/the path to the partition file that is stored in the GroupingOptions extra configuration settings. These settings get passed correctly for the MR job, but not for the Spark ones.

        Attachments

        1. CRUNCH-556.patch
          4 kB
          Josh Wills
        2. CRUNCH-556b.patch
          22 kB
          Josh Wills

          Issue Links

            Activity

              People

              • Assignee:
                jwills Josh Wills
                Reporter:
                jwills Josh Wills
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: