Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-437

Fix Crunch Spark duplicate value aggregation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.8.4, 0.11.0
    • None
    • None

    Description

      The current Crunch-on-Spark mapside combiner uses a Multimap of key-value pairs to cache values for local aggregation. This is awesome, except it means that identical key-value outputs before a shuffle will only have one copy in the Multimap, which means that the aggregation counts may not be correct. We should fix it to use a proper Map<K, List<V>> to ensure that duplicate values are aggregated correctly.

      Attachments

        1. CRUNCH-437.patch
          7 kB
          Josh Wills

        Activity

          People

            Unassigned Unassigned
            jwills Josh Wills
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: