Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3936

Remove auto join elimination and introduce aggregateMessages

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: GraphX
    • Labels:
      None

      Description

      This is actually a ticket with two separate problems:

      1. Remove auto join elimination

      2. Introduce a new fundamental primitive aggregateMessages

      For the first one, description provided by Pedro:

      There seems to be a bug with the GraphX byte code inspection, specifically in BytecodeUtils.

      These are the unit tests I wrote to expose the problem:
      https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121

      The first two tests pass, the second two tests fail. This exposes a problem with inspection of methods in closures, in this case within maps. Specifically, it seems like there is a problem with inspection of non-inline methods in a closure.

      For the 2nd one, see pull request https://github.com/apache/spark/pull/3100

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ankurd Ankur Dave
                Reporter:
                pedrorodriguez Pedro Rodriguez
                Shepherd:
                Ankur Dave
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: