Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3936

Remove auto join elimination and introduce aggregateMessages

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.2.0
    • GraphX
    • None

    Description

      This is actually a ticket with two separate problems:

      1. Remove auto join elimination

      2. Introduce a new fundamental primitive aggregateMessages

      For the first one, description provided by Pedro:

      There seems to be a bug with the GraphX byte code inspection, specifically in BytecodeUtils.

      These are the unit tests I wrote to expose the problem:
      https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121

      The first two tests pass, the second two tests fail. This exposes a problem with inspection of methods in closures, in this case within maps. Specifically, it seems like there is a problem with inspection of non-inline methods in a closure.

      For the 2nd one, see pull request https://github.com/apache/spark/pull/3100

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ankurd Ankur Dave
            pedrorodriguez Pedro Rodriguez
            Ankur Dave Ankur Dave
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment