[SPARK-3936] Remove auto join elimination and introduce aggregateMessages

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: GraphX
• Assignee: Ankur Dave
• Reporter: Pedro Rodriguez
• Shepherd: Ankur Dave
• Created: 13/Oct/14 22:53
• Updated: 26/Nov/14 05:43
• Resolved: 12/Nov/14 07:47

Description

This is actually a ticket with two separate problems:

1. Remove auto join elimination

2. Introduce a new fundamental primitive aggregateMessages

For the first one, description provided by Pedro:

There seems to be a bug with the GraphX byte code inspection, specifically in BytecodeUtils.

These are the unit tests I wrote to expose the problem:
https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121

The first two tests pass, the second two tests fail. This exposes a problem with inspection of methods in closures, in this case within maps. Specifically, it seems like there is a problem with inspection of non-inline methods in a closure.

For the 2nd one, see pull request https://github.com/apache/spark/pull/3100

Issue Links

• is depended upon by: SPARK-3664 Graduate GraphX from alpha to stable
• links to: [Github] Pull Request #2815 (jegonzal)
• links to: [Github] Pull Request #3100 (ankurdave)