Description
This is actually a ticket with two separate problems:
1. Remove auto join elimination
2. Introduce a new fundamental primitive aggregateMessages
For the first one, description provided by Pedro:
There seems to be a bug with the GraphX byte code inspection, specifically in BytecodeUtils.
These are the unit tests I wrote to expose the problem:
https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
The first two tests pass, the second two tests fail. This exposes a problem with inspection of methods in closures, in this case within maps. Specifically, it seems like there is a problem with inspection of non-inline methods in a closure.
For the 2nd one, see pull request https://github.com/apache/spark/pull/3100
Attachments
Attachments
Issue Links
- is depended upon by
-
SPARK-3664 Graduate GraphX from alpha to stable
- Resolved
- links to