Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7088

[REGRESSION] Spark 1.3.1 breaks analysis of third-party logical plans

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.5.0
    • Component/s: SQL
    • Labels:

      Description

      We're using some custom logical plans. We are now migrating from Spark 1.3.0 to 1.3.1 and found a few incompatible API changes. All of them seem to be in internal code, so we understand that. But now the ResolveReferences rule, that used to work with third-party logical plans just does not work, without any possible workaround that I'm aware other than just copying ResolveReferences rule and using it with our own fix.

      The change in question is this section of code:

              }.headOption.getOrElse { // Only handle first case, others will be fixed on the next pass.
                sys.error(
                  s"""
                    |Failure when resolving conflicting references in Join:
                    |$plan
                    |
                    |Conflicting attributes: ${conflictingAttributes.mkString(",")}
                    """.stripMargin)
              }
      

      Which causes the following error on analysis:

      Failure when resolving conflicting references in Join:
      'Project ['l.name,'r.name,'FUNC1('l.node,'r.node) AS c2#37,'FUNC2('l.node,'r.node) AS c3#38,'FUNC3('r.node,'l.node) AS c4#39]
       'Join Inner, None
        Subquery l
         Subquery h
          Project [name#12,node#36]
           CustomPlan H, u, (p#13L = s#14L), [ord#15 ASC], IS NULL p#13L, node#36
            Subquery v
             Subquery h_src
              LogicalRDD [name#12,p#13L,s#14L,ord#15], MapPartitionsRDD[1] at mapPartitions at ExistingRDD.scala:37
        Subquery r
         Subquery h
          Project [name#40,node#36]
           CustomPlan H, u, (p#41L = s#42L), [ord#43 ASC], IS NULL pred#41L, node#36
            Subquery v
             Subquery h_src
              LogicalRDD [name#40,p#41L,s#42L,ord#43], MapPartitionsRDD[1] at mapPartitions at ExistingRDD.scala:37
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                smolav Santiago M. Mola
                Reporter:
                smolav Santiago M. Mola
                Shepherd:
                Michael Armbrust
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: