Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14643

Remove overloaded methods which become ambiguous in Scala 2.12

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Blocker
    • Resolution: Won't Fix
    • 2.4.0
    • None
    • Build, Project Infra
    • None

    Description

      Spark 1.x's Dataset API runs into subtle source incompatibility problems for Java 8 and Scala 2.12 users when Spark is built against Scala 2.12. In a nutshell, the current API has overloaded methods whose signatures are ambiguous when resolving calls that use the Java 8 lambda syntax (only if Spark is build against Scala 2.12).

      This issue is somewhat subtle, so there's a full writeup at https://docs.google.com/document/d/1P_wmH3U356f079AYgSsN53HKixuNdxSEvo8nw_tgLgM/edit?usp=sharing which describes the exact circumstances under which the current APIs are problematic. The writeup also proposes a solution which involves the removal of certain overloads only in Scala 2.12 builds of Spark and the introduction of implicit conversions for retaining source compatibility.

      We don't need to implement any of these changes until we add Scala 2.12 support since the changes must only be applied when building against Scala 2.12 and will be done via traits + shims which are mixed in via per-Scala-version source directories (like how we handle the Scala-version-specific parts of the REPL). For now, this JIRA acts as a placeholder so that the parent JIRA reflects the complete set of tasks which need to be finished for 2.12 support.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            joshrosen Josh Rosen
            Votes:
            11 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment