Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14220

Build and test Spark against Scala 2.12

    Details

    • Type: Umbrella
    • Status: Reopened
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Build, Project Infra
    • Labels:
      None

      Description

      This umbrella JIRA tracks the requirements for building and testing Spark against the current Scala 2.12 milestone.

        Issue Links

          Activity

          Hide
          rxin Reynold Xin added a comment -

          Concluding the investigation here. After extensive investigation, the belief is that we can make Spark 2.x work with Scala 2.12 when it officially comes out.

          Show
          rxin Reynold Xin added a comment - Concluding the investigation here. After extensive investigation, the belief is that we can make Spark 2.x work with Scala 2.12 when it officially comes out.
          Hide
          jodersky Jakob Odersky added a comment -

          Scala 2.12 was just officially announced

          Show
          jodersky Jakob Odersky added a comment - Scala 2.12 was just officially announced
          Hide
          rxin Reynold Xin added a comment -

          Yea in reality it's going to be really painful to upgrade.

          Show
          rxin Reynold Xin added a comment - Yea in reality it's going to be really painful to upgrade.
          Hide
          jodersky Jakob Odersky added a comment - - edited

          At least most dependencies will probably make 2.12 builds available, now that it is considered binary-stable. The closure cleaning and byte code manipulation stuff is a whole different story though...

          Show
          jodersky Jakob Odersky added a comment - - edited At least most dependencies will probably make 2.12 builds available, now that it is considered binary-stable. The closure cleaning and byte code manipulation stuff is a whole different story though...
          Hide
          joshrosen Josh Rosen added a comment -

          SPARK-14643 is likely to be the hardest task.

          Show
          joshrosen Josh Rosen added a comment - SPARK-14643 is likely to be the hardest task.
          Hide
          antonkulaga antonkulaga added a comment -

          Any progress on this?

          Show
          antonkulaga antonkulaga added a comment - Any progress on this?
          Hide
          srowen Sean Owen added a comment -

          antonkulaga what you see is what you get here. You can answer this by looking at linked JIRAs, looking at related commits, etc.

          Show
          srowen Sean Owen added a comment - antonkulaga what you see is what you get here. You can answer this by looking at linked JIRAs, looking at related commits, etc.
          Hide
          xwinus Vaclav Svejcar added a comment -

          Hi, I understand it's not an easy task to build Spark for Scala 2.12, but according to this ticket, 2.12 support should be in release 2.3.0, that's still two releases ahead. Do you have any ETA for this? Thanks.

          Show
          xwinus Vaclav Svejcar added a comment - Hi, I understand it's not an easy task to build Spark for Scala 2.12, but according to this ticket, 2.12 support should be in release 2.3.0, that's still two releases ahead. Do you have any ETA for this? Thanks.
          Hide
          rxin Reynold Xin added a comment -

          I just removed the target version given the amount of work. Sorry no ETA at this point. It is just a huge amount of work required.

          Show
          rxin Reynold Xin added a comment - I just removed the target version given the amount of work. Sorry no ETA at this point. It is just a huge amount of work required.
          Hide
          srowen Sean Owen added a comment -

          I have a branch locally for 2.12. That much isn't a lot of work. The dependencies maybe. I agree that it's probably not going to happen for 2.3.x but maybe by the end of the year.

          Show
          srowen Sean Owen added a comment - I have a branch locally for 2.12. That much isn't a lot of work. The dependencies maybe. I agree that it's probably not going to happen for 2.3.x but maybe by the end of the year.
          Hide
          rxin Reynold Xin added a comment -

          Making it build isn't that much work, but getting the API to work (especially the Dataset one) will be a huge.

          Show
          rxin Reynold Xin added a comment - Making it build isn't that much work, but getting the API to work (especially the Dataset one) will be a huge.
          Hide
          fwbrasil Flavio Brasil added a comment -

          Reynold Xin Could you expand your last comment? Is it hard because of the issue described in this doc ? Scala 2.12 already supports automatic conversion from scala functions to java functional interfaces, maybe that wasn't the case at the time that the doc was written.

          Spark is one of the last dependencies at Twitter that doesn't support Scala 2.12. Would it be possible for us to collaborate on this?

          Show
          fwbrasil Flavio Brasil added a comment - Reynold Xin Could you expand your last comment? Is it hard because of the issue described in this doc ? Scala 2.12 already supports automatic conversion from scala functions to java functional interfaces, maybe that wasn't the case at the time that the doc was written. Spark is one of the last dependencies at Twitter that doesn't support Scala 2.12. Would it be possible for us to collaborate on this?
          Hide
          rxin Reynold Xin added a comment -

          If all those issues have been released than it would be easy. Do you want to take a stab at that and see if it works with the DataFrame API?

          Show
          rxin Reynold Xin added a comment - If all those issues have been released than it would be easy. Do you want to take a stab at that and see if it works with the DataFrame API?
          Hide
          luo8828959 CasterKKK added a comment -

          Hello I wonder is everything ok? When could we use the spark with scala 2.12. Don't forget that the scala 2.13 is coming.
          And I have to say that the iteration of spark recently is mush slower than before.

          Show
          luo8828959 CasterKKK added a comment - Hello I wonder is everything ok? When could we use the spark with scala 2.12. Don't forget that the scala 2.13 is coming. And I have to say that the iteration of spark recently is mush slower than before.
          Hide
          srowen Sean Owen added a comment -

          It is hung up on a Scala compiler problem fixed in 2.12.4 and the closure cleaner method above which will be a little tricky. Jump in and work on them if interested

          Show
          srowen Sean Owen added a comment - It is hung up on a Scala compiler problem fixed in 2.12.4 and the closure cleaner method above which will be a little tricky. Jump in and work on them if interested

            People

            • Assignee:
              Unassigned
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              36 Vote for this issue
              Watchers:
              75 Start watching this issue

              Dates

              • Created:
                Updated:

                Development