Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Spark 2.1 will be released soon, create this ticket to track zeppelin to support spark 2.1.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zjffdu opened a pull request:

          https://github.com/apache/zeppelin/pull/1787

          ZEPPELIN-1815. Support Spark 2.1

              1. What is this PR for?
                This PR is to update the max supported spark version to be 2.1 as Spark 2.1 is released.
                I didn't change pom file and travis script to update spark version. I think we can wait for some time until we confirm spark 2.1 is stable enough.
              1. What type of PR is it?
                [Improvement]
              1. Todos
          • [ ] - Task
              1. What is the Jira issue?
              1. How should this be tested?
                Tested on spark 2.1
              1. Screenshots (if appropriate)
                ![sparkr_progress](https://cloud.githubusercontent.com/assets/164491/21373523/b380f44a-c75a-11e6-9004-e85009f9143a.gif)
              1. Questions:
          • Does the licenses files need update? No
          • Is there breaking changes for older versions? No
          • Does this needs documentation? No

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1815

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/zeppelin/pull/1787.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1787


          commit f7e72c93901dc291b3d322b4862ba83ae714d247
          Author: Jeff Zhang <zjffdu@apache.org>
          Date: 2016-12-21T00:01:48Z

          ZEPPELIN-1815. Support Spark 2.1


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zjffdu opened a pull request: https://github.com/apache/zeppelin/pull/1787 ZEPPELIN-1815 . Support Spark 2.1 What is this PR for? This PR is to update the max supported spark version to be 2.1 as Spark 2.1 is released. I didn't change pom file and travis script to update spark version. I think we can wait for some time until we confirm spark 2.1 is stable enough. What type of PR is it? [Improvement] Todos [ ] - Task What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1815 How should this be tested? Tested on spark 2.1 Screenshots (if appropriate) ! [sparkr_progress] ( https://cloud.githubusercontent.com/assets/164491/21373523/b380f44a-c75a-11e6-9004-e85009f9143a.gif ) Questions: Does the licenses files need update? No Is there breaking changes for older versions? No Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1815 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1787.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1787 commit f7e72c93901dc291b3d322b4862ba83ae714d247 Author: Jeff Zhang <zjffdu@apache.org> Date: 2016-12-21T00:01:48Z ZEPPELIN-1815 . Support Spark 2.1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          @Leemoonsoo @felixcheung Please help review.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 @Leemoonsoo @felixcheung Please help review.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Looks fine. Would you like to get this in before Spark 2.1 artifacts are in maven?
          I think we should reword the JIRA if that's the case - I don't think we should say we support 2.1 until we have 2.1 in the build (pom), travis etc.

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 Looks fine. Would you like to get this in before Spark 2.1 artifacts are in maven? I think we should reword the JIRA if that's the case - I don't think we should say we support 2.1 until we have 2.1 in the build (pom), travis etc.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          and btw, could you please test with the last 2.1 RC5 instead of snapshot builds.

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 and btw, could you please test with the last 2.1 RC5 instead of snapshot builds.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user bikassaha commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          In a different jira, should we consider removing this check for 2.1, 2.2 since its unlikely that Spark will break common APIs in dot releases. Perhaps the policy can change to adding a check when a known incompatible Spark version shows up (until that gets supported in Zeppelin). Thoughts?

          Show
          githubbot ASF GitHub Bot added a comment - Github user bikassaha commented on the issue: https://github.com/apache/zeppelin/pull/1787 In a different jira, should we consider removing this check for 2.1, 2.2 since its unlikely that Spark will break common APIs in dot releases. Perhaps the policy can change to adding a check when a known incompatible Spark version shows up (until that gets supported in Zeppelin). Thoughts?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          @bikassaha zeppelin use spark repl api heavily. And spark repl is not stable as RDD/DataFrame api (It happens in spark 1.x), so it is possible to have changes in spark repl for spark feature release. Another thing is that zeppelin may need some update to support spark new features, so we need to check spark version.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 @bikassaha zeppelin use spark repl api heavily. And spark repl is not stable as RDD/DataFrame api (It happens in spark 1.x), so it is possible to have changes in spark repl for spark feature release. Another thing is that zeppelin may need some update to support spark new features, so we need to check spark version.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          2.1.0 artifacts are on central repo. @zjffdu do you want to update this PR to add build against 2.1?

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 2.1.0 artifacts are on central repo. @zjffdu do you want to update this PR to add build against 2.1?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          the 2.1 profile failed, could you check?

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 the 2.1 profile failed, could you check?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          @felixcheung The test fail is due to api change in spark 2.1.0.
          SPARK-17919 remove timeout, not sure why. I don't think we need to remove it.
          https://github.com/apache/spark/pull/15471/files. @felixcheung I see you are the reviewer, you may know the background.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 @felixcheung The test fail is due to api change in spark 2.1.0. SPARK-17919 remove timeout, not sure why. I don't think we need to remove it. https://github.com/apache/spark/pull/15471/files . @felixcheung I see you are the reviewer, you may know the background.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          ah, that's unfortunate. the feedback was it would be inconsistent and confusing to have multiple default values for the timeout, instead it is moved to the top of the [function](https://github.com/apache/spark/pull/15471/files#diff-8ed17b5ecd168e47a2433cc9b3bc5b17R157) - but I can see the caller of connectBackend (though not officially supported) can be broken by this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 ah, that's unfortunate. the feedback was it would be inconsistent and confusing to have multiple default values for the timeout, instead it is moved to the top of the [function] ( https://github.com/apache/spark/pull/15471/files#diff-8ed17b5ecd168e47a2433cc9b3bc5b17R157 ) - but I can see the caller of connectBackend (though not officially supported) can be broken by this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          CI failure doesn't seem related.
          ```
          java.lang.AssertionError: expected:<FINISHED> but was:<ERROR>
          at org.junit.Assert.fail(Assert.java:88)
          at org.junit.Assert.failNotEquals(Assert.java:834)
          at org.junit.Assert.assertEquals(Assert.java:118)
          at org.junit.Assert.assertEquals(Assert.java:144)
          at org.apache.zeppelin.rest.ZeppelinSparkClusterTest.zRunTest(ZeppelinSparkClusterTest.java:347)

          ```

          LGTM.

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 CI failure doesn't seem related. ``` java.lang.AssertionError: expected:<FINISHED> but was:<ERROR> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.zeppelin.rest.ZeppelinSparkClusterTest.zRunTest(ZeppelinSparkClusterTest.java:347) ``` LGTM.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/zeppelin/pull/1787

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/zeppelin/pull/1787
          Hide
          jongyoul Jongyoul Lee added a comment -

          Issue resolved by pull request 1787
          https://github.com/apache/zeppelin/pull/1787

          Show
          jongyoul Jongyoul Lee added a comment - Issue resolved by pull request 1787 https://github.com/apache/zeppelin/pull/1787
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ejono commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          When I compile Zeppelin (from master, including this change) against Spark 2.1.0, I hit the following error:

          [ERROR] /Volumes/Amazon/oss-workspace/src/zeppelin/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java:[1454,26] constructor SecurityManager in class org.apache.spark.SecurityManager cannot be applied to given types;
          required: org.apache.spark.SparkConf,scala.Option<byte[]>
          found: org.apache.spark.SparkConf
          reason: actual and formal argument lists differ in length
          [ERROR] /Volumes/Amazon/oss-workspace/src/zeppelin/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java:[1465,22] constructor SecurityManager in class org.apache.spark.SecurityManager cannot be applied to given types;
          required: org.apache.spark.SparkConf,scala.Option<byte[]>
          found: org.apache.spark.SparkConf
          reason: actual and formal argument lists differ in length

          It doesn't matter if I'm using Scala 2.10 or 2.11. I can reproduce this with a command as simple as:

          mvn clean package -DskipTests -Pspark-2.0 -Dspark.version=2.1.0

          I'm not all that familiar with Scala, but I don't understand why this is a problem, as this Optional<byte[]> argument of the SecurityManager class (from Spark's source code) has a default of None, so I wouldn't think that it would be necessary to specify this second parameter to the constructor in Zeppelin code. Is it not possible for Java code to take advantage of the default value specified in the Scala code?

          Also, isn't this breaking the Zeppelin CI? I don't know where to find the Zeppelin CI, so could somebody please provide a link?

          Show
          githubbot ASF GitHub Bot added a comment - Github user ejono commented on the issue: https://github.com/apache/zeppelin/pull/1787 When I compile Zeppelin (from master, including this change) against Spark 2.1.0, I hit the following error: [ERROR] /Volumes/Amazon/oss-workspace/src/zeppelin/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java: [1454,26] constructor SecurityManager in class org.apache.spark.SecurityManager cannot be applied to given types; required: org.apache.spark.SparkConf,scala.Option<byte[]> found: org.apache.spark.SparkConf reason: actual and formal argument lists differ in length [ERROR] /Volumes/Amazon/oss-workspace/src/zeppelin/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java: [1465,22] constructor SecurityManager in class org.apache.spark.SecurityManager cannot be applied to given types; required: org.apache.spark.SparkConf,scala.Option<byte[]> found: org.apache.spark.SparkConf reason: actual and formal argument lists differ in length It doesn't matter if I'm using Scala 2.10 or 2.11. I can reproduce this with a command as simple as: mvn clean package -DskipTests -Pspark-2.0 -Dspark.version=2.1.0 I'm not all that familiar with Scala, but I don't understand why this is a problem, as this Optional<byte[]> argument of the SecurityManager class (from Spark's source code) has a default of None, so I wouldn't think that it would be necessary to specify this second parameter to the constructor in Zeppelin code. Is it not possible for Java code to take advantage of the default value specified in the Scala code? Also, isn't this breaking the Zeppelin CI? I don't know where to find the Zeppelin CI, so could somebody please provide a link?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ejono commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Answering my own questions:

          I still haven't found a definitive answer to whether or not Java classes may take advantage of default parameter values defined in Scala classes, but it's pretty apparent that you cannot do this. This means that in order to support both Spark < 2.1.0 and Spark >= 2.1.0, we'd need to do a little more reflection magic to support the two possible versions of the SecurityManager constructor.

          As for the link to the Jenkins CI, I found https://travis-ci.org/apache/zeppelin. I don't at all understand though how the new Spark 2.1.0 profile is building successfully when I've found it so easy to reproduce this compilation error.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ejono commented on the issue: https://github.com/apache/zeppelin/pull/1787 Answering my own questions: I still haven't found a definitive answer to whether or not Java classes may take advantage of default parameter values defined in Scala classes, but it's pretty apparent that you cannot do this. This means that in order to support both Spark < 2.1.0 and Spark >= 2.1.0, we'd need to do a little more reflection magic to support the two possible versions of the SecurityManager constructor. As for the link to the Jenkins CI, I found https://travis-ci.org/apache/zeppelin . I don't at all understand though how the new Spark 2.1.0 profile is building successfully when I've found it so easy to reproduce this compilation error.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Thanks @ejono for reporting this, I can also reproduce it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 Thanks @ejono for reporting this, I can also reproduce it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          yea, this is a bit puzzling...

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 yea, this is a bit puzzling...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ejono commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Thanks for confirming. The fix would not be too difficult, but yes, I'm puzzled as to why the CI isn't broken, so maybe we need to figure that out first.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ejono commented on the issue: https://github.com/apache/zeppelin/pull/1787 Thanks for confirming. The fix would not be too difficult, but yes, I'm puzzled as to why the CI isn't broken, so maybe we need to figure that out first.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          @ejono The root cause is that SPARK_VER doesn't take effect in travis, I created ZEPPELIN-1918. It would be better to fix the issue here with ZEPPELIN-1918 together. I would be happy to help if you want to contribute on this. BTW, you might need to create another profile for spark 2.1.0, because spark 2.1.0 use another version of pyspark.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 @ejono The root cause is that SPARK_VER doesn't take effect in travis, I created ZEPPELIN-1918 . It would be better to fix the issue here with ZEPPELIN-1918 together. I would be happy to help if you want to contribute on this. BTW, you might need to create another profile for spark 2.1.0, because spark 2.1.0 use another version of pyspark.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jongyoul commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Oh, sorry I also didn't check it carefully. I'll check it manually when you guys create new PR for handling a new issue

          Show
          githubbot ASF GitHub Bot added a comment - Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1787 Oh, sorry I also didn't check it carefully. I'll check it manually when you guys create new PR for handling a new issue
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          this #1856 should help too

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 this #1856 should help too
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          ping @ejono Do you want to contribute it ? If not, I will fix it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 ping @ejono Do you want to contribute it ? If not, I will fix it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user felixcheung commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Let's get this fix ASAP - we are going to cut 0.7 release in 2 days

          Show
          githubbot ASF GitHub Bot added a comment - Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1787 Let's get this fix ASAP - we are going to cut 0.7 release in 2 days
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zjffdu commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Sure, let me fix it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1787 Sure, let me fix it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ejono commented on the issue:

          https://github.com/apache/zeppelin/pull/1787

          Sorry, I was not able to work on this over the past few days. Thank you very much for fixing it, @zjffdu!

          Show
          githubbot ASF GitHub Bot added a comment - Github user ejono commented on the issue: https://github.com/apache/zeppelin/pull/1787 Sorry, I was not able to work on this over the past few days. Thank you very much for fixing it, @zjffdu!

            People

            • Assignee:
              zjffdu Jeff Zhang
              Reporter:
              zjffdu Jeff Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development