Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26090

Resolve most miscellaneous deprecation and build warnings for Spark 3

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: ML, Spark Core, SQL
    • Labels:
    • Target Version/s:
    • Docs Text:
      StageInfo.attemptId has been removed in favor of StageInfo.attemptNumber()

      Description

      The build has a lot of deprecation warnings. Some are new in Scala 2.12 and Java 11. We've fixed some, but I wanted to take a pass at fixing lots of easy miscellaneous ones here.

      They're too numerous and small to list here; see the pull request. Some highlights:

      • @BeanInfo is deprecated in 2.12, and BeanInfo classes are pretty ancient in Java. Instead, case classes can explicitly declare getters
      • Eta expansion of zero-arg methods; foo() becomes () => foo() in many cases
      • Floating-point Range is inexact and deprecated, like 0.0 to 100.0 by 1.0
      • finalize() is finally deprecated (just needs to be suppressed)
      • StageInfo.attempId was deprecated and easiest to remove here

      I'm not now going to touch some chunks of deprecation warnings:

      • Parquet deprecations
      • Hive deprecations (particularly serde2 classes)
      • Deprecations in generated code (mostly Thriftserver CLI)
      • ProcessingTime deprecations (we may need to revive this class as internal)
      • many MLlib deprecations because they concern methods that may be removed anyway
      • a few Kinesis deprecations I couldn't figure out
      • Mesos get/setRole, which I don't know well
      • Kafka/ZK deprecations (e.g. poll())
      • Kinesis
      • a few other ones that will probably resolve by deleting a deprecated method

        Attachments

          Activity

            People

            • Assignee:
              srowen Sean Owen
              Reporter:
              srowen Sean Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: