[SPARK-11806] Spark 2.0 deprecations and removals

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Labels: releasenotes
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 18/Nov/15 06:19
• Updated: 12/Jun/16 18:45
• Resolved: 18/May/16 17:18

Description

This is an umbrella ticket to track things we are deprecating and removing in Spark 2.0.

Issue Links

• relates to: SPARK-12169 SparkR 2.0

Sub-Tasks

• Remove support for Hadoop < 2.2 (i.e. Hadoop 1 and 2.0): Resolved
• Remove Bagel: Resolved
• Remove Bagel: Resolved
• Remove Guava's "Optional" from public API: Resolved
• SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]: Resolved
• Remove Mesos fine-grained mode subject to discussions: Resolved
• Remove YARN Client / ClientArguments: Resolved
• Remove history serving functionality from standalone Master: Resolved
• Remove closure serializer: Resolved
• Java mapPartitions Iterator->Iterable is inconsistent with Scala's Iterator->Iterator: Resolved
• Remove usage of Hadoop deprecated APIs and reflection that supported 1.x: Resolved
• Rename "dynamic allocation" to "elastic scaling": Closed
• Remove JobLogger: Resolved
• Remove HTTPBroadcast: Resolved
• Remove TTL-based metadata cleaning (spark.cleaner.ttl): Resolved
• Confusing behavior of fold function of RDD in pyspark: Resolved
• Remove deprecated methods in SQL / DataFrames: Resolved
• Java count(AprroxDistinct)ByKey methods return Scala Long not Java: Resolved
• Remove some deprecated APIs in RDD/SparkContext: Resolved
• Remove SparkContext preferred locations constructor: Resolved
• Clean up build warnings: 2.0.0 edition: Resolved
• Remove unused code related to Hadoop 0.23: Resolved
• Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used: Resolved
• Remove block manager's internal "external block store" API: Resolved
• Remove submit python/R scripts through pyspark/sparkR: Resolved
• Remove HistoryServer old multiple files format: Resolved
• Deprecate TaskContext.isRunningLocally(): Resolved
• Change default build to Hadoop 2.7: Resolved
• Follow up SPARK-12707, Update documentation and other related code: Resolved
• Spark on Yarn remove deprecated configs for 2.0: Closed
• Remove usage of ADD_FILES in pyspark: Resolved
• Remove deprecate APIs from GraphX: Resolved
• Deprecate "yarn-client" and "yarn-cluster": Resolved
• Logging: Make it either stable or private[spark]: Resolved
• Move UserDefinedFunction into sql.expressions package: Resolved
• Upgrade to Jersey 2: Resolved
• Remove SparkContext.metricsSystem: Resolved
• Remove the support of SIMR cluster manager: Resolved
• Remove reference to Tachyon in cluster & release script: Resolved
• Move network/* modules into common/network-*: Resolved
• Move tags and unsafe modules into common: Resolved
• Move docker, extras modules into external: Resolved
• Move misc top-level build files into appropriate subdirs: Resolved
• remove typed operations (map, flatMap, mapPartitions) from Python DataFrame: Resolved
• Move streaming-flume, streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages: Resolved
• Move docs of streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages: Resolved
• Consider removing Kafka modules from Spark / Spark Streaming: Closed
• Remove Streaming Akka docs from Spark: Resolved
• Move streaming-flume back to Spark: Resolved
• Remove straggler references to Tachyon: Resolved
• Require Java 8 for Spark 2.x: Closed
• Change SparkListener from a trait to an abstract class, and remove JavaSparkListener: Resolved
• Remove HashShuffleManager: Resolved
• Put examples packaging on a diet: Resolved
• Java sampleByKey methods take ju.Map but with Scala Double values; results in type Object: Resolved
• Remove support for "zk://" master URL: Resolved
• Add a deprecation warning for Python 2.6: Resolved