Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2585

Remove special handling of Hadoop JobConf

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Won't Fix
    • None
    • None
    • Spark Core
    • None

    Description

      This is a follow up to SPARK-2521 and should close SPARK-2546 (provided the implementation does not use shared conf objects). We no longer need to specially broadcast the Hadoop configuration since we are broadcasting RDD data anyways.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4083 Remove all unnecessary broadcasts

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-2758 UnionRDD's UnionPartition should not reference parent RDDs

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-2546 Configuration object thread safety issue

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #1648 [Github] Pull Request #1648 (rxin)

          Pull request #2683 [Github] Pull Request #2683 (JoshRosen)

          Pull request #2935 [Github] Pull Request #2935 (davies)

          Activity

            People

              joshrosen Josh Rosen
              pwendell Patrick Wendell
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: