Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2585

Remove special handling of Hadoop JobConf

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      This is a follow up to SPARK-2521 and should close SPARK-2546 (provided the implementation does not use shared conf objects). We no longer need to specially broadcast the Hadoop configuration since we are broadcasting RDD data anyways.

        Attachments

          Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4083 Remove all unnecessary broadcasts

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-2758 UnionRDD's UnionPartition should not reference parent RDDs

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-2546 Configuration object thread safety issue

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #1648 [Github] Pull Request #1648 (rxin)

          Pull request #2683 [Github] Pull Request #2683 (JoshRosen)

          Pull request #2935 [Github] Pull Request #2935 (davies)

            Activity

              People

              • Assignee:
                joshrosen Josh Rosen
                Reporter:
                pwendell Patrick Wendell
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: