Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6904

SparkSql - HiveContext - optimize reading partition data from metastore

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.3.0
    • None
    • SQL
    • None

    Description

      I was trying out spark sql using the HiveContext and doing a select on a partitioned table with lots of partitions (16,000+). It took over 6 minutes before it even started the job. It looks like it was querying the Hive metastore and got a good chunk of data back. Which I'm guessing is info on the partitions. Running the same query using hive takes 45 seconds for the entire job.

      It would be nice if we could optimize on the partitions when reading from the metastore.

      Attachments

        Issue Links

        duplicates

        Sub-task - The sub-task of the issue SPARK-6910 Support for pushing predicates down to metastore for partition pruning

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tgraves Thomas Graves
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment