Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-14165

Remove Hive file listing during split computation

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 4.0.0
    • Component/s: None
    • Target Version/s:

      Description

      The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's FileInputFormat.java will list the files during split computation anyway to determine their size. One way to remove this is to catch the InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the Hive side instead of doing the file listing beforehand.

      For S3 select queries on partitioned tables, this results in a 2x speedup.

        Attachments

        1. HIVE-14165.02.patch
          4 kB
          Abdullah Yousufi
        2. HIVE-14165.03.patch
          4 kB
          Abdullah Yousufi
        3. HIVE-14165.04.patch
          4 kB
          Sahil Takiar
        4. HIVE-14165.05.patch
          4 kB
          Sahil Takiar
        5. HIVE-14165.06.patch
          4 kB
          Sahil Takiar
        6. HIVE-14165.07.patch
          4 kB
          Sahil Takiar
        7. HIVE-14165.patch
          4 kB
          Abdullah Yousufi

          Issue Links

            Activity

              People

              • Assignee:
                pvargacl Peter Varga
                Reporter:
                ayousufi Abdullah Yousufi
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m