Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21225

ACID: getAcidState() should cache a recursive dir listing locally

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Transactions
    • Labels:
      None

      Description

      Currently getAcidState() makes 3 calls into the FS api which could be answered by making a single recursive listDir call and reusing the same data to check for isRawFormat() and isValidBase().

      All delta operations for a single partition can go against a single listed directory snapshot instead of interacting with the NameNode or ObjectStore within the inner loop.

        Attachments

        1. async-pid-44-2.svg
          619 kB
          Gopal V
        2. HIVE-21225.1.patch
          13 kB
          Vaibhav Gumashta
        3. HIVE-21225.2.patch
          53 kB
          Vaibhav Gumashta
        4. HIVE-21225.3.patch
          9 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                gopalv Gopal V
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: