Hive
  1. Hive
  2. HIVE-2030

isEmptyPath() to use ContentSummary cache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      addInputPaths() calls isEmptyPath() for every input path. Now every call is a DFS namenode call. Making isEmptyPath() to use cached ContentSummary, we should be able to avoid some namenode calls and reduce latency in the case of multiple partitions.

      1. HIVE-2030.1.patch
        5 kB
        Siying Dong
      2. HIVE-2030.2.patch
        5 kB
        Siying Dong
      3. HIVE-2030.3.patch
        4 kB
        Siying Dong

        Activity

        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Component/s Query Processor [ 12312586 ]
        He Yongqiang made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        He Yongqiang added a comment -

        committed. thanks siying!

        Show
        He Yongqiang added a comment - committed. thanks siying!
        Hide
        He Yongqiang added a comment -

        running tests with the new patch

        Show
        He Yongqiang added a comment - running tests with the new patch
        Siying Dong made changes -
        Attachment HIVE-2030.3.patch [ 12473313 ]
        Hide
        He Yongqiang added a comment -

        siying, can you update the patch?

        Show
        He Yongqiang added a comment - siying, can you update the patch?
        Hide
        He Yongqiang added a comment -

        okay, will test and commit.

        Show
        He Yongqiang added a comment - okay, will test and commit.
        Hide
        Siying Dong added a comment -

        Yongqiang, I don't quite understand your comment. If there is a cache miss, we call the original method. We never make things worse.

        Show
        Siying Dong added a comment - Yongqiang, I don't quite understand your comment. If there is a cache miss, we call the original method. We never make things worse.
        Siying Dong made changes -
        Attachment HIVE-2030.2.patch [ 12473069 ]
        Hide
        Siying Dong added a comment -

        In the case of Exception, we don't populate cache. It's to make sure cache never gets wrong value.

        Show
        Siying Dong added a comment - In the case of Exception, we don't populate cache. It's to make sure cache never gets wrong value.
        He Yongqiang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        He Yongqiang added a comment -

        The ContentSummary is not guaranteed to be populated. Even it is, it seems this information is not passed to the child process. (So this is not empty only when executing with local mode)

        Show
        He Yongqiang added a comment - The ContentSummary is not guaranteed to be populated. Even it is, it seems this information is not passed to the child process. (So this is not empty only when executing with local mode)
        Siying Dong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siying Dong made changes -
        Field Original Value New Value
        Attachment HIVE-2030.1.patch [ 12472891 ]
        Siying Dong created issue -

          People

          • Assignee:
            Siying Dong
            Reporter:
            Siying Dong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development