Uploaded image for project: 'Apache HAWQ'
  1. Apache HAWQ
  2. HAWQ-1622

Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.4.0.0
    • PXF
    • None

    Description

      Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down PXF request response time significantly when several threads work concurrently as it locks FileSystem cache and holds the lock while the cleanup of DFSClients is completed.

      This can be avoided by caching the proxy UGI for a given proxy user between requests. Care must be taken to remove the cached entry after some pre-defined TTL if and only if there are no current threads using any FileSystem entries held by the cache. A combination of TTL-based cache with ref-counting might be utilized to achieve this.

       

      For some example of this, see: 

      https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java

      Caching UGIs might be tricky when Kerberos support is implemented later, see: https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            lavjain Lav Jain
            adenissov Alexander Denissov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment