Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17128

Operation Logging leaks file descriptors as the log4j Appender is never closed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Logging
    • None

    Description

      HIVE-16061 and HIVE-16400 changed operation logging to use the Log4j2 RoutingAppender to automatically output the log for each query into each individual operation log file. As log4j does not know when a query is finished it keeps the OutputStream in the Appender open even when the query completes. The stream holds a file descriptor and so we leak file descriptors. Note that we are already careful to close any streams reading from the operation log file.

      Fix

      To fix this we use a technique described in the comments of LOG4J2-510 which uses reflection to close the appender. The test in TestOperationLoggingLayout will be extended to check that the Appender is closed.

      Attachments

        1. HIVE-17128.1.patch
          11 kB
          Andrew Sherman
        2. HIVE-17128.2.patch
          11 kB
          Andrew Sherman
        3. HIVE-17128.3.patch
          15 kB
          Andrew Sherman

        Issue Links

          Activity

            People

              asherman Andrew Sherman
              asherman Andrew Sherman
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: