Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17128

Operation Logging leaks file descriptors as the log4j Appender is never closed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Logging
    • Labels:
      None

      Description

      HIVE-16061 and HIVE-16400 changed operation logging to use the Log4j2 RoutingAppender to automatically output the log for each query into each individual operation log file. As log4j does not know when a query is finished it keeps the OutputStream in the Appender open even when the query completes. The stream holds a file descriptor and so we leak file descriptors. Note that we are already careful to close any streams reading from the operation log file.

      Fix

      To fix this we use a technique described in the comments of LOG4J2-510 which uses reflection to close the appender. The test in TestOperationLoggingLayout will be extended to check that the Appender is closed.

        Attachments

        1. HIVE-17128.1.patch
          11 kB
          Andrew Sherman
        2. HIVE-17128.2.patch
          11 kB
          Andrew Sherman
        3. HIVE-17128.3.patch
          15 kB
          Andrew Sherman

          Issue Links

            Activity

              People

              • Assignee:
                asherman Andrew Sherman
                Reporter:
                asherman Andrew Sherman
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: