Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5099

Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta, 3.0.0-alpha1
    • Fix Version/s: 2.1.1-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In Namenode#copyEditLogSegmentsToSharedDir() method, we open a collection of EditLogInputStreams to read and apply to shareEditlog. In readOpt() method, we will open the underlying log file on disk. After applying all the opts, we do not close the collection of streams currently. This lead to a file handle leak on Windows as later we would fail to delete those files.

      This happens in TestInitializeSharedEdits test case, where we explicitly called Namenode# initializeSharedEdits(), where copyEditLogSegmentsToSharedDir() is used. Later we fail to create new MiniDFSCluster with the following exception.

      java.io.IOException: Could not fully delete C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1
      	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:334)
      	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316)
      	at org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      …
      

        Attachments

        1. HDFS-5099-trunk.patch
          3 kB
          Chuan Liu

          Issue Links

            Activity

              People

              • Assignee:
                chuanliu Chuan Liu
                Reporter:
                chuanliu Chuan Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: