Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2271

startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, BackUPNode will proceed with exceptions.

      1. HDFS-2271.1.patch
        7 kB
        Uma Maheswara Rao G
      2. HDFS-2271.1.patch
        7 kB
        Uma Maheswara Rao G
      3. HDFS-2271.patch
        7 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Suresh Srinivas added a comment -

        Uma, can you please keep the description brief and add the details description to a comment to avoid lengthy emails.

        Show
        Suresh Srinivas added a comment - Uma, can you please keep the description brief and add the details description to a comment to avoid lengthy emails.
        Hide
        Uma Maheswara Rao G added a comment -

        Following is the trace.

        2011-04-09 21:24:43,892 ERROR namenode.FSNamesystem (FSEditLog.java:logSync(1207)) - Unable to sync edit log.
        org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException: /hdfsdata/7/patch/name/data/current/edits (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog$EditLogFileOutputStream.<init>(FSEditLog.java:150)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createEditLogFile(FSEditLog.java:427)
        at org.apache.hadoop.hdfs.server.namenode.BackupStorage.startJournalSpool(BackupStorage.java:275)
        at org.apache.hadoop.hdfs.server.namenode.BackupNode.journal(BackupNode.java:327)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099)

        at org.apache.hadoop.ipc.Client.call(Client.java:953)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:239)
        at $Proxy4.journal(Unknown Source)
        at org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.send(EditLogBackupOutputStream.java:165)
        at org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.flushAndSync(EditLogBackupOutputStream.java:145)
        at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:83)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1204)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCheckpoint(FSNamesystem.java:5164)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCheckpoint(NameNode.java:547)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099)
        2011-04-09 21:24:43,893 ERROR namenode.FSNamesystem (FSEditLog.java:processIOError(557)) - Unable to log edits to 10.18.53.203:50100
        2011-04-09 21:24:44,353 WARN mortbay.log (Slf4jLog.java:warn(76)) - /getimage: java.io.IOException: GetImage failed. java.io.IOException: Inconsistent checkpoint fields.
        LV = -18 namespaceID = 2007855034 cTime = 0; checkpointTime = 1302380284708.
        Expecting respectively: -18; 2007855034; 0; 1302380284709
        at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:70)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.validateCheckpointUpload(FSImage.java:1732)
        at org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        Solution can be:
        When it failes to createEditLogFile, we can collect all the failed directories and can call processIOError with that storageDirectories list

        synchronized void startJournalSpool(NamenodeRegistration nnReg)
         throws IOException {
        ..................
        ...................
        
         ArrayList<StorageDirectory> errorSDs = new ArrayList<StorageDirectory>();
            // create journal spool directories
            for(Iterator<StorageDirectory> it =
                                  dirIterator(NameNodeDirType.EDITS); it.hasNext();) {
              ...................
             ...................     
              // create edit file if missing
              File eFile = getEditFile(sd);
              if(!eFile.exists())
              {
            	  try
            	  {
            		  editLog.createEditLogFile(eFile);
            	  }
            	  catch(IOException ie)
            	  {
                    LOG.error("Unable to save edits for " + sd.getRoot(), ie);
                    errorSDs.add(sd);
                  }
              }
            }
            processIOError(errorSDs, true);
        
        
        Show
        Uma Maheswara Rao G added a comment - Following is the trace. 2011-04-09 21:24:43,892 ERROR namenode.FSNamesystem (FSEditLog.java:logSync(1207)) - Unable to sync edit log. org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException: /hdfsdata/7/patch/name/data/current/edits (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) at org.apache.hadoop.hdfs.server.namenode.FSEditLog$EditLogFileOutputStream.<init>(FSEditLog.java:150) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createEditLogFile(FSEditLog.java:427) at org.apache.hadoop.hdfs.server.namenode.BackupStorage.startJournalSpool(BackupStorage.java:275) at org.apache.hadoop.hdfs.server.namenode.BackupNode.journal(BackupNode.java:327) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099) at org.apache.hadoop.ipc.Client.call(Client.java:953) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:239) at $Proxy4.journal(Unknown Source) at org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.send(EditLogBackupOutputStream.java:165) at org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.flushAndSync(EditLogBackupOutputStream.java:145) at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:83) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1204) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCheckpoint(FSNamesystem.java:5164) at org.apache.hadoop.hdfs.server.namenode.NameNode.startCheckpoint(NameNode.java:547) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099) 2011-04-09 21:24:43,893 ERROR namenode.FSNamesystem (FSEditLog.java:processIOError(557)) - Unable to log edits to 10.18.53.203:50100 2011-04-09 21:24:44,353 WARN mortbay.log (Slf4jLog.java:warn(76)) - /getimage: java.io.IOException: GetImage failed. java.io.IOException: Inconsistent checkpoint fields. LV = -18 namespaceID = 2007855034 cTime = 0; checkpointTime = 1302380284708. Expecting respectively: -18; 2007855034; 0; 1302380284709 at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:70) at org.apache.hadoop.hdfs.server.namenode.FSImage.validateCheckpointUpload(FSImage.java:1732) at org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) Solution can be: When it failes to createEditLogFile, we can collect all the failed directories and can call processIOError with that storageDirectories list synchronized void startJournalSpool(NamenodeRegistration nnReg) throws IOException { .................. ................... ArrayList<StorageDirectory> errorSDs = new ArrayList<StorageDirectory>(); // create journal spool directories for (Iterator<StorageDirectory> it = dirIterator(NameNodeDirType.EDITS); it.hasNext();) { ................... ................... // create edit file if missing File eFile = getEditFile(sd); if (!eFile.exists()) { try { editLog.createEditLogFile(eFile); } catch (IOException ie) { LOG.error( "Unable to save edits for " + sd.getRoot(), ie); errorSDs.add(sd); } } } processIOError(errorSDs, true );
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Suresh,
        Thanks for the feedback...Updated it.

        --Thanks

        Show
        Uma Maheswara Rao G added a comment - Hi Suresh, Thanks for the feedback...Updated it. --Thanks
        Hide
        Konstantin Shvachko added a comment -

        I am wondering what are you running? BackupNode was introduced when LV was -19, you seem to have -18. Can you reproduce it on say 0.22?

        Show
        Konstantin Shvachko added a comment - I am wondering what are you running? BackupNode was introduced when LV was -19, you seem to have -18. Can you reproduce it on say 0.22?
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        My code base is from 0.20.1 and back ported the BackupNode
        In 0.22 also same code is available. Code snippet i pasted above is from 0.22.
        Logs are from my production systems.(0.20.1 code base).

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, My code base is from 0.20.1 and back ported the BackupNode In 0.22 also same code is available. Code snippet i pasted above is from 0.22. Logs are from my production systems.(0.20.1 code base).
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Updated the patch for 22branch.

        Can you please take a look?

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Updated the patch for 22branch. Can you please take a look? Thanks Uma
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493359/HDFS-2271.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1220//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493359/HDFS-2271.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1220//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Konstantin, Can you please take a look?

        Show
        Uma Maheswara Rao G added a comment - Konstantin, Can you please take a look?
        Hide
        Uma Maheswara Rao G added a comment -

        Re-based the patch as per the latest changes.

        Show
        Uma Maheswara Rao G added a comment - Re-based the patch as per the latest changes.
        Hide
        Uma Maheswara Rao G added a comment -

        Test patch results:

        [exec] +1 @author. The patch does not contain any @author tags.

        [exec] +1 tests included. The patch appears to include 3 new or modified tests.

        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.

        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        [exec] +1 system test framework. The patch passed system test framework compile.

        ===============================================================
        Finised Build
        ===============================================================

        Show
        Uma Maheswara Rao G added a comment - Test patch results: [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] +1 system test framework. The patch passed system test framework compile. =============================================================== Finised Build ===============================================================
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494275/HDFS-2271.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1237//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494275/HDFS-2271.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1237//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        Good catch. One nit.
        In TestBackupNode.testStartJournalSpoolHandlesIOExceptions() please use BASE_DIR rather than writing to /tmp. That way we guarantee that clean target removes everything created on the previous test run.

        Show
        Konstantin Shvachko added a comment - Good catch. One nit. In TestBackupNode.testStartJournalSpoolHandlesIOExceptions() please use BASE_DIR rather than writing to /tmp. That way we guarantee that clean target removes everything created on the previous test run.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Konstantin,

        Thanks a lot for taking a look!

        Addressed the comment with this patch.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Konstantin, Thanks a lot for taking a look! Addressed the comment with this patch. Thanks Uma
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494356/HDFS-2271.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1238//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494356/HDFS-2271.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1238//console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Uma.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Uma.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #84 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/84/)
        HDFS-2271. startJournalSpool should invoke processIOError. Contributed by Uma Maheswara Rao G.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170925
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupStorage.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #84 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/84/ ) HDFS-2271 . startJournalSpool should invoke processIOError. Contributed by Uma Maheswara Rao G. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170925 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupStorage.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java
        Hide
        Arun C Murthy added a comment -

        Is this not required for 0.23/0.24 ?

        Show
        Arun C Murthy added a comment - Is this not required for 0.23/0.24 ?
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Arun,

        Thanks for taking a look!

        This Code & design has been refactored in trunk. ( as part of HDFS-1073).
        Most of the cases handled with HDFS-1073.
        I am checking the trunk code considering this scenarios, if i find such cases i will file the JIRAs.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Arun, Thanks for taking a look! This Code & design has been refactored in trunk. ( as part of HDFS-1073 ). Most of the cases handled with HDFS-1073 . I am checking the trunk code considering this scenarios, if i find such cases i will file the JIRAs. Thanks Uma

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development