Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15446

CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path

    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After allowing snapshot creation for a path say /app-logs , when we try to create snapshot on
      /.reserved/raw/app-logs , its successful with snapshot creation but later when Standby Namenode is restarted and tries to load the edit record OP_CREATE_SNAPSHOT , we see it failing and Standby Namenode shuts down with an exception "ava.io.FileNotFoundException: Directory does not exist: /.reserved/raw/app-logs" .

      Here are the steps to reproduce :

      # hdfs dfs -ls /.reserved/raw/
      Found 15 items
      drwxrwxrwt   - yarn   hadoop          0 2020-06-29 10:27 /.reserved/raw/app-logs
      drwxr-xr-x   - hive   hadoop          0 2020-06-29 10:29 /.reserved/raw/prod
      ++++++++++++++
      [root@c3230-node2 ~]# hdfs dfsadmin -allowSnapshot /app-logs
      Allowing snapshot on /app-logs succeeded
      [root@c3230-node2 ~]# hdfs dfsadmin -allowSnapshot /prod
      Allowing snapshot on /prod succeeded
      ++++++++++++++
      # hdfs lsSnapshottableDir
      drwxrwxrwt 0 yarn hadoop 0 2020-06-29 10:27 1 65536 /app-logs
      drwxr-xr-x 0 hive hadoop 0 2020-06-29 10:29 1 65536 /prod
      ++++++++++++++
      [root@c3230-node2 ~]# hdfs dfs -createSnapshot /.reserved/raw/app-logs testSS
      Created snapshot /.reserved/raw/app-logs/.snapshot/testSS
      

      Exception we see in Standby namenode while loading the snapshot creation edit record.

      2020-06-29 10:33:25,488 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode.
      java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/app-logs
              at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.valueOf(INodeDirectory.java:60)
              at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.getSnapshottableRoot(SnapshotManager.java:259)
              at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:307)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:772)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:257)
      

        Attachments

        1. HDFS-15446.003.patch
          7 kB
          Stephen O'Donnell
        2. HDFS-15446.002.patch
          5 kB
          Stephen O'Donnell
        3. HDFS-15446.001.patch
          6 kB
          Stephen O'Donnell

          Activity

            People

            • Assignee:
              sodonnell Stephen O'Donnell
              Reporter:
              smajeti Srinivasu Majeti
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: