VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The directory scanner periodically compiles a report of differences between the datanode's on-disk and in-memory state.

      The code to generate the reports is in DirectoryScanner#scan and DirectoryScanner#getDiskReport. It looks like the volume field in ScanInfo is not correctly initialized while compiling the diffs. This was not an issue before but now we depend on the volume information being present. The bug triggers the following NPE during a scan if a block is present in the Datanode's in-memory block map but missing on disk:

      java.lang.NullPointerException
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1404)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:416)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:365)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:695)
      

      Another NPE exposed in BlockManager#reportDiff

      java.lang.NullPointerException: null
              at java.util.TreeMap.getEntry(TreeMap.java:324)
              at java.util.TreeMap.remove(TreeMap.java:580)
              at java.util.TreeSet.remove(TreeSet.java:259)
              at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1836)
              at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1709)
              at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1637)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:978)
              at org.apache.hadoop.
      

        Attachments

        1. h5412.02.patch
          16 kB
          Arpit Agarwal
        2. h5412.03.patch
          22 kB
          Arpit Agarwal

          Activity

            People

            • Assignee:
              arp Arpit Agarwal
              Reporter:
              arp Arpit Agarwal

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment