Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3332

NullPointerException in DN when directoryscanner is trying to report bad blocks

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.0.0-alpha, 3.0.0
    • Component/s: datanode
    • Labels:
      None
    • Environment:

      HDFS

    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      There is 1 NN and 1 DN (NN is started with HA conf)
      I corrupted 1 block and found

      2012-04-27 09:59:01,214 INFO  datanode.DataNode (BPServiceActor.java:blockReport(401)) - BlockReport of 2 blocks took 0 msec to generate and 5 msecs for RPC and NN processing
      2012-04-27 09:59:01,214 INFO  datanode.DataNode (BPServiceActor.java:blockReport(420)) - sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@3b756db3
      2012-04-27 09:59:01,726 INFO  datanode.DirectoryScanner (DirectoryScanner.java:scan(390)) - BlockPool BP-2087868617-10.18.40.95-1335500488012 Total blocks: 2, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:1
      2012-04-27 09:59:01,727 WARN  impl.FsDatasetImpl (FsDatasetImpl.java:checkAndUpdate(1366)) - Updating size of block -4466699320171028643 from 1024 to 1034
      2012-04-27 09:59:01,727 WARN  impl.FsDatasetImpl (FsDatasetImpl.java:checkAndUpdate(1374)) - Reporting the block blk_-4466699320171028643_1004 as corrupt due to length mismatch
      2012-04-27 09:59:01,728 DEBUG ipc.Client (Client.java:sendParam(807)) - IPC Client (1957050620) connection to /10.18.40.95:8020 from root sending #257
      2012-04-27 09:59:01,730 DEBUG ipc.Client (Client.java:receiveResponse(848)) - IPC Client (1957050620) connection to /10.18.40.95:8020 from root got value #257
      2012-04-27 09:59:01,730 DEBUG ipc.ProtobufRpcEngine (ProtobufRpcEngine.java:invoke(193)) - Call: reportBadBlocks 2
      2012-04-27 09:59:01,731 ERROR datanode.DirectoryScanner (DirectoryScanner.java:run(288)) - Exception during DirectoryScanner execution - will continue next cycle
      java.lang.NullPointerException
      	at org.apache.hadoop.hdfs.protocol.DatanodeID.<init>(DatanodeID.java:66)
      	at org.apache.hadoop.hdfs.protocol.DatanodeInfo.<init>(DatanodeInfo.java:87)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportBadBlocks(BPServiceActor.java:238)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.reportBadBlocks(BPOfferService.java:187)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.reportBadBlocks(DataNode.java:559)
      	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(FsDatasetImpl.java:1377)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:318)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:284)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      

      Here when Directory scanner is trying to report badblock we got a NPE.

        Activity

        amith created issue -
        amith made changes -
        Field Original Value New Value
        Attachment HDFS-3332.patch [ 12525315 ]
        amith made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Uma Maheswara Rao G made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.0 [ 12320353 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            amith
            Reporter:
            amith
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development