Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9781

FsDatasetImpl#getBlockReports can occasionally throw NullPointerException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.8.0, 3.0.0-alpha2
    • datanode
    • None
    • Jenkins

    • Reviewed

    Description

      FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused TestFsDatasetImpl#testRemoveVolumeBeingWritten to time out, because the test waits for the call to FsDatasetImpl#getBlockReports to complete without exceptions.

      Additionally, the test should be updated to identify an expected exception, using GenericTestUtils.assertExceptionContains()

      Exception in thread "Thread-20" java.lang.NullPointerException
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
      2016-02-08 15:47:30,379 [Thread-21] WARN  impl.TestFsDatasetImpl (TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect the test
      java.io.IOException: Failed to move meta file for ReplicaBeingWritten, blk_0_0, RBW
        getNumBytes()     = 0
        getBytesOnDisk()  = 0
        getVisibleLength()= 0
        getVolume()       = /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
        getBlockFile()    = /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
        bytesAcked=0
        bytesOnDisk=0 from /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta to /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:857)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addFinalizedBlock(BlockPoolSlice.java:295)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:819)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1620)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1601)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1ResponderThread.run(TestFsDatasetImpl.java:603)
      Caused by: java.io.IOException: renameTo(src=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta, dst=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta) failed.
              at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:873)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:855)
              ... 5 more
      2016-02-08 15:47:34,381 [Thread-19] INFO  impl.FsDatasetImpl (FsVolumeList.java:waitVolumeRemoved(287)) - Volume reference is released.
      2016-02-08 15:47:34,384 [Thread-19] INFO  impl.TestFsDatasetImpl (TestFsDatasetImpl.java:testRemoveVolumeBeingWritten(622)) - Volumes removed
      

      Attachments

        1. HDFS-9781.002.patch
          5 kB
          Manoj Govindassamy
        2. HDFS-9781.003.patch
          6 kB
          Manoj Govindassamy
        3. HDFS-9781.004.patch
          5 kB
          Manoj Govindassamy
        4. HDFS-9781.01.patch
          3 kB
          Xiao Chen
        5. HDFS-9781.branch-2.001.patch
          5 kB
          Xiao Chen
        6. HDFS-9781-branch-2.001.patch
          5 kB
          Manoj Govindassamy
        7. HDFS-9781-branch-2.002.patch
          7 kB
          Manoj Govindassamy
        8. HDFS-9781-branch-2.003.patch
          7 kB
          Manoj Govindassamy
        9. HDFS-9781-branch-2.004.patch
          7 kB
          Manoj Govindassamy

        Issue Links

          Activity

            People

              manojg Manoj Govindassamy
              weichiu Wei-Chiu Chuang
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: