Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9781

FsDatasetImpl#getBlockReports can occasionally throw NullPointerException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Environment:

      Jenkins

    • Hadoop Flags:
      Reviewed

      Description

      FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused TestFsDatasetImpl#testRemoveVolumeBeingWritten to time out, because the test waits for the call to FsDatasetImpl#getBlockReports to complete without exceptions.

      Additionally, the test should be updated to identify an expected exception, using GenericTestUtils.assertExceptionContains()

      Exception in thread "Thread-20" java.lang.NullPointerException
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
      2016-02-08 15:47:30,379 [Thread-21] WARN  impl.TestFsDatasetImpl (TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect the test
      java.io.IOException: Failed to move meta file for ReplicaBeingWritten, blk_0_0, RBW
        getNumBytes()     = 0
        getBytesOnDisk()  = 0
        getVisibleLength()= 0
        getVolume()       = /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
        getBlockFile()    = /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
        bytesAcked=0
        bytesOnDisk=0 from /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta to /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:857)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addFinalizedBlock(BlockPoolSlice.java:295)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:819)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1620)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1601)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1ResponderThread.run(TestFsDatasetImpl.java:603)
      Caused by: java.io.IOException: renameTo(src=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta, dst=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta) failed.
              at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:873)
              at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:855)
              ... 5 more
      2016-02-08 15:47:34,381 [Thread-19] INFO  impl.FsDatasetImpl (FsVolumeList.java:waitVolumeRemoved(287)) - Volume reference is released.
      2016-02-08 15:47:34,384 [Thread-19] INFO  impl.TestFsDatasetImpl (TestFsDatasetImpl.java:testRemoveVolumeBeingWritten(622)) - Volumes removed
      

        Attachments

        1. HDFS-9781.002.patch
          5 kB
          Manoj Govindassamy
        2. HDFS-9781.003.patch
          6 kB
          Manoj Govindassamy
        3. HDFS-9781.004.patch
          5 kB
          Manoj Govindassamy
        4. HDFS-9781.01.patch
          3 kB
          Xiao Chen
        5. HDFS-9781.branch-2.001.patch
          5 kB
          Xiao Chen
        6. HDFS-9781-branch-2.001.patch
          5 kB
          Manoj Govindassamy
        7. HDFS-9781-branch-2.002.patch
          7 kB
          Manoj Govindassamy
        8. HDFS-9781-branch-2.003.patch
          7 kB
          Manoj Govindassamy
        9. HDFS-9781-branch-2.004.patch
          7 kB
          Manoj Govindassamy

          Issue Links

            Activity

              People

              • Assignee:
                manojg Manoj Govindassamy
                Reporter:
                jojochuang Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: