Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13238

Missing EC Data block throws warn message with full stackTrace

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: erasure-coding, hdfs
    • Labels:
      None

      Description

      If an EC data block is missing/ corrupted, then the following warning message is thrown when client tries to read the file. 

      $ hdfs dfs -get /user/abc/file1
      2018-03-06 22:53:32,156 WARN impl.BlockReaderFactory: I/O error constructing remote block reader.
      java.io.IOException: Got error, status=ERROR, status message opReadBlock BP-1641043599-127.0.0.1-1520368608283:blk_-9223372036854775776_1002 received exception java.io.FileNotFoundException: BlockId -9223372036854775776 is not valid., for OP_READ_BLOCK, self=/127.0.0.1.0.2:60502, remote=/127.0.0.1:9866, for file /user/abc/file1, for pool BP-1641043599-127.0.0.1-1520368608283 block -9223372036854775776_1002
      at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)
      at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.checkSuccess(BlockReaderRemote.java:447)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:415)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:860)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:756)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:390)
      at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.createBlockReader(DFSStripedInputStream.java:256)
      at org.apache.hadoop.hdfs.StripeReader.readChunk(StripeReader.java:293)
      at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:323)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:318)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:815)
      at java.io.DataInputStream.read(DataInputStream.java:100)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
      at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
      at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
      at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
      at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
      at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
      at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
      2018-03-06 22:53:32,167 WARN hdfs.DFSClient: Failed to connect to /127.0.0.1:9866 for blockBP-1641043599-127.0.0.1-1520368608283:blk_-9223372036854775776_1002
      java.io.IOException: Got error, status=ERROR, status message opReadBlock BP-1641043599-127.0.0.1-1520368608283:blk_-9223372036854775776_1002 received exception java.io.FileNotFoundException: BlockId -9223372036854775776 is not valid., for OP_READ_BLOCK, self=/127.0.0.1:60502, remote=/127.0.0.1:9866, for file /user/abc/file1, for pool BP-1641043599-127.0.0.1-1520368608283 block -9223372036854775776_1002
      at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)
      at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.checkSuccess(BlockReaderRemote.java:447)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:415)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:860)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:756)
      at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:390)
      at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.createBlockReader(DFSStripedInputStream.java:256)
      at org.apache.hadoop.hdfs.StripeReader.readChunk(StripeReader.java:293)
      at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:323)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:318)
      at org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
      at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:815)
      at java.io.DataInputStream.read(DataInputStream.java:100)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
      at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
      at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
      at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
      at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
      at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
      at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
      at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
      at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
      at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
      at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
      2018-03-06 22:53:32,182 WARN hdfs.DFSClient: [DatanodeInfoWithStorage[127.0.0.1:9866,DS-133ffa99-2ddb-4er4-9v8c-e14f3a432b8e,DISK]] are unavailable and all striping blocks on them are lost. IgnoredNodes = null

      It would be nice if this full stack trace was not thrown to the client. We can instead put out a warning message that a data block is missing and will be reconstructed (or as the case may be).

        Attachments

          Activity

            People

            • Assignee:
              bharatviswa Bharat Viswanadham
              Reporter:
              hanishakoneru Hanisha Koneru
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: