Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13903

write data into hdfs is very slow while the hdfs has mounted to local file system via nfs3

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.1
    • None
    • hdfs
    • None

    Description

      I have installed the hadoop 3.1.1, and mount hdfs to local filesystem via command of "mount -t nfs -o vers=3,nolock,rw,async,wsize=32768,rsize=32768 hdfsserver:/ /mydir"

       

      When I use use cp command to copy files to the mounted dir(/mydire). The speed is very slow. And some exception occurred as the follow:

       

      2018-09-07 11:04:51,952 INFO security.ShellBasedIdMapping: Update cache now

      2018-09-07 11:08:48,305 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:08:48,806 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:08:48,816 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:08:48,829 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:08:49,001 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:10:24,975 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:11:19,747 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:11:19,756 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:11:19,768 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:17,220 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:17,233 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:35,109 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:36,398 ERROR nfs3.RpcProgramNfs3: Setting file size is not supported when setattr, fileId: 27204

      2018-09-07 11:12:44,424 ERROR nfs3.RpcProgramNfs3: Setting file size is not supported when setattr, fileId: 27204

      2018-09-07 11:12:53,382 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:53,394 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:12:59,658 ERROR nfs3.RpcProgramNfs3: Setting file size is not supported when setattr, fileId: 27204

      2018-09-07 11:13:17,519 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:17,533 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:18,602 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:18,613 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:18,933 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:42,596 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:13:42,608 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:18:30,308 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:19:23,438 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:19:23,446 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:19:23,666 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:19:31,833 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:19:51,967 INFO security.ShellBasedIdMapping: Update cache now

      2018-09-07 11:20:02,884 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751181_10452

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:03,007 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751182_10453

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:03,377 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751185_10456

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:03,513 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751186_10457

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:03,704 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751188_10459

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:04,060 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751191_10462

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:08,213 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751206_10477

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      2018-09-07 11:20:31,866 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:21:32,239 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:22:32,245 INFO security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

      2018-09-07 11:22:35,394 WARN hdfs.DataStreamer: Exception for BP-1952000504-10.56.233.182-1536129789677:blk_1073751712_10983

      java.io.EOFException: Unexpected EOF while trying to read response from server

              at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:549)

              at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)

              at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)

      Attachments

        Activity

          People

            Unassigned Unassigned
            Chunbo Liao Chunbo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: