Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      Hit the following exception in TestPersistBlocks after running a few hundred iterations of the test in a loop.

      java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "Arpit-MB-Pro.local/192.168.0.103"; destination host is: "localhost":57470;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
              at org.apache.hadoop.ipc.Client.call(Client.java:1410)
              at org.apache.hadoop.ipc.Client.call(Client.java:1359)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
              at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:185)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
              at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:348)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1259)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1107)
              at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:515)
      Caused by: java.io.EOFException: null
              at java.io.DataInputStream.readInt(DataInputStream.java:375)
              at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1050)
              at org.apache.hadoop.ipc.Client$Connection.run(Client.java:945)
      

        Issue Links

          Activity

          Hide
          Kihwal Lee added a comment -

          Saw this in a precommit build.

          2016-08-12 23:40:13,900 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active
          2016-08-12 23:40:13,901 [IPC Server handler 7 on 57494] INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7028)) - allowed=true
          	ugi=jenkins (auth:SIMPLE)	ip=/127.0.0.1	cmd=getfileinfo	src=/data	dst=null	perm=null	proto=rpc
          2016-08-12 23:40:13,906 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster
          2016-08-12 23:40:13,906 [main] ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(579)) - Failed to close file: /data with inode: 16386
          java.io.EOFException: End of File Exception between local host is: "6fa8fece0684/172.17.0.3"; destination host is: "localhost":57494; : java.io.EOFException;
           For more details see:  http://wiki.apache.org/hadoop/EOFException
          	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:815)
          	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:779)
          	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499)
          	at org.apache.hadoop.ipc.Client.call(Client.java:1441)
          	at org.apache.hadoop.ipc.Client.call(Client.java:1351)
          	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
          	at com.sun.proxy.$Proxy19.addBlock(Unknown Source)
          

          Compared to a successful run, only 4 blocks made to the edit log instead of 5. Then after the NN restart, the test did getFileIno() and started shutting down the cluster without writing more data or closing the file. It might have failed the assert, but don't see it in the log. The above exception seems to be generated during this shutdown.

          Show
          Kihwal Lee added a comment - Saw this in a precommit build. 2016-08-12 23:40:13,900 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active 2016-08-12 23:40:13,901 [IPC Server handler 7 on 57494] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7028)) - allowed=true ugi=jenkins (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/data dst=null perm=null proto=rpc 2016-08-12 23:40:13,906 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster 2016-08-12 23:40:13,906 [main] ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(579)) - Failed to close file: /data with inode: 16386 java.io.EOFException: End of File Exception between local host is: "6fa8fece0684/172.17.0.3"; destination host is: "localhost":57494; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:815) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:779) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234) at com.sun.proxy.$Proxy19.addBlock(Unknown Source) Compared to a successful run, only 4 blocks made to the edit log instead of 5. Then after the NN restart, the test did getFileIno() and started shutting down the cluster without writing more data or closing the file. It might have failed the assert, but don't see it in the log. The above exception seems to be generated during this shutdown.

            People

            • Assignee:
              Unassigned
              Reporter:
              Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development