Hadoop Common
  1. Hadoop Common
  2. HADOOP-9410

S3 filesystem hangs on FileSystem.listFiles()

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
    • Environment:

      Talking to S3 West Coast and S3 EU

      Description

      A test in HADOOP-9258 of the Hadoop 2+ API call FileSystem.listFiles() hangs repeatedly when using the s3:// filesystem

      1. HADOOP-9410.patch
        2 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          Stack trace from Tom White

          main" prio=5 tid=105000000 nid=0x100601000 runnable [1005fd000]
             java.lang.Thread.State: RUNNABLE
          	at java.net.SocketInputStream.socketRead0(Native Method)
          	at java.net.SocketInputStream.read(SocketInputStream.java:129)
          	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
          	at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
          	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
          	- locked <7bb659288> (a java.lang.Object)
          	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
          	at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
          	- locked <7bb659338> (a com.sun.net.ssl.internal.ssl.AppInputStream)
          	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
          	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
          	- locked <7bb66ad28> (a java.io.BufferedInputStream)
          	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
          	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
          	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
          	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
          	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
          	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
          	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
          	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:357)
          	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
          	at org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083)
          	at org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsImpl(RestS3Service.java:1046)
          	at org.jets3t.service.S3Service.listObjects(S3Service.java:1299)
          	at org.jets3t.service.S3Service.listObjects(S3Service.java:1271)
          	at org.jets3t.service.S3Service.listObjects(S3Service.java:1137)
          	at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.listSubPaths(Jets3tFileSystemStore.java:279)
          	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:202)
          	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1430)
          	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1470)
          	at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1745)
          	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1744)
          	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1727)
          	at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:1820)
          	at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:1797)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.assertListFilesFinds(FileSystemContractBaseTest.java:719)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.testLSRootDir(FileSystemContractBaseTest.java:704)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at junit.framework.TestCase.runTest(TestCase.java:168)
          	at junit.framework.TestCase.runBare(TestCase.java:134)
          	at junit.framework.TestResult$1.protect(TestResult.java:110)
          	at junit.framework.TestResult.runProtected(TestResult.java:128)
          	at junit.framework.TestResult.run(TestResult.java:113)
          	at junit.framework.TestCase.run(TestCase.java:124)
          	at junit.framework.TestSuite.runTest(TestSuite.java:243)
          	at junit.framework.TestSuite.run(TestSuite.java:238)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
          	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
          
          Show
          Steve Loughran added a comment - Stack trace from Tom White main" prio=5 tid=105000000 nid=0x100601000 runnable [1005fd000] java.lang. Thread .State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863) - locked <7bb659288> (a java.lang. Object ) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820) at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) - locked <7bb659338> (a com.sun.net.ssl.internal.ssl.AppInputStream) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked <7bb66ad28> (a java.io.BufferedInputStream) at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.jets3t.service.impl. rest .httpclient.RestS3Service.performRequest(RestS3Service.java:357) at org.jets3t.service.impl. rest .httpclient.RestS3Service.performRestGet(RestS3Service.java:686) at org.jets3t.service.impl. rest .httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083) at org.jets3t.service.impl. rest .httpclient.RestS3Service.listObjectsImpl(RestS3Service.java:1046) at org.jets3t.service.S3Service.listObjects(S3Service.java:1299) at org.jets3t.service.S3Service.listObjects(S3Service.java:1271) at org.jets3t.service.S3Service.listObjects(S3Service.java:1137) at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.listSubPaths(Jets3tFileSystemStore.java:279) at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:202) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1430) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1470) at org.apache.hadoop.fs.FileSystem$4.<init>(FileSystem.java:1745) at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1744) at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1727) at org.apache.hadoop.fs.FileSystem$5.handleFileStat(FileSystem.java:1820) at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:1797) at org.apache.hadoop.fs.FileSystemContractBaseTest.assertListFilesFinds(FileSystemContractBaseTest.java:719) at org.apache.hadoop.fs.FileSystemContractBaseTest.testLSRootDir(FileSystemContractBaseTest.java:704) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
          Hide
          Steve Loughran added a comment -

          This is the test which exhibits the problem. It does not fix it.

          Show
          Steve Loughran added a comment - This is the test which exhibits the problem. It does not fix it.
          Hide
          Steve Loughran added a comment -

          pushing test-only-patch through to verify that it doesn't break anything else

          Show
          Steve Loughran added a comment - pushing test-only-patch through to verify that it doesn't break anything else
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2334//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2334//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2334//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2334//console This message is automatically generated.
          Hide
          Andrei Savu added a comment -

          Could this be generated by some sort of connection pool starvation? Similar to HADOOP-10203.

          Show
          Andrei Savu added a comment - Could this be generated by some sort of connection pool starvation? Similar to HADOOP-10203 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3483//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3483//console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          Good question: we should rerun the tests now and see if it is still there.

          Show
          Steve Loughran added a comment - Good question: we should rerun the tests now and see if it is still there.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6321/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6321/console This message was automatically generated.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6342/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12574072/HADOOP-9410.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6342/console This message was automatically generated.

            People

            • Assignee:
              Unassigned
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development