Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5814

NativeS3FileSystem doesn't report progress when writing

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
    • Tags:
      hadoop common, S3Native

      Description

      This results in timeouts since the whole file is uploaded in the close method. See http://www.mail-archive.com/core-user@hadoop.apache.org/msg09881.html.

      One solution is to keep a reference to the Progressable passed in to the NativeS3FsOutputStream's constructor, and progress it during writes, and while copying the backup file to S3 in the close method.

      1. HADOOP-5814.patch
        14 kB
        Subroto Sanyal
      2. HADOOP-5814(1).patch
        18 kB
        Subroto Sanyal

        Activity

        Hide
        aw Allen Wittenauer added a comment -

        I suspect this is stale. If not, please open a new jira. Thanks!

        Show
        aw Allen Wittenauer added a comment - I suspect this is stale. If not, please open a new jira. Thanks!
        Hide
        j_caplan Joshua Caplan added a comment -

        Doesn't MAPREDUCE-1905 render this patch useless?

        Show
        j_caplan Joshua Caplan added a comment - Doesn't MAPREDUCE-1905 render this patch useless?
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console This message is automatically generated.
        Hide
        subrotosanyal Subroto Sanyal added a comment -

        The patch is on the trunk version.
        The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.

        Show
        subrotosanyal Subroto Sanyal added a comment - The patch is on the trunk version. The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.

          People

          • Assignee:
            Unassigned
            Reporter:
            tomwhite Tom White
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development