Hadoop Common
  1. Hadoop Common
  2. HADOOP-5814

NativeS3FileSystem doesn't report progress when writing

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
    • Tags:
      hadoop common, S3Native

      Description

      This results in timeouts since the whole file is uploaded in the close method. See http://www.mail-archive.com/core-user@hadoop.apache.org/msg09881.html.

      One solution is to keep a reference to the Progressable passed in to the NativeS3FsOutputStream's constructor, and progress it during writes, and while copying the backup file to S3 in the close method.

      1. HADOOP-5814(1).patch
        18 kB
        Subroto Sanyal
      2. HADOOP-5814.patch
        14 kB
        Subroto Sanyal

        Activity

        Hide
        Subroto Sanyal added a comment -

        The patch is on the trunk version.
        The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.

        Show
        Subroto Sanyal added a comment - The patch is on the trunk version. The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console This message is automatically generated.
        Hide
        Joshua Caplan added a comment -

        Doesn't MAPREDUCE-1905 render this patch useless?

        Show
        Joshua Caplan added a comment - Doesn't MAPREDUCE-1905 render this patch useless?

          People

          • Assignee:
            Unassigned
            Reporter:
            Tom White
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development