Hadoop Common
  1. Hadoop Common
  2. HADOOP-5814

NativeS3FileSystem doesn't report progress when writing

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
    • Tags:
      hadoop common, S3Native

      Description

      This results in timeouts since the whole file is uploaded in the close method. See http://www.mail-archive.com/core-user@hadoop.apache.org/msg09881.html.

      One solution is to keep a reference to the Progressable passed in to the NativeS3FsOutputStream's constructor, and progress it during writes, and while copying the backup file to S3 in the close method.

      1. HADOOP-5814(1).patch
        18 kB
        Subroto Sanyal
      2. HADOOP-5814.patch
        14 kB
        Subroto Sanyal

        Activity

        Tom White created issue -
        Devaraj K made changes -
        Field Original Value New Value
        Assignee Devaraj K [ devaraj.k ]
        Hide
        Subroto Sanyal added a comment -

        The patch is on the trunk version.
        The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.

        Show
        Subroto Sanyal added a comment - The patch is on the trunk version. The patch creates a input Stream which handles the Progressable reference and makes the call periodically while reading from the underlying stream. The patch is tested in our cluster and works fine.
        Subroto Sanyal made changes -
        Attachment HADOOP-5814.patch [ 12493897 ]
        Subroto Sanyal made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Labels S3Native
        Tags hadoop common, S3Native
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493897/HADOOP-5814.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/157//console This message is automatically generated.
        Subroto Sanyal made changes -
        Attachment HADOOP-5814(1).patch [ 12493903 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493903/HADOOP-5814%281%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/159//console This message is automatically generated.
        Devaraj K made changes -
        Assignee Devaraj K [ devaraj.k ]
        Subroto Sanyal made changes -
        Hadoop Flags [Reviewed]
        Subroto Sanyal made changes -
        Hadoop Flags [Reviewed]
        Hide
        Joshua Caplan added a comment -

        Doesn't MAPREDUCE-1905 render this patch useless?

        Show
        Joshua Caplan added a comment - Doesn't MAPREDUCE-1905 render this patch useless?
        Hide
        Allen Wittenauer added a comment -

        I suspect this is stale. If not, please open a new jira. Thanks!

        Show
        Allen Wittenauer added a comment - I suspect this is stale. If not, please open a new jira. Thanks!
        Allen Wittenauer made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Incomplete [ 4 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        850d 12h 54m 1 Subroto Sanyal 10/Sep/11 06:27
        Patch Available Patch Available Resolved Resolved
        1046d 14h 7m 1 Allen Wittenauer 22/Jul/14 20:34

          People

          • Assignee:
            Unassigned
            Reporter:
            Tom White
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development