Hadoop Common
  1. Hadoop Common
  2. HADOOP-8136

Enhance hadoop to use a newer version (0.8.1) of the jets3t library

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 1.0.0, 0.22.0, 0.23.3
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      Ubuntu 11.04, 64 bit, JDK 1.6.0_30

    • Target Version/s:

      Description

      Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
      The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.

      Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.

      It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.

        Issue Links

          Activity

          Hide
          Jagane Sundar added a comment -

          Here's my first cut at a patch for this jira. Some notes:
          1. Most of the changes are on account of the jets3t-0.8.1 library throwing a ServiceException, instead of a S3ServiceException.
          2. There is also a piece of brittle code that was comparing an exception string with 'ResponseCode=404' to decide whether the S3 server threw a 404 error. I replaced that with a more rational test of response code.
          3. I turned down the logging by setting
          org.apache.hadoop.fs.s3native=ERROR
          org.jets3t.service.impl.rest.httpclient.RestStorageService=ERROR
          in log4j.properties

          Show
          Jagane Sundar added a comment - Here's my first cut at a patch for this jira. Some notes: 1. Most of the changes are on account of the jets3t-0.8.1 library throwing a ServiceException, instead of a S3ServiceException. 2. There is also a piece of brittle code that was comparing an exception string with 'ResponseCode=404' to decide whether the S3 server threw a 404 error. I replaced that with a more rational test of response code. 3. I turned down the logging by setting org.apache.hadoop.fs.s3native=ERROR org.jets3t.service.impl.rest.httpclient.RestStorageService=ERROR in log4j.properties
          Hide
          Jagane Sundar added a comment -

          Apply the patch HADOOP-5836-for_branch_1_0.patch before applying this patch.

          Show
          Jagane Sundar added a comment - Apply the patch HADOOP-5836 -for_branch_1_0.patch before applying this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517196/HADOOP-8136-0-for_branch_1_0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517196/HADOOP-8136-0-for_branch_1_0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Jagane, can you attach a patch based on trunk, so Jenkins can apply the patch and test it?

          Show
          Suresh Srinivas added a comment - Jagane, can you attach a patch based on trunk, so Jenkins can apply the patch and test it?
          Hide
          Matt Foley added a comment -

          Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks.

          Show
          Matt Foley added a comment - Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks.
          Hide
          Jagane Sundar added a comment - - edited

          I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today.

          Jagane

          Show
          Jagane Sundar added a comment - - edited I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today. Jagane
          Hide
          Matt Foley added a comment -

          Not resolved in time for 1.0.3. Please consider continuing the contribution in 1.1.0. Thank you.

          Show
          Matt Foley added a comment - Not resolved in time for 1.0.3. Please consider continuing the contribution in 1.1.0. Thank you.
          Hide
          Matt Foley added a comment -

          Moved to 1.2.0 upon release of 1.1.0.

          Show
          Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
          Hide
          Steve Loughran added a comment -

          @Jagane -can you provide the patch for this against the current trunk? I also think if we can do a simple version increment without doing any major rewrites, that would be more likely to get into branch-1.

          Show
          Steve Loughran added a comment - @Jagane -can you provide the patch for this against the current trunk? I also think if we can do a simple version increment without doing any major rewrites, that would be more likely to get into branch-1.
          Hide
          Leif Jackson added a comment -

          The most recent release of jets3t library 0.9.0 now supports S3 server side encryption @Jagane if you are still working on this please update and use 0.9.0. Thanks.

          Show
          Leif Jackson added a comment - The most recent release of jets3t library 0.9.0 now supports S3 server side encryption @Jagane if you are still working on this please update and use 0.9.0. Thanks.
          Hide
          Jordan Mendelson added a comment -

          HADOOP-9454 includes a patch for 0.9.0 against the current trunk plus support for multipart uploads.

          Show
          Jordan Mendelson added a comment - HADOOP-9454 includes a patch for 0.9.0 against the current trunk plus support for multipart uploads.
          Hide
          Matt Foley added a comment -

          Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

          Show
          Matt Foley added a comment - Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.
          Hide
          Steve Loughran added a comment -

          Superceded by the HADOOP-9623 patch to move up to 0.9.0

          Show
          Steve Loughran added a comment - Superceded by the HADOOP-9623 patch to move up to 0.9.0

            People

            • Assignee:
              Unassigned
              Reporter:
              Jagane Sundar
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development