Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 2.8.0, 3.0.0-alpha3
    • Component/s: fs/adl
    • Labels:
      None

      Description

      ADLS has multiple upgrades since the version 2.0.11 we are using: 2.1.1, 2.1.2, and 2.1.4. Change list: https://github.com/Azure/azure-data-lake-store-java/blob/master/CHANGES.md.

        Activity

        Hide
        stevel@apache.org Steve Loughran added a comment -

        does this change any dependencies?

        Show
        stevel@apache.org Steve Loughran added a comment - does this change any dependencies?
        Hide
        jzhuge John Zhuge added a comment -

        I don't think so. Atul Sikaria, could you comment?

        Should also wait for HADOOP-13927 fix which might require ADLS backend fix and SDK api-version bump.

        Show
        jzhuge John Zhuge added a comment - I don't think so. Atul Sikaria , could you comment? Should also wait for HADOOP-13927 fix which might require ADLS backend fix and SDK api-version bump.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        if HADOOP-13927 requires an update, the two could be merged together...this isn't a library which people are using yet, so we could avoid the usual "make this a standalone issue under HADOOP-9991", which is there to make a list of things that could break downstream code

        Show
        stevel@apache.org Steve Loughran added a comment - if HADOOP-13927 requires an update, the two could be merged together...this isn't a library which people are using yet, so we could avoid the usual "make this a standalone issue under HADOOP-9991 ", which is there to make a list of things that could break downstream code
        Hide
        jzhuge John Zhuge added a comment -

        Microsoft confirms HADOOP-13927 is a backend bug and its fix will not need SDK update.

        Show
        jzhuge John Zhuge added a comment - Microsoft confirms HADOOP-13927 is a backend bug and its fix will not need SDK update.
        Hide
        jzhuge John Zhuge added a comment -

        Patch 001

        • Update ADLS SDK to version 2.1.4

        Testing done

        • Live unit tests
        Show
        jzhuge John Zhuge added a comment - Patch 001 Update ADLS SDK to version 2.1.4 Testing done Live unit tests
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 31s trunk passed
        +1 compile 0m 16s trunk passed
        +1 mvnsite 0m 18s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        +1 mvnsite 0m 15s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 10s the patch passed
        +1 unit 3m 35s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        21m 31s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13962
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846748/HADOOP-13962.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 7ac534d9833e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 467f5f1
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11415/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11415/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 31s trunk passed +1 compile 0m 16s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 14s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 10s the patch passed +1 unit 3m 35s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 21m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13962 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846748/HADOOP-13962.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 7ac534d9833e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 467f5f1 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11415/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11415/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jzhuge John Zhuge added a comment -

        Patch 001 also passed basic dfs -ls/-put/-cat and teragen/terasort/teravalidate.

        Show
        jzhuge John Zhuge added a comment - Patch 001 also passed basic dfs -ls/-put/-cat and teragen/terasort/teravalidate.
        Hide
        ASikaria Atul Sikaria added a comment -

        Looks good to me, and my own tests seem to work with this change.

        + Vishwajeet Dusane, to confirm his tests are passing.

        Show
        ASikaria Atul Sikaria added a comment - Looks good to me, and my own tests seem to work with this change. + Vishwajeet Dusane , to confirm his tests are passing.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        +1 for the patch - I ran live tests with the patch and test are passing with this change except for the 1 test, known issue HADOOP-13927.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - +1 for the patch - I ran live tests with the patch and test are passing with this change except for the 1 test, known issue HADOOP-13927 .
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        +1. LGTM.

        Committed to trunk, thanks John Zhuge.

        Show
        eddyxu Lei (Eddy) Xu added a comment - +1. LGTM. Committed to trunk, thanks John Zhuge .
        Hide
        jzhuge John Zhuge added a comment -

        Thanks Lei (Eddy) Xu for the commit. Thanks Steve Loughran, Atul Sikaria, and Vishwajeet Dusane for the review.

        Show
        jzhuge John Zhuge added a comment - Thanks Lei (Eddy) Xu for the commit. Thanks Steve Loughran , Atul Sikaria , and Vishwajeet Dusane for the review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11156 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11156/)
        HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) (lei: rev ccf2d662443fc169835d66b715441529658b245c)

        • (edit) hadoop-tools/hadoop-azure-datalake/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11156 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11156/ ) HADOOP-13962 . Update ADLS SDK to 2.1.4. (John Zhuge via lei) (lei: rev ccf2d662443fc169835d66b715441529658b245c) (edit) hadoop-tools/hadoop-azure-datalake/pom.xml

          People

          • Assignee:
            jzhuge John Zhuge
            Reporter:
            jzhuge John Zhuge
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development