Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13900

Remove snapshot version of SDK dependency from Azure Data Lake Store File System

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/adl
    • Labels:
      None

      Description

      Azure Data Lake Store File System dependent Azure Data Lake Store SDK is released and has not need for further snapshot version dependency. This JIRA removes the SDK snapshot dependency to released SDK candidate. There is not functional change in the SDK and no impact to live contract test.

      1. HDFS-11240-001.patch
        1 kB
        Vishwajeet Dusane
      2. HDFS-11240-002.patch
        0.9 kB
        Vishwajeet Dusane

        Activity

        Hide
        liuml07 Mingliang Liu added a comment -

        https://mvnrepository.com/artifact/com.microsoft.azure/azure-data-lake-store-sdk

        Is the 2.0.11 the one we should be picking up? in version numbering terms, it should be newer. Thanks,

        Show
        liuml07 Mingliang Liu added a comment - https://mvnrepository.com/artifact/com.microsoft.azure/azure-data-lake-store-sdk Is the 2.0.11 the one we should be picking up? in version numbering terms, it should be newer. Thanks,
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Mingliang Liu - Yes that is the version we should be picking up. Thanks for the quick response. I have updated the patch with change in pom.xml

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Mingliang Liu - Yes that is the version we should be picking up. Thanks for the quick response. I have updated the patch with change in pom.xml
        Hide
        liuml07 Mingliang Liu added a comment -

        I think we can simply remove the repositories related code that is commented out. Maven repository and its mirrors have those artifact. Ping Steve Loughran for 2nd opinion.

        Other than that, +1 pending on 1) Jenkins 2) integration tests results.

        Can you also run the integration tests and post comment here about the results? Thanks,

        Show
        liuml07 Mingliang Liu added a comment - I think we can simply remove the repositories related code that is commented out. Maven repository and its mirrors have those artifact. Ping Steve Loughran for 2nd opinion. Other than that, +1 pending on 1) Jenkins 2) integration tests results. Can you also run the integration tests and post comment here about the results? Thanks,
        Hide
        liuml07 Mingliang Liu added a comment -

        Moved this to the HADOOP Common project.

        Show
        liuml07 Mingliang Liu added a comment - Moved this to the HADOOP Common project.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 45s trunk passed
        +1 compile 0m 16s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 javadoc 0m 14s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 13s the patch passed
        +1 javac 0m 13s the patch passed
        +1 mvnsite 0m 15s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 10s the patch passed
        +1 unit 3m 31s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        14m 42s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11240
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843032/HDFS-11240-001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux ee45fa5c22e6 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0b033e
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17849/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17849/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 45s trunk passed +1 compile 0m 16s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 16s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 10s the patch passed +1 unit 3m 31s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 14m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11240 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843032/HDFS-11240-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux ee45fa5c22e6 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0b033e Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17849/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17849/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Thanks you Mingliang Liu for moving this JIRA to appropriately. I will take care of this going forward.

        I reran the integration test with 2.0.11 SDK version and all test (722 integration test) are passing. repositories related code is removed, unfortunatly patch i posted had the commented code. Raising 2nd iteration with removed repositories.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks you Mingliang Liu for moving this JIRA to appropriately. I will take care of this going forward. I reran the integration test with 2.0.11 SDK version and all test (722 integration test) are passing. repositories related code is removed, unfortunatly patch i posted had the commented code. Raising 2nd iteration with removed repositories .
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Removed repositories commented code from the pom.xml.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Removed repositories commented code from the pom.xml .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 14s trunk passed
        +1 mvnsite 0m 17s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 13s the patch passed
        +1 compile 0m 11s the patch passed
        +1 javac 0m 11s the patch passed
        +1 mvnsite 0m 15s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 9s the patch passed
        +1 unit 3m 31s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        13m 21s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13900
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843032/HDFS-11240-001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 9ec2a57c8747 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0b033e
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11266/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11266/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 45s trunk passed +1 compile 0m 14s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 12s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 11s the patch passed +1 javac 0m 11s the patch passed +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 9s the patch passed +1 unit 3m 31s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 13m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13900 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843032/HDFS-11240-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 9ec2a57c8747 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0b033e Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11266/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11266/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 49s trunk passed
        +1 compile 0m 15s trunk passed
        +1 mvnsite 0m 17s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 javadoc 0m 11s trunk passed
        +1 mvninstall 0m 15s the patch passed
        +1 compile 0m 11s the patch passed
        +1 javac 0m 11s the patch passed
        +1 mvnsite 0m 15s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 8s the patch passed
        +1 unit 3m 27s hadoop-azure-datalake in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        13m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13900
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843037/HDFS-11240-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux d1579ce75e99 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0b033e
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11267/testReport/
        modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11267/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 49s trunk passed +1 compile 0m 15s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 11s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 11s the patch passed +1 javac 0m 11s the patch passed +1 mvnsite 0m 15s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 8s the patch passed +1 unit 3m 27s hadoop-azure-datalake in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13900 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843037/HDFS-11240-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux d1579ce75e99 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0b033e Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11267/testReport/ modules C: hadoop-tools/hadoop-azure-datalake U: hadoop-tools/hadoop-azure-datalake Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11267/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1.

        we should never have any snapshot dependencies in the code, hence, no repos pulling them in either.

        Show
        stevel@apache.org Steve Loughran added a comment - +1. we should never have any snapshot dependencies in the code, hence, no repos pulling them in either.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        Sorry Steve Loughran about snapshot dependency, earlier the SDK version was not available in public maven repository hence we had to add snapshot dependency. Also thanks for the +1.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - Sorry Steve Loughran about snapshot dependency, earlier the SDK version was not available in public maven repository hence we had to add snapshot dependency. Also thanks for the +1.
        Hide
        liuml07 Mingliang Liu added a comment -

        Committed to trunk branch. Thanks Vishwajeet Dusane for contribution, and thanks Steve Loughran for review and offline discussion.

        Show
        liuml07 Mingliang Liu added a comment - Committed to trunk branch. Thanks Vishwajeet Dusane for contribution, and thanks Steve Loughran for review and offline discussion.
        Hide
        vishwajeet.dusane Vishwajeet Dusane added a comment -

        @Thank you Mingliang Liu and Steve Loughran for review.

        Show
        vishwajeet.dusane Vishwajeet Dusane added a comment - @Thank you Mingliang Liu and Steve Loughran for review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10990 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10990/)
        HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data (liuml07: rev ef34bf2bb92a4e8def6617b185ae72db81450de8)

        • (edit) hadoop-tools/hadoop-azure-datalake/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10990 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10990/ ) HADOOP-13900 . Remove snapshot version of SDK dependency from Azure Data (liuml07: rev ef34bf2bb92a4e8def6617b185ae72db81450de8) (edit) hadoop-tools/hadoop-azure-datalake/pom.xml

          People

          • Assignee:
            vishwajeet.dusane Vishwajeet Dusane
            Reporter:
            vishwajeet.dusane Vishwajeet Dusane
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development