Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12520

Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The hadoop-azure tests support execution against the live Azure Storage service if the developer specifies the key to an Azure Storage account. The configuration works by overwriting the src/test/resources/azure-test.xml file. This can be an error-prone process. The azure-test.xml file is checked into revision control to show an example. There is a risk that the tester could overwrite azure-test.xml containing the keys and then accidentally commit the keys to revision control. This would leak the keys to the world for potential use by an attacker. This issue proposes to use XInclude to isolate the keys into a separate file, ignored by git, which will never be committed to revision control. This is very similar to the setup already used by hadoop-aws for integration testing.

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        I'm attaching the proposed patch.

        Show
        cnauroth Chris Nauroth added a comment - I'm attaching the proposed patch.
        Hide
        wheat9 Haohui Mai added a comment -

        +1 pending jenkins.

        Show
        wheat9 Haohui Mai added a comment - +1 pending jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 7s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvnsite 0m 0s trunk passed
        +1 mvnsite 0m 0s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        0m 34s



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769129/HADOOP-12520.001.patch
        JIRA Issue HADOOP-12520
        Optional Tests asflicense site mvnsite unit xml
        uname Linux cd10933ea417 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-2392ab4/dev-support/personality/hadoop.sh
        git revision trunk / 68ce93c
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7959/testReport/
        modules C: . hadoop-tools/hadoop-azure U: .
        Max memory used 33MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7959/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvnsite 0m 0s trunk passed +1 mvnsite 0m 0s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 0m 34s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769129/HADOOP-12520.001.patch JIRA Issue HADOOP-12520 Optional Tests asflicense site mvnsite unit xml uname Linux cd10933ea417 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-2392ab4/dev-support/personality/hadoop.sh git revision trunk / 68ce93c Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7959/testReport/ modules C: . hadoop-tools/hadoop-azure U: . Max memory used 33MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7959/console This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        Thanks for the review, Haohui. I have committed this to trunk and branch-2.

        Show
        cnauroth Chris Nauroth added a comment - Thanks for the review, Haohui. I have committed this to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8717/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • .gitignore
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8717/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml .gitignore
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2536 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2536/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • .gitignore
        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2536 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2536/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) .gitignore hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #606 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/606/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • .gitignore
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #606 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/606/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml .gitignore
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • .gitignore
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) hadoop-tools/hadoop-azure/src/site/markdown/index.md .gitignore hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • .gitignore
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-common-project/hadoop-common/CHANGES.txt .gitignore
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • .gitignore
        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml .gitignore hadoop-tools/hadoop-azure/src/site/markdown/index.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/)
        HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404)

        • .gitignore
        • hadoop-tools/hadoop-azure/src/site/markdown/index.md
        • hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/ ) HADOOP-12520 . Use XInclude in hadoop-azure test configuration to isolate (cnauroth: rev 73822de7c38e189f7654444ff48d15cbe0df7404) .gitignore hadoop-tools/hadoop-azure/src/site/markdown/index.md hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml hadoop-common-project/hadoop-common/CHANGES.txt

          People

          • Assignee:
            cnauroth Chris Nauroth
            Reporter:
            cnauroth Chris Nauroth
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development