Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13687

Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      Currently, downstream projects that want to integrate with different Hadoop-compatible file systems like WASB and S3A need to list dependencies on each one. This creates an ongoing maintenance burden for those projects, because they need to update their build whenever a new Hadoop-compatible file system is introduced. This issue proposes adding a new artifact that transitively includes all Hadoop-compatible file systems. Similar to hadoop-client, this new artifact will consist of just a pom.xml listing the individual dependencies. Downstream users can depend on this artifact to sweep in everything, and picking up a new file system in a future version will be just a matter of updating the Hadoop dependency version.

      1. HADOOP-13687-branch-2.001.patch
        6 kB
        Chris Nauroth
      2. HADOOP-13687-branch-2.002.patch
        6 kB
        Chris Nauroth
      3. HADOOP-13687-branch-2.003.patch
        8 kB
        Chris Nauroth
      4. HADOOP-13687-trunk.001.patch
        7 kB
        Chris Nauroth
      5. HADOOP-13687-trunk.002.patch
        6 kB
        Chris Nauroth
      6. HADOOP-13687-trunk.003.patch
        469 kB
        Chris Nauroth
      7. HADOOP-13687-trunk.004.patch
        8 kB
        Chris Nauroth
      8. HADOOP-13687-trunk.005.patch
        8 kB
        Chris Nauroth
      9. HADOOP-13687-trunk.006.patch
        8 kB
        Mingliang Liu
      10. HADOOP-13687-trunk.006.patch
        8 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment -

          Shouldn't they be picking this up via the hadoop classpath? Isn't this going to mean that every HCFS will need to ship an artifact with Hadoop, i.e., what about the ones that don't?

          Show
          aw Allen Wittenauer added a comment - Shouldn't they be picking this up via the hadoop classpath? Isn't this going to mean that every HCFS will need to ship an artifact with Hadoop, i.e., what about the ones that don't?
          Hide
          cnauroth Chris Nauroth added a comment -

          Some applications are choosing not to run with the Hadoop classpath, and sometimes even supporting a deployment model where they are not co-located with a full Hadoop distro. For those applications, they can't rely on execution through hadoop jar or java -cp $(hadoop classpath) or similar.

          Another use case for this is to support integration testing in downstream components, where they'll be able to add a test dependency on this new artifact and run against WASB, S3A, etc.

          For applications that do not need to operate under this limitation, I agree that relying on the Hadoop classpath is the better approach.

          I think your last comment is referring to HCFS implementations that reside outside of the Apache Hadoop source tree. The intent is not to force those to ship an artifact within Hadoop, and also the intent is not for Hadoop to add an external Maven dependency on them. The scope of this only applies to the implementations already shipped in-tree. Specifically, that's hadoop-aws, hadoop-azure, hadoop-azure-datalake and hadoop-openstack.

          Show
          cnauroth Chris Nauroth added a comment - Some applications are choosing not to run with the Hadoop classpath, and sometimes even supporting a deployment model where they are not co-located with a full Hadoop distro. For those applications, they can't rely on execution through hadoop jar or java -cp $(hadoop classpath) or similar. Another use case for this is to support integration testing in downstream components, where they'll be able to add a test dependency on this new artifact and run against WASB, S3A, etc. For applications that do not need to operate under this limitation, I agree that relying on the Hadoop classpath is the better approach. I think your last comment is referring to HCFS implementations that reside outside of the Apache Hadoop source tree. The intent is not to force those to ship an artifact within Hadoop, and also the intent is not for Hadoop to add an external Maven dependency on them. The scope of this only applies to the implementations already shipped in-tree. Specifically, that's hadoop-aws, hadoop-azure, hadoop-azure-datalake and hadoop-openstack.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching revision 001 patches for trunk and branch-2.

          • I named the new module hadoop-hcfs-client, though I'm open to suggestions.
          • The pom.xml has dependencies on all HCFS implementations within the Hadoop source tree.
          • I've used exclusions to trim down the transitive dependencies, taking inspiration from hadoop-client. Maybe we could trim this down even more.
          • Due to a Maven quirk, I needed to put in explicit dependencies on a few Jackson jars. Without this, my test application that depended on hadoop-hcfs-client was picking up version 2.5.3 of jackson-databind transitively through the AWS SDK. That would conflict with our other Jackson jars, which are 2.2.3. This is a little baffling to me. I would have expected Maven to version manage down to 2.2.3 for my application, just like what happens when you build hadoop-hcfs-client itself.
          • I updated a hadoop-openstack dependency on the hadoop-common test jar so that it has test scope instead of compile scope. This way, hadoop-hcfs-client users won't pick up the hadoop-common test jar transitively.

          I tested a full distro and wrote a small custom application with a dependency on hadoop-hcfs-client to verify that functionality works correctly against all HCFS implementations.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching revision 001 patches for trunk and branch-2. I named the new module hadoop-hcfs-client, though I'm open to suggestions. The pom.xml has dependencies on all HCFS implementations within the Hadoop source tree. I've used exclusions to trim down the transitive dependencies, taking inspiration from hadoop-client. Maybe we could trim this down even more. Due to a Maven quirk, I needed to put in explicit dependencies on a few Jackson jars. Without this, my test application that depended on hadoop-hcfs-client was picking up version 2.5.3 of jackson-databind transitively through the AWS SDK. That would conflict with our other Jackson jars, which are 2.2.3. This is a little baffling to me. I would have expected Maven to version manage down to 2.2.3 for my application, just like what happens when you build hadoop-hcfs-client itself. I updated a hadoop-openstack dependency on the hadoop-common test jar so that it has test scope instead of compile scope. This way, hadoop-hcfs-client users won't pick up the hadoop-common test jar transitively. I tested a full distro and wrote a small custom application with a dependency on hadoop-hcfs-client to verify that functionality works correctly against all HCFS implementations.
          Hide
          aw Allen Wittenauer added a comment -

          I think your last comment is referring to HCFS implementations that reside outside of the Apache Hadoop source tree.

          Yes.

          The intent is not to force those to ship an artifact within Hadoop

          However, the summary of this JIRA is "includes all Hadoop-compatible file systems" ... Users who don't know any better are going to assume this really does mean "all" if we advertise it that way. (Given how itchy folks like IBM are about HCFS for their proprietary stuff, that isn't going to go over very well.)

          On the patch, from a build-system perspective, I think it makes more sense to create a hadoop-hcfs dir, move the various file systems out of tools, and put this code in there with them. This way there is a clear path of what is expected especially if/when more file systems get added. It's also an opportunity to pull these out of the tools dir in the distribution and actually make them separate components. That'd make life easier in lots of ways.

          Show
          aw Allen Wittenauer added a comment - I think your last comment is referring to HCFS implementations that reside outside of the Apache Hadoop source tree. Yes. The intent is not to force those to ship an artifact within Hadoop However, the summary of this JIRA is "includes all Hadoop-compatible file systems" ... Users who don't know any better are going to assume this really does mean "all" if we advertise it that way. (Given how itchy folks like IBM are about HCFS for their proprietary stuff, that isn't going to go over very well.) On the patch, from a build-system perspective, I think it makes more sense to create a hadoop-hcfs dir, move the various file systems out of tools, and put this code in there with them. This way there is a clear path of what is expected especially if/when more file systems get added. It's also an opportunity to pull these out of the tools dir in the distribution and actually make them separate components. That'd make life easier in lots of ways.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have updated the JIRA title for accuracy, stating that it only covers the HCFS implementations that ship with Hadoop.

          Show
          cnauroth Chris Nauroth added a comment - I have updated the JIRA title for accuracy, stating that it only covers the HCFS implementations that ship with Hadoop.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 50s trunk passed
          +1 compile 9m 11s trunk passed
          +1 mvnsite 2m 59s trunk passed
          +1 mvneclipse 0m 53s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 43s the patch passed
          +1 compile 8m 31s the patch passed
          +1 javac 8m 31s the patch passed
          +1 mvnsite 3m 14s the patch passed
          +1 mvneclipse 1m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          +1 javadoc 1m 34s the patch passed
          +1 unit 0m 9s hadoop-project in the patch passed.
          +1 unit 0m 15s hadoop-openstack in the patch passed.
          +1 unit 0m 12s hadoop-hcfs-client in the patch passed.
          +1 unit 35m 44s hadoop-tools in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          78m 46s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831815/HADOOP-13687-trunk.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 337ada727832 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 00160f7
          Default Java 1.8.0_101
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10678/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-tools/hadoop-hcfs-client hadoop-tools U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10678/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 50s trunk passed +1 compile 9m 11s trunk passed +1 mvnsite 2m 59s trunk passed +1 mvneclipse 0m 53s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 43s the patch passed +1 compile 8m 31s the patch passed +1 javac 8m 31s the patch passed +1 mvnsite 3m 14s the patch passed +1 mvneclipse 1m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 7s The patch has no ill-formed XML file. +1 javadoc 1m 34s the patch passed +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 0m 15s hadoop-openstack in the patch passed. +1 unit 0m 12s hadoop-hcfs-client in the patch passed. +1 unit 35m 44s hadoop-tools in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 78m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831815/HADOOP-13687-trunk.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 337ada727832 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 00160f7 Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10678/testReport/ modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-tools/hadoop-hcfs-client hadoop-tools U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10678/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching revision 002 patches for trunk and branch-2, with the following changes:

          1. Steve Loughran suggested that the artifact name hadoop-hcfs-client might still be misleading, something else that could make people think it really sweeps in every HCFS in the world. I switched the name to hadoop-cloud-storage.
          2. The Jackson dependency problem that I mentioned earlier is really a bug in the hadoop-aws build, and it's just a coincidence that I spotted it while testing this new artifact. I have filed HADOOP-13692 with its own patch to track that separately.

          I repeated the same testing mentioned earlier with a custom application that depends on the new artifact.

          On the patch, from a build-system perspective, I think it makes more sense to create a hadoop-hcfs dir, move the various file systems out of tools, and put this code in there with them. This way there is a clear path of what is expected especially if/when more file systems get added. It's also an opportunity to pull these out of the tools dir in the distribution and actually make them separate components. That'd make life easier in lots of ways.

          There was some discussion a few months ago on the dev list about a separate sub-tree for the file systems, but the participants concluded that it wasn't valuable. Can you describe in more detail what problems you see with the current structure and how separating the file systems out of hadoop-tools makes life easier? Maybe we missed something. hadoop-tools is getting bloated, but I figured Hadoop 3 shell profiles with classpath customization was sufficient to mitigate that.

          I'm reluctant to take on a revamp of the tree in scope of this JIRA, but maybe we can lay some groundwork.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching revision 002 patches for trunk and branch-2, with the following changes: Steve Loughran suggested that the artifact name hadoop-hcfs-client might still be misleading, something else that could make people think it really sweeps in every HCFS in the world. I switched the name to hadoop-cloud-storage . The Jackson dependency problem that I mentioned earlier is really a bug in the hadoop-aws build, and it's just a coincidence that I spotted it while testing this new artifact. I have filed HADOOP-13692 with its own patch to track that separately. I repeated the same testing mentioned earlier with a custom application that depends on the new artifact. On the patch, from a build-system perspective, I think it makes more sense to create a hadoop-hcfs dir, move the various file systems out of tools, and put this code in there with them. This way there is a clear path of what is expected especially if/when more file systems get added. It's also an opportunity to pull these out of the tools dir in the distribution and actually make them separate components. That'd make life easier in lots of ways. There was some discussion a few months ago on the dev list about a separate sub-tree for the file systems, but the participants concluded that it wasn't valuable. Can you describe in more detail what problems you see with the current structure and how separating the file systems out of hadoop-tools makes life easier? Maybe we missed something. hadoop-tools is getting bloated, but I figured Hadoop 3 shell profiles with classpath customization was sufficient to mitigate that. I'm reluctant to take on a revamp of the tree in scope of this JIRA, but maybe we can lay some groundwork.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 31s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 41s trunk passed
          +1 mvnsite 2m 23s trunk passed
          +1 mvneclipse 1m 1s trunk passed
          +1 javadoc 1m 8s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 42s the patch passed
          +1 compile 6m 40s the patch passed
          +1 javac 6m 40s the patch passed
          +1 mvnsite 2m 38s the patch passed
          +1 mvneclipse 1m 3s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          +1 javadoc 1m 18s the patch passed
          +1 unit 0m 8s hadoop-project in the patch passed.
          +1 unit 0m 13s hadoop-openstack in the patch passed.
          +1 unit 0m 11s hadoop-cloud-storage in the patch passed.
          +1 unit 34m 11s hadoop-tools in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          69m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832063/HADOOP-13687-trunk.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 982ed866438b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf37217
          Default Java 1.8.0_101
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10698/testReport/
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-tools/hadoop-cloud-storage hadoop-tools U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10698/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 31s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 6m 41s trunk passed +1 mvnsite 2m 23s trunk passed +1 mvneclipse 1m 1s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 6m 40s the patch passed +1 javac 6m 40s the patch passed +1 mvnsite 2m 38s the patch passed +1 mvneclipse 1m 3s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. +1 javadoc 1m 18s the patch passed +1 unit 0m 8s hadoop-project in the patch passed. +1 unit 0m 13s hadoop-openstack in the patch passed. +1 unit 0m 11s hadoop-cloud-storage in the patch passed. +1 unit 34m 11s hadoop-tools in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 69m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832063/HADOOP-13687-trunk.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 982ed866438b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf37217 Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10698/testReport/ modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-tools/hadoop-cloud-storage hadoop-tools U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10698/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          (this is actually a comment on patch-1; didn't hit submit in time, so some of the comments are probably obsolete)

          1. I like the idea, as for, say, SPARK-7481 it'd simplify tracking an expanding list of object stores.
          2. I'd prefer the name hadoop-cloud-storage. Why? (a) it's what it is, and (b) avoids us having to reject patches related to adding clients of external filesystems, ones not tested in ASF releases.
          3. I see the appeal of AW's suggest of a new source tree. Maybe we could start with hadoop-cloud-storage/hadoop-cloud-storage for this, move the trunk only hadoop-adl work in there, and move the others (azure, openstack, aws) at our leisure.
          4. I think you could be more aggressive about the dependencies of the openstack stuff; I suspect there is stuff there which could/should be tagged as scope=provided, so tuning down the transitiveness more.
          5. Allen Wittenauer there's no chance of Yetus doing a mvn dependencies > target/dependencies.txt operation on any patch which does poms? Or perhaps we add the policy: all patches which update dependencies must attached the changed dependency graph
          Show
          stevel@apache.org Steve Loughran added a comment - (this is actually a comment on patch-1; didn't hit submit in time, so some of the comments are probably obsolete) I like the idea, as for, say, SPARK-7481 it'd simplify tracking an expanding list of object stores. I'd prefer the name hadoop-cloud-storage . Why? (a) it's what it is, and (b) avoids us having to reject patches related to adding clients of external filesystems, ones not tested in ASF releases . I see the appeal of AW's suggest of a new source tree. Maybe we could start with hadoop-cloud-storage/hadoop-cloud-storage for this, move the trunk only hadoop-adl work in there, and move the others (azure, openstack, aws) at our leisure. I think you could be more aggressive about the dependencies of the openstack stuff; I suspect there is stuff there which could/should be tagged as scope=provided, so tuning down the transitiveness more. Allen Wittenauer there's no chance of Yetus doing a mvn dependencies > target/dependencies.txt operation on any patch which does poms? Or perhaps we add the policy: all patches which update dependencies must attached the changed dependency graph
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching revision 003 patches for trunk and branch-2, showing the structure Steve suggested in his last comment.

          hadoop-cloud-storage-project
          |- hadoop-azure-datalake
          `- hadoop-cloud-storage
          

          The trunk patch now looks huge because of git mv hadoop-tools/hadoop-azure-datalake hadoop-cloud-storage-project. The branch-2 patch is still small, because hadoop-azure-datalake doesn't exist there.

          One thing that wasn't clear to me is if people are suggesting a change to just the source layout or also the distro layout. Would we move the jars out of share/hadoop/tools and into a new share/hadoop/cloud-storage directory? It would be a backward-incompatible change, and I don't think it would add much value, so I haven't made that change in this revision. If anyone wants to lobby hard for a change in the distro layout, then we'll need additional changes to introduce a hadoop-cloud-storage-dist module, with hadoop-project-dist as its parent, the hadoop.component property set to cloud-storage, and a new cloud-storage.xml descriptor file under hadoop-assemblies.

          I think you could be more aggressive about the dependencies of the openstack stuff; I suspect there is stuff there which could/should be tagged as scope=provided, so tuning down the transitiveness more.

          I haven't gone any further yet with this. Right now, the only additional dependency that clients of hadoop-cloud-storage sweep in transitively is commons-httpclient 3.1, which is required until we break that dependency (tracked elsewhere in another JIRA). I really wanted to get rid of that test-jar dependency though.

          Allen Wittenauer there's no chance of Yetus doing a mvn dependencies > target/dependencies.txt operation on any patch which does poms? Or perhaps we add the policy: all patches which update dependencies must attached the changed dependency graph

          I think this could potentially become a feature request for Yetus pre-commit to run mvn dependency:list before and after the patch and diff the results. If anything changes, it could render a -0 in the report (not blocking the patch, but flagging that the dependency changes are worth further review).

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching revision 003 patches for trunk and branch-2, showing the structure Steve suggested in his last comment. hadoop-cloud-storage-project |- hadoop-azure-datalake `- hadoop-cloud-storage The trunk patch now looks huge because of git mv hadoop-tools/hadoop-azure-datalake hadoop-cloud-storage-project . The branch-2 patch is still small, because hadoop-azure-datalake doesn't exist there. One thing that wasn't clear to me is if people are suggesting a change to just the source layout or also the distro layout. Would we move the jars out of share/hadoop/tools and into a new share/hadoop/cloud-storage directory? It would be a backward-incompatible change, and I don't think it would add much value, so I haven't made that change in this revision. If anyone wants to lobby hard for a change in the distro layout, then we'll need additional changes to introduce a hadoop-cloud-storage-dist module, with hadoop-project-dist as its parent, the hadoop.component property set to cloud-storage , and a new cloud-storage.xml descriptor file under hadoop-assemblies . I think you could be more aggressive about the dependencies of the openstack stuff; I suspect there is stuff there which could/should be tagged as scope=provided, so tuning down the transitiveness more. I haven't gone any further yet with this. Right now, the only additional dependency that clients of hadoop-cloud-storage sweep in transitively is commons-httpclient 3.1, which is required until we break that dependency (tracked elsewhere in another JIRA). I really wanted to get rid of that test-jar dependency though. Allen Wittenauer there's no chance of Yetus doing a mvn dependencies > target/dependencies.txt operation on any patch which does poms? Or perhaps we add the policy: all patches which update dependencies must attached the changed dependency graph I think this could potentially become a feature request for Yetus pre-commit to run mvn dependency:list before and after the patch and diff the results. If anything changes, it could render a -0 in the report (not blocking the patch, but flagging that the dependency changes are worth further review).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 60 new or modified test files.
          0 mvndep 0m 47s Maven dependency ordering for branch
          +1 mvninstall 6m 57s trunk passed
          +1 compile 6m 58s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 9m 18s trunk passed
          +1 mvneclipse 1m 0s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools .
          +1 findbugs 0m 53s trunk passed
          +1 javadoc 4m 20s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 9m 5s the patch passed
          +1 compile 6m 59s the patch passed
          +1 javac 6m 59s the patch passed
          +1 checkstyle 1m 31s the patch passed
          +1 mvnsite 9m 21s the patch passed
          +1 mvneclipse 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 12s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-cloud-storage-project .
          +1 findbugs 1m 11s the patch passed
          +1 javadoc 4m 23s the patch passed
          -1 unit 96m 12s root in the patch failed.
          -1 asflicense 0m 29s The patch generated 5 ASF License warnings.
          184m 42s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
            hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832222/HADOOP-13687-trunk.003.patch
          Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
          uname Linux 7340e580deb5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6a38d11
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project/hadoop-azure-datalake hadoop-tools hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-cloud-storage-project . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 60 new or modified test files. 0 mvndep 0m 47s Maven dependency ordering for branch +1 mvninstall 6m 57s trunk passed +1 compile 6m 58s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 9m 18s trunk passed +1 mvneclipse 1m 0s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools . +1 findbugs 0m 53s trunk passed +1 javadoc 4m 20s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 9m 5s the patch passed +1 compile 6m 59s the patch passed +1 javac 6m 59s the patch passed +1 checkstyle 1m 31s the patch passed +1 mvnsite 9m 21s the patch passed +1 mvneclipse 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 12s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-tools hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-cloud-storage-project . +1 findbugs 1m 11s the patch passed +1 javadoc 4m 23s the patch passed -1 unit 96m 12s root in the patch failed. -1 asflicense 0m 29s The patch generated 5 ASF License warnings. 184m 42s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832222/HADOOP-13687-trunk.003.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 7340e580deb5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a38d11 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project/hadoop-azure-datalake hadoop-tools hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-cloud-storage-project . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10707/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Reviewed all the POMs; LTGM.

          +1

          Show
          stevel@apache.org Steve Loughran added a comment - Reviewed all the POMs; LTGM. +1
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve, thank you for your review and +1. I will hold off committing until end of week in case the other participants want further discussion.

          Show
          cnauroth Chris Nauroth added a comment - Steve, thank you for your review and +1. I will hold off committing until end of week in case the other participants want further discussion.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Still +1 on this...

          Show
          stevel@apache.org Steve Loughran added a comment - Still +1 on this...
          Hide
          cnauroth Chris Nauroth added a comment -

          HADOOP-13037 is a significant patch to hadoop-azure-datalake, which would be invalidated by the restructuring I've done here. Before I commit this patch, I'm going to be a good citizen and prepare a rebased revision of the current HADOOP-13037 patch.

          Show
          cnauroth Chris Nauroth added a comment - HADOOP-13037 is a significant patch to hadoop-azure-datalake, which would be invalidated by the restructuring I've done here. Before I commit this patch, I'm going to be a good citizen and prepare a rebased revision of the current HADOOP-13037 patch.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Chris: how about we get the module in without any changes to ADL, then that can be moved under this bit of the source tree after all its changes go in?

          Show
          stevel@apache.org Steve Loughran added a comment - Chris: how about we get the module in without any changes to ADL, then that can be moved under this bit of the source tree after all its changes go in?
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve Loughran, this sounds like a good plan to me. I think this is just a matter of taking the branch-2 patch and applying the same thing to trunk. I'm attaching a trunk patch rev 004 to test that out.

          I am about a week away from having access to a viable build environment for testing a rebased HADOOP-13037 patch, so this is a good way to get the new hadoop-cloud-storage module into place more quickly.

          Show
          cnauroth Chris Nauroth added a comment - Steve Loughran , this sounds like a good plan to me. I think this is just a matter of taking the branch-2 patch and applying the same thing to trunk. I'm attaching a trunk patch rev 004 to test that out. I am about a week away from having access to a viable build environment for testing a rebased HADOOP-13037 patch, so this is a good way to get the new hadoop-cloud-storage module into place more quickly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 52s trunk passed
          +1 compile 10m 54s trunk passed
          +1 mvnsite 9m 43s trunk passed
          +1 mvneclipse 1m 4s trunk passed
          +1 javadoc 4m 19s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          -1 mvninstall 0m 5s hadoop-cloud-storage-project in the patch failed.
          -1 mvninstall 0m 5s hadoop-cloud-storage in the patch failed.
          +1 compile 9m 21s the patch passed
          +1 javac 9m 21s the patch passed
          -1 mvnsite 0m 18s hadoop-cloud-storage-project in the patch failed.
          -1 mvnsite 0m 17s hadoop-cloud-storage in the patch failed.
          -1 mvneclipse 0m 17s hadoop-cloud-storage-project in the patch failed.
          -1 mvneclipse 0m 17s hadoop-cloud-storage in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          -1 javadoc 0m 22s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0)
          -1 javadoc 0m 18s hadoop-cloud-storage-project in the patch failed.
          -1 javadoc 0m 17s hadoop-cloud-storage in the patch failed.
          +1 unit 0m 19s hadoop-project in the patch passed.
          +1 unit 0m 23s hadoop-openstack in the patch passed.
          -1 unit 0m 18s hadoop-cloud-storage-project in the patch failed.
          -1 unit 0m 17s hadoop-cloud-storage in the patch failed.
          -1 asflicense 0m 39s The patch generated 2 ASF License warnings.
          52m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837806/HADOOP-13687-trunk.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 5320c4df1736 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acd509d
          Default Java 1.8.0_101
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 10m 54s trunk passed +1 mvnsite 9m 43s trunk passed +1 mvneclipse 1m 4s trunk passed +1 javadoc 4m 19s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch -1 mvninstall 0m 5s hadoop-cloud-storage-project in the patch failed. -1 mvninstall 0m 5s hadoop-cloud-storage in the patch failed. +1 compile 9m 21s the patch passed +1 javac 9m 21s the patch passed -1 mvnsite 0m 18s hadoop-cloud-storage-project in the patch failed. -1 mvnsite 0m 17s hadoop-cloud-storage in the patch failed. -1 mvneclipse 0m 17s hadoop-cloud-storage-project in the patch failed. -1 mvneclipse 0m 17s hadoop-cloud-storage in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. -1 javadoc 0m 22s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) -1 javadoc 0m 18s hadoop-cloud-storage-project in the patch failed. -1 javadoc 0m 17s hadoop-cloud-storage in the patch failed. +1 unit 0m 19s hadoop-project in the patch passed. +1 unit 0m 23s hadoop-openstack in the patch passed. -1 unit 0m 18s hadoop-cloud-storage-project in the patch failed. -1 unit 0m 17s hadoop-cloud-storage in the patch failed. -1 asflicense 0m 39s The patch generated 2 ASF License warnings. 52m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837806/HADOOP-13687-trunk.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 5320c4df1736 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acd509d Default Java 1.8.0_101 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project_hadoop-cloud-storage.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project_hadoop-cloud-storage.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project_hadoop-cloud-storage.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project_hadoop-cloud-storage.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project_hadoop-cloud-storage.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11008/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          jenkins was unhappy because the pom still said 2.9.0-SNAPSHOT

          Show
          stevel@apache.org Steve Loughran added a comment - jenkins was unhappy because the pom still said 2.9.0-SNAPSHOT
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm uploading trunk revision 005 to correct the version numbers in the pom.xml files.

          Show
          cnauroth Chris Nauroth added a comment - I'm uploading trunk revision 005 to correct the version numbers in the pom.xml files.
          Hide
          liuml07 Mingliang Liu added a comment -

          Like the idea and +1 on the v5 patch pending on Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - Like the idea and +1 on the v5 patch pending on Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 10m 52s trunk passed
          +1 mvnsite 10m 27s trunk passed
          +1 mvneclipse 1m 10s trunk passed
          +1 javadoc 4m 45s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          -1 mvninstall 0m 4s hadoop-cloud-storage-project in the patch failed.
          -1 mvninstall 0m 5s hadoop-cloud-storage in the patch failed.
          +1 compile 10m 3s the patch passed
          +1 javac 10m 3s the patch passed
          -1 mvnsite 0m 18s hadoop-cloud-storage-project in the patch failed.
          -1 mvnsite 0m 17s hadoop-cloud-storage in the patch failed.
          -1 mvneclipse 0m 17s hadoop-cloud-storage-project in the patch failed.
          -1 mvneclipse 0m 18s hadoop-cloud-storage in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          -1 javadoc 0m 21s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0)
          -1 javadoc 0m 16s hadoop-cloud-storage-project in the patch failed.
          -1 javadoc 0m 16s hadoop-cloud-storage in the patch failed.
          +1 unit 0m 17s hadoop-project in the patch passed.
          +1 unit 0m 27s hadoop-openstack in the patch passed.
          -1 unit 0m 16s hadoop-cloud-storage-project in the patch failed.
          -1 unit 0m 15s hadoop-cloud-storage in the patch failed.
          -1 asflicense 0m 38s The patch generated 2 ASF License warnings.
          52m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837995/HADOOP-13687-trunk.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 5a5aa19992ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbb133c
          Default Java 1.8.0_101
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project.txt
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project_hadoop-cloud-storage.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 10m 52s trunk passed +1 mvnsite 10m 27s trunk passed +1 mvneclipse 1m 10s trunk passed +1 javadoc 4m 45s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch -1 mvninstall 0m 4s hadoop-cloud-storage-project in the patch failed. -1 mvninstall 0m 5s hadoop-cloud-storage in the patch failed. +1 compile 10m 3s the patch passed +1 javac 10m 3s the patch passed -1 mvnsite 0m 18s hadoop-cloud-storage-project in the patch failed. -1 mvnsite 0m 17s hadoop-cloud-storage in the patch failed. -1 mvneclipse 0m 17s hadoop-cloud-storage-project in the patch failed. -1 mvneclipse 0m 18s hadoop-cloud-storage in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. -1 javadoc 0m 21s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) -1 javadoc 0m 16s hadoop-cloud-storage-project in the patch failed. -1 javadoc 0m 16s hadoop-cloud-storage in the patch failed. +1 unit 0m 17s hadoop-project in the patch passed. +1 unit 0m 27s hadoop-openstack in the patch passed. -1 unit 0m 16s hadoop-cloud-storage-project in the patch failed. -1 unit 0m 15s hadoop-cloud-storage in the patch failed. -1 asflicense 0m 38s The patch generated 2 ASF License warnings. 52m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837995/HADOOP-13687-trunk.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 5a5aa19992ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbb133c Default Java 1.8.0_101 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvninstall-hadoop-cloud-storage-project_hadoop-cloud-storage.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvnsite-hadoop-cloud-storage-project_hadoop-cloud-storage.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-mvneclipse-hadoop-cloud-storage-project_hadoop-cloud-storage.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-javadoc-hadoop-cloud-storage-project_hadoop-cloud-storage.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-unit-hadoop-cloud-storage-project_hadoop-cloud-storage.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11021/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching trunk revision 006, once again attempting to fix the version numbers.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching trunk revision 006, once again attempting to fix the version numbers.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 48s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 11m 11s trunk passed
          +1 mvnsite 11m 23s trunk passed
          +1 mvneclipse 1m 4s trunk passed
          +1 javadoc 5m 22s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 41s the patch passed
          +1 compile 10m 36s the patch passed
          +1 javac 10m 36s the patch passed
          +1 mvnsite 1m 33s the patch passed
          +1 mvneclipse 1m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          -1 javadoc 0m 23s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0)
          +1 unit 0m 19s hadoop-project in the patch passed.
          +1 unit 0m 24s hadoop-openstack in the patch passed.
          +1 unit 0m 21s hadoop-cloud-storage-project in the patch passed.
          +1 unit 0m 21s hadoop-cloud-storage in the patch passed.
          -1 asflicense 0m 40s The patch generated 9 ASF License warnings.
          57m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13687
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838263/HADOOP-13687-trunk.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux cddf1384aa38 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / de3a5f8
          Default Java 1.8.0_101
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 48s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 11m 11s trunk passed +1 mvnsite 11m 23s trunk passed +1 mvneclipse 1m 4s trunk passed +1 javadoc 5m 22s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 41s the patch passed +1 compile 10m 36s the patch passed +1 javac 10m 36s the patch passed +1 mvnsite 1m 33s the patch passed +1 mvneclipse 1m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. -1 javadoc 0m 23s hadoop-tools_hadoop-openstack generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) +1 unit 0m 19s hadoop-project in the patch passed. +1 unit 0m 24s hadoop-openstack in the patch passed. +1 unit 0m 21s hadoop-cloud-storage-project in the patch passed. +1 unit 0m 21s hadoop-cloud-storage in the patch passed. -1 asflicense 0m 40s The patch generated 9 ASF License warnings. 57m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13687 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838263/HADOOP-13687-trunk.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux cddf1384aa38 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de3a5f8 Default Java 1.8.0_101 javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-openstack.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-tools/hadoop-openstack hadoop-cloud-storage-project hadoop-cloud-storage-project/hadoop-cloud-storage U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11045/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          The Javadoc warnings are not new, and I will not address them as part of this patch.

          The license warnings seem to be getting triggered on Eclipse-generated files, so I don't believe they are immediately addressable.

          Show
          cnauroth Chris Nauroth added a comment - The Javadoc warnings are not new, and I will not address them as part of this patch. The license warnings seem to be getting triggered on Eclipse-generated files, so I don't believe they are immediately addressable.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Will commit this in a second.

          Show
          liuml07 Mingliang Liu added a comment - +1 Will commit this in a second.
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk and branch-2 branches. Thanks for your contribution, Chris Nauroth! Thanks for your review and discussion Steve Loughran and Allen Wittenauer.

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk and branch-2 branches. Thanks for your contribution, Chris Nauroth ! Thanks for your review and discussion Steve Loughran and Allen Wittenauer .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10813 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10813/)
          HADOOP-13687. Provide a unified dependency artifact that transitively (liuml07: rev 89354f0475efa8e393697b1ddc227c94a76b5923)

          • (add) hadoop-cloud-storage-project/pom.xml
          • (add) hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml
          • (edit) hadoop-tools/hadoop-openstack/pom.xml
          • (edit) hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10813 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10813/ ) HADOOP-13687 . Provide a unified dependency artifact that transitively (liuml07: rev 89354f0475efa8e393697b1ddc227c94a76b5923) (add) hadoop-cloud-storage-project/pom.xml (add) hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml (edit) hadoop-tools/hadoop-openstack/pom.xml (edit) hadoop-project/pom.xml

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development