Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: backlog
    • Component/s: general
    • Labels:
      None
    1. BIGTOP-1179-tez-rpm.patch
      16 kB
      Oleg Danilov
    2. BIGTOP-1179.-Add-Apache-Tez-to-bigtop-updated-to-0.4.patch
      16 kB
      Olaf Flebbe
    3. 0002-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch
      16 kB
      Giridharan Kesavan
    4. 0001-BIGTOP-1179.-Include-tez-0.5.2.patch
      16 kB
      Olaf Flebbe
    5. 0001-BIGTOP-1179.-Include-tez-0.5.2.patch
      16 kB
      Olaf Flebbe
    6. 0001-BIGTOP-1179.-Include-tez-0.5.2.patch
      16 kB
      Olaf Flebbe
    7. 0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop-updated-to-0.5.patch
      16 kB
      Olaf Flebbe
    8. 0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch
      16 kB
      Giridharan Kesavan
    9. 0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch
      16 kB
      Olaf Flebbe
    10. 0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch
      16 kB
      Olaf Flebbe
    11. 0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch
      17 kB
      Olaf Flebbe

      Issue Links

        Activity

        Hide
        Sean Mackrory added a comment -

        +1 on including it

        Show
        Sean Mackrory added a comment - +1 on including it
        Hide
        Roman Shaposhnik added a comment -

        Would love to review a patch!

        Show
        Roman Shaposhnik added a comment - Would love to review a patch!
        Hide
        Giridharan Kesavan added a comment -

        This patch adds tez rpm/deb packaging.

        Show
        Giridharan Kesavan added a comment - This patch adds tez rpm/deb packaging.
        Hide
        Konstantin Boudnik added a comment - - edited

        Considering an embryonal state of the Tez project tests would be the must in my opinion

        Show
        Konstantin Boudnik added a comment - - edited Considering an embryonal state of the Tez project tests would be the must in my opinion
        Hide
        Giridharan Kesavan added a comment -

        Konstantin Boudnik
        sure, I can work on the integration tests.

        Show
        Giridharan Kesavan added a comment - Konstantin Boudnik sure, I can work on the integration tests.
        Hide
        Peter Linnell added a comment -

        Minor nits:

        Recommends for ntp ? Why not make it a dependency ? Just curious..

        Please wrap %description at 69 chars. It makes reading it in the rpm gui tools a lot nicer. In fact nothing in an rpm spec should exceed 69 chars. I know I know... It is a style thing

        please /bin/bash not bin/sh and declare if for the debian variants. We've long standardized on bash for the shell.

        Otherwise if my eyes do not fail me looks good and +1 for including it.

        Show
        Peter Linnell added a comment - Minor nits: Recommends for ntp ? Why not make it a dependency ? Just curious.. Please wrap %description at 69 chars. It makes reading it in the rpm gui tools a lot nicer. In fact nothing in an rpm spec should exceed 69 chars. I know I know... It is a style thing please /bin/bash not bin/sh and declare if for the debian variants. We've long standardized on bash for the shell. Otherwise if my eyes do not fail me looks good and +1 for including it.
        Hide
        Giridharan Kesavan added a comment -

        Peter Linnell Thanks, for the review comments.

        ntp shoudn't be there actually. as the jars are expected to end up in hdfs as part of deployment.

        with the 0002, I 've addressed all your comments.

        Show
        Giridharan Kesavan added a comment - Peter Linnell Thanks, for the review comments. ntp shoudn't be there actually. as the jars are expected to end up in hdfs as part of deployment. with the 0002, I 've addressed all your comments.
        Hide
        Peter Linnell added a comment -

        I still see a couple of bin/sh there. The rest looks good and +1 once you get the blocker resolved.

        Show
        Peter Linnell added a comment - I still see a couple of bin/sh there. The rest looks good and +1 once you get the blocker resolved.
        Hide
        Roman Shaposhnik added a comment -

        +1 from my side (baring Peter Linnell comments). Of course, before we can commit it to the release we'd have to have some tests, but I see you seem to have that covered in a separate JIRA. Also, if you could outline the deployment strategy for TEZ – that'd be great. Updating our Puppet code for a new component is also highly desirable.

        Show
        Roman Shaposhnik added a comment - +1 from my side (baring Peter Linnell comments). Of course, before we can commit it to the release we'd have to have some tests, but I see you seem to have that covered in a separate JIRA. Also, if you could outline the deployment strategy for TEZ – that'd be great. Updating our Puppet code for a new component is also highly desirable.
        Hide
        Andrew Purtell added a comment -

        I am -0 on the current patch. Bigtop components have one thing in common: they affect end users of the stack, either directly as a new user facing component, or indirectly as an enabler of something that does. We can include Tez now, but will it hook up to anything? If a Bigtop user installs the Tez package will it do more than just take up space?

        Show
        Andrew Purtell added a comment - I am -0 on the current patch. Bigtop components have one thing in common: they affect end users of the stack, either directly as a new user facing component, or indirectly as an enabler of something that does. We can include Tez now, but will it hook up to anything? If a Bigtop user installs the Tez package will it do more than just take up space?
        Hide
        Andrew Purtell added a comment -

        If this patch can provide some kind of end user benefit story then I will gladly add a +1.

        Show
        Andrew Purtell added a comment - If this patch can provide some kind of end user benefit story then I will gladly add a +1.
        Hide
        Konstantin Boudnik added a comment -

        Yup, that is my contention as well: what's the use of Tez in the ecosystem? What's the use case, essentially? Are we including this just "because we can"? (C) Pepsi

        Show
        Konstantin Boudnik added a comment - Yup, that is my contention as well: what's the use of Tez in the ecosystem? What's the use case, essentially? Are we including this just "because we can"? (C) Pepsi
        Hide
        Hitesh Shah added a comment -

        Giridharan Kesavan asked me to chime in on this jira.

        Tez usage:

        • Tez can be used as a standalone library similar to MapReduce or can be a library leveraged by other ecossytem projects like Hive/Pig to build their applications on.
        • For the former, tez already has a MR compatibility layer built-in and can be used by anyone to run MR jobs using the Tez execution engine. However, there is no 100% compatibility and there are likely to be some undiscovered bugs but for most part, we have tested most of the basic MR jobs against Tez and seen no issues. Users can also convert their existing MR jobs ( or chains of MR jobs ) to a native Tez DAG by directly using the Tez apis. The benefits of using the tez execution engine for existing MR jobs range from features such as container re-use ( I believe there was some level of JVM re-use support in hadoop-1.x but not in 2.x ), options of using more optimized sorters to improve shuffle performance, etc. When directly using the native APIs, the benefits are even greater.
        • Hive already has support for running natively against Tez in trunk. I believe the hive-0.13 release will be the first release with tez supported. Likewise, Pig already has a highly active developer branch for supporting pig jobs to run natively against Tez. I believe they plan to do an early preview from the branch itself at some point.
        • Even the previous releases of Hive and Pig can work against Tez. Both Hive and Pig work against MR so they can just be changed to using the tez execution engine via a config switch.

        For tests, you can use Tez's native jobs such as OrderedWordCount or MRRSleep to do basic testing. Also, you can run any MR job against Tez by changing the MR config - mapreduce.framework.name

        From a deployment point of view, Tez is very simple to deploy. It is mainly a client-side install with no requirement of jars being deployed to all nodes in the cluster. This has an additional benefit of supporting multiple versions of Tez on a single cluster.

        Feel free to send a mail to dev@tez.incubator.apache.org if you have more questions.

        Show
        Hitesh Shah added a comment - Giridharan Kesavan asked me to chime in on this jira. Tez usage: Tez can be used as a standalone library similar to MapReduce or can be a library leveraged by other ecossytem projects like Hive/Pig to build their applications on. For the former, tez already has a MR compatibility layer built-in and can be used by anyone to run MR jobs using the Tez execution engine. However, there is no 100% compatibility and there are likely to be some undiscovered bugs but for most part, we have tested most of the basic MR jobs against Tez and seen no issues. Users can also convert their existing MR jobs ( or chains of MR jobs ) to a native Tez DAG by directly using the Tez apis. The benefits of using the tez execution engine for existing MR jobs range from features such as container re-use ( I believe there was some level of JVM re-use support in hadoop-1.x but not in 2.x ), options of using more optimized sorters to improve shuffle performance, etc. When directly using the native APIs, the benefits are even greater. Hive already has support for running natively against Tez in trunk. I believe the hive-0.13 release will be the first release with tez supported. Likewise, Pig already has a highly active developer branch for supporting pig jobs to run natively against Tez. I believe they plan to do an early preview from the branch itself at some point. Even the previous releases of Hive and Pig can work against Tez. Both Hive and Pig work against MR so they can just be changed to using the tez execution engine via a config switch. For tests, you can use Tez's native jobs such as OrderedWordCount or MRRSleep to do basic testing. Also, you can run any MR job against Tez by changing the MR config - mapreduce.framework.name From a deployment point of view, Tez is very simple to deploy. It is mainly a client-side install with no requirement of jars being deployed to all nodes in the cluster. This has an additional benefit of supporting multiple versions of Tez on a single cluster. Feel free to send a mail to dev@tez.incubator.apache.org if you have more questions.
        Hide
        Konstantin Boudnik added a comment -

        I am pretty sure we can do many things with Tez. The question was is it used by the Bigtop components' that are/will be released in the 0.8.0 scope?

        Show
        Konstantin Boudnik added a comment - I am pretty sure we can do many things with Tez. The question was is it used by the Bigtop components' that are/will be released in the 0.8.0 scope?
        Hide
        Andrew Purtell added a comment -

        Feel free to send a mail to dev@tez.incubator.apache.org if you have more questions.

        Pardon, but if there are questions about a potential contribution it's really incumbent on the proposer to satisfy them.

        Tez doc talks about Hive and Pig as "end users". Neither Hive 0.13 or any unreleased version of Pig are in a BOM for any Bigtop release (yet). When that happens then it makes a lot of sense to include Tez. I think Hitesh Shah's explanation helps paint the broader picture, and I found this to be a helpful further resource: http://tez.incubator.apache.org/install.html.

        So it sounds like what is mainly missing here are integration tests that test and demonstrate what is relevant for someone wanting to use Tez directly, since there will not be an available version of Hive or Pig that transparently takes advantage of it (yet), which would be a necessary precondition for commit in my opinion.

        Show
        Andrew Purtell added a comment - Feel free to send a mail to dev@tez.incubator.apache.org if you have more questions. Pardon, but if there are questions about a potential contribution it's really incumbent on the proposer to satisfy them. Tez doc talks about Hive and Pig as "end users". Neither Hive 0.13 or any unreleased version of Pig are in a BOM for any Bigtop release (yet). When that happens then it makes a lot of sense to include Tez. I think Hitesh Shah 's explanation helps paint the broader picture, and I found this to be a helpful further resource: http://tez.incubator.apache.org/install.html . So it sounds like what is mainly missing here are integration tests that test and demonstrate what is relevant for someone wanting to use Tez directly, since there will not be an available version of Hive or Pig that transparently takes advantage of it (yet), which would be a necessary precondition for commit in my opinion.
        Hide
        Hitesh Shah added a comment -

        Konstantin Boudnik I am not sure what versions of components are in Bigtop-0.8.0. To re-iterate what I mentioned previously, Hive 0.13 will have native Tez support and therefore a direct dependency on Tez. As for Pig, they are still on a branch so I am not sure what release version of Pig will have a direct Tez dependency.

        If the case for adding a new component to Bigtop is formed on the basis of inter-component dependencies, then the above should help resolve the issue for you.

        My previous comment was made in reference to a combination of issues raised in other comments, one of which was having an end user benefit story to make a case for adding a new component. Hopefully, my earlier comment addresses all the potential benefits of Tez for an end-user in the Hadoop ecosystem be it by using Tez indirectly via MR/Hive/Pig or by directly writing new code against the Tez apis.

        Show
        Hitesh Shah added a comment - Konstantin Boudnik I am not sure what versions of components are in Bigtop-0.8.0. To re-iterate what I mentioned previously, Hive 0.13 will have native Tez support and therefore a direct dependency on Tez. As for Pig, they are still on a branch so I am not sure what release version of Pig will have a direct Tez dependency. If the case for adding a new component to Bigtop is formed on the basis of inter-component dependencies, then the above should help resolve the issue for you. My previous comment was made in reference to a combination of issues raised in other comments, one of which was having an end user benefit story to make a case for adding a new component. Hopefully, my earlier comment addresses all the potential benefits of Tez for an end-user in the Hadoop ecosystem be it by using Tez indirectly via MR/Hive/Pig or by directly writing new code against the Tez apis.
        Hide
        Andrew Purtell added a comment -

        Hive 0.13 will have native Tez support and therefore a direct dependency on Tez. As for Pig, they are still on a branch so I am not sure what release version of Pig will have a direct Tez dependency.

        Direct dependencies make a prima facie case for inclusion, that's not the issue right now.

        Show
        Andrew Purtell added a comment - Hive 0.13 will have native Tez support and therefore a direct dependency on Tez. As for Pig, they are still on a branch so I am not sure what release version of Pig will have a direct Tez dependency. Direct dependencies make a prima facie case for inclusion, that's not the issue right now.
        Hide
        Bruno Mahé added a comment -

        If the case for adding a new component to Bigtop is formed on the basis of inter-component dependencies, then the above should help resolve the issue for you.

        No, there is no such rule.
        Seems like people are more concerned with the maintenance cost of including Apache TEZ (Incubating). Having some components of the Apache Hadoop stack depending on it or having users writing their code against Apache TEZ (Incubating) makes it worthwhile.
        Having someone volunteering for maintaining it could also put some mind more at ease.

        Given that some work has been done to integrate Apache TEZ (Incubating) in Apache Bigtop and it is being used in multiple projects, I am in favor of adding it.
        I also quickly looked at the patch and saw some references to TEZ. I believe it should be Apache TEZ (Incubating).

        Show
        Bruno Mahé added a comment - If the case for adding a new component to Bigtop is formed on the basis of inter-component dependencies, then the above should help resolve the issue for you. No, there is no such rule. Seems like people are more concerned with the maintenance cost of including Apache TEZ (Incubating). Having some components of the Apache Hadoop stack depending on it or having users writing their code against Apache TEZ (Incubating) makes it worthwhile. Having someone volunteering for maintaining it could also put some mind more at ease. Given that some work has been done to integrate Apache TEZ (Incubating) in Apache Bigtop and it is being used in multiple projects, I am in favor of adding it. I also quickly looked at the patch and saw some references to TEZ. I believe it should be Apache TEZ (Incubating).
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Is there an idea of when component releases make into a BOM? Like which Bigtop release can potentially have Hive 0.13? And the current slated version is 0.12? I looked around but couldn't find this info. Any pointers appreciated. Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - Is there an idea of when component releases make into a BOM? Like which Bigtop release can potentially have Hive 0.13? And the current slated version is 0.12? I looked around but couldn't find this info. Any pointers appreciated. Thanks!
        Hide
        Konstantin Boudnik added a comment -

        Vinod, I don't think Bigtop community is in a position to make this kind of call. As far as I know Hive 0.13 hasn't been released so far. Hence, Bigtop 0.8.0 is aiming to include 0.12 to the BOM.

        Please ping us once Hive 0.13 is out and depends on where we are in the release cycle the BOM's content can be potentially re-voted.

        Show
        Konstantin Boudnik added a comment - Vinod, I don't think Bigtop community is in a position to make this kind of call. As far as I know Hive 0.13 hasn't been released so far. Hence, Bigtop 0.8.0 is aiming to include 0.12 to the BOM. Please ping us once Hive 0.13 is out and depends on where we are in the release cycle the BOM's content can be potentially re-voted.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Sure, tx for the info.

        Show
        Vinod Kumar Vavilapalli added a comment - Sure, tx for the info.
        Hide
        Olaf Flebbe added a comment -

        ... Just to mention hive 0.13.0 is in bigtop-trunk (the last time I checked) and 0.13.1 is released back in june

        Show
        Olaf Flebbe added a comment - ... Just to mention hive 0.13.0 is in bigtop-trunk (the last time I checked) and 0.13.1 is released back in june
        Hide
        Konstantin Boudnik added a comment -

        Oh well - too late to add to 0.8.0. I want to have it released next week. We just went through a bunch of polishing and fixing and hopefully the cluster tests will be ok. Let's bump up Hive in 0.9.0 which, hopefully, will be done just in a few weeks.

        Show
        Konstantin Boudnik added a comment - Oh well - too late to add to 0.8.0. I want to have it released next week. We just went through a bunch of polishing and fixing and hopefully the cluster tests will be ok. Let's bump up Hive in 0.9.0 which, hopefully, will be done just in a few weeks.
        Hide
        Olaf Flebbe added a comment - - edited

        Updated Patch for tez 0.5.0

        Unfortunately tez 0.5.0 breaks hive 0.13.x so we have to stall 0.5.0 until hive 0.14 is released.

        http://search-hadoop.com/m/tQlj5opT&subj=Re+run+hive+atop+tez

        Show
        Olaf Flebbe added a comment - - edited Updated Patch for tez 0.5.0 Unfortunately tez 0.5.0 breaks hive 0.13.x so we have to stall 0.5.0 until hive 0.14 is released. http://search-hadoop.com/m/tQlj5opT&subj=Re+run+hive+atop+tez
        Hide
        Roman Shaposhnik added a comment -

        Olaf Flebbe this type of information would be super useful to collate now that we'll be compiling BOM for Bigtop 0.9.0. Please keep an eye on that JIRA: BIGTOP-1480

        Show
        Roman Shaposhnik added a comment - Olaf Flebbe this type of information would be super useful to collate now that we'll be compiling BOM for Bigtop 0.9.0. Please keep an eye on that JIRA: BIGTOP-1480
        Hide
        Olaf Flebbe added a comment -

        Updated (Downgraded the patch to tez-0.4.1 .

        Unfortunately the download from incubator is broken, since it is packed on a mac with thousands of ._* files confusing the debian build system. For now I manually removed this file in the dl/tar* download. Can fix it if this patch is likely to be applied.

        The Hive performance gain is overwhelming, if one follows the instructions at hortonworks closely.

        Show
        Olaf Flebbe added a comment - Updated (Downgraded the patch to tez-0.4.1 . Unfortunately the download from incubator is broken, since it is packed on a mac with thousands of ._* files confusing the debian build system. For now I manually removed this file in the dl/tar* download. Can fix it if this patch is likely to be applied. The Hive performance gain is overwhelming, if one follows the instructions at hortonworks closely.
        Hide
        Olaf Flebbe added a comment -

        This one works with hive 0.13

        Show
        Olaf Flebbe added a comment - This one works with hive 0.13
        Hide
        Oleg Danilov added a comment -

        I've added the slightly corrected version of the latest patch (thanks to Olaf Flebbe). Fixed some issues with -incubating suffix.

        Show
        Oleg Danilov added a comment - I've added the slightly corrected version of the latest patch (thanks to Olaf Flebbe ). Fixed some issues with -incubating suffix.
        Hide
        Roman Shaposhnik added a comment -

        Nice to see progress on this. Would anybody be interested in picking up the smoke tests bit? That seems to be the only thing preventing Tez inclusion into 0.9.0

        Show
        Roman Shaposhnik added a comment - Nice to see progress on this. Would anybody be interested in picking up the smoke tests bit? That seems to be the only thing preventing Tez inclusion into 0.9.0
        Hide
        jay vyas added a comment -

        I've got a smoke test in bigtop-1190 which I attached that works pretty good as a hive on tez smoke test.

        Only comment there was that needed to be moved into smoke tests under hive/ and that can serve as the test.

        Show
        jay vyas added a comment - I've got a smoke test in bigtop-1190 which I attached that works pretty good as a hive on tez smoke test. Only comment there was that needed to be moved into smoke tests under hive/ and that can serve as the test.
        Hide
        Olaf Flebbe added a comment - - edited

        Uploaded a refined patch to Version 0.5.2

        a hive 0.14.x snapshot compiled cleanly with it.

        0001-BIGTOP-1179.-Include-tez-0.5.2.patch

        Show
        Olaf Flebbe added a comment - - edited Uploaded a refined patch to Version 0.5.2 a hive 0.14.x snapshot compiled cleanly with it. 0001- BIGTOP-1179 .-Include-tez-0.5.2.patch
        Hide
        Olaf Flebbe added a comment -

        Updated corrupted tez.1 file

        Show
        Olaf Flebbe added a comment - Updated corrupted tez.1 file
        Hide
        Olaf Flebbe added a comment -

        Sigh... another typo still remained

        Show
        Olaf Flebbe added a comment - Sigh... another typo still remained
        Hide
        Roman Shaposhnik added a comment -

        Trying to understand what this is blocked on: is this just Hive 0.14.x and lack of tests for Tez?

        Show
        Roman Shaposhnik added a comment - Trying to understand what this is blocked on: is this just Hive 0.14.x and lack of tests for Tez?
        Hide
        Olaf Flebbe added a comment - - edited

        Refined the patch to include tez.

        • Updated debian control files to give less warnings (still compatible with ubuntu LTS).
        • removed sl4j*.jar because already provided by hadoop

        This patch is a prerequiste for hive 0.14.x

        I have no clue how automated tests are done in bigtop so I cannot provide tests. Please give directions.

        Tested and running with "my" workload: hive-0.14 on tez.

        Show
        Olaf Flebbe added a comment - - edited Refined the patch to include tez. Updated debian control files to give less warnings (still compatible with ubuntu LTS). removed sl4j*.jar because already provided by hadoop This patch is a prerequiste for hive 0.14.x I have no clue how automated tests are done in bigtop so I cannot provide tests. Please give directions. Tested and running with "my" workload: hive-0.14 on tez.
        Hide
        jay vyas added a comment -

        Hi Olaf. One really simple and reproducible way to do this which I think should work!

        • build tez gradle tez-yum on local machine.
        • apply the tez smoke test patch from the other jira, which runs a hive job w tez configuration.
        • spin up a vagrant cluster by running vagrant up (see bigtop-deploy/vm). You will want to run vagrant-puppet (for vbox cluster) or docker-puppet (for docker cluster).
        • vagrant ssh into your cluster after vagrant up completes.
        • yum install tez (vagrant now supports loading rpms from output/ directory on local host )!
        • run the smoke-tests with argument -Dsmoke.tests=tez
        Show
        jay vyas added a comment - Hi Olaf. One really simple and reproducible way to do this which I think should work! build tez gradle tez-yum on local machine. apply the tez smoke test patch from the other jira, which runs a hive job w tez configuration. spin up a vagrant cluster by running vagrant up (see bigtop-deploy/vm). You will want to run vagrant-puppet (for vbox cluster) or docker-puppet (for docker cluster). vagrant ssh into your cluster after vagrant up completes. yum install tez (vagrant now supports loading rpms from output/ directory on local host )! run the smoke-tests with argument -Dsmoke.tests=tez
        Hide
        Olaf Flebbe added a comment -

        Ok, need some time to work this out. Is this documented anywhere on the web page/ wiki ?

        Show
        Olaf Flebbe added a comment - Ok, need some time to work this out. Is this documented anywhere on the web page/ wiki ?
        Hide
        Olaf Flebbe added a comment -

        Updated patch: Source File name was wrong in RPM SPECS file

        Show
        Olaf Flebbe added a comment - Updated patch: Source File name was wrong in RPM SPECS file
        Hide
        Olaf Flebbe added a comment - - edited

        Please review Roman Shaposhnik]

        Show
        Olaf Flebbe added a comment - - edited Please review Roman Shaposhnik ]
        Hide
        Peter Linnell added a comment -

        +1 on the latest patch. Seems sane and I see no glaring errors. I'm +1 on committing if the smoke tests are working.

        Show
        Peter Linnell added a comment - +1 on the latest patch. Seems sane and I see no glaring errors. I'm +1 on committing if the smoke tests are working.
        Hide
        Roman Shaposhnik added a comment -

        One last question: Olaf Flebbe would you be comfortable signing up for long term support of Tez in Bigtop? If so, could you, please update MAINTAINERS.txt as part of this patch?

        Show
        Roman Shaposhnik added a comment - One last question: Olaf Flebbe would you be comfortable signing up for long term support of Tez in Bigtop? If so, could you, please update MAINTAINERS.txt as part of this patch?
        Hide
        Olaf Flebbe added a comment -

        Today I found out that this version of tez seems to have a problem with HA failover. But since tez has to be enabled first, which is not the case for Bigtop, this is not a big deal right now.

        Since we have now hive-0.14 in bigtop we should update to 0.5.3 at least (if not 0.6.0).

        But I have to test first ...

        Show
        Olaf Flebbe added a comment - Today I found out that this version of tez seems to have a problem with HA failover. But since tez has to be enabled first, which is not the case for Bigtop, this is not a big deal right now. Since we have now hive-0.14 in bigtop we should update to 0.5.3 at least (if not 0.6.0). But I have to test first ...
        Hide
        Olaf Flebbe added a comment -

        done

        Show
        Olaf Flebbe added a comment - done
        Hide
        Olaf Flebbe added a comment -

        Ok, Patch reworked for 0.6.0 and tested at my setup. Please review in order to get TEZ config in in a seperate JIRA ticket.

        Show
        Olaf Flebbe added a comment - Ok, Patch reworked for 0.6.0 and tested at my setup. Please review in order to get TEZ config in in a seperate JIRA ticket.
        Hide
        YoungWoo Kim added a comment -

        Applied your patch and then I got an error like below:

        $ ./gradlew tez-rpm
        
        ...
        
        + cd /home/ywkim/workspace/bigtop/build/tez/rpm/BUILD
        + rm -rf tez-0.6.0
        + /usr/bin/gzip -dc /home/ywkim/workspace/bigtop/build/tez/rpm/SOURCES/apache-tez-0.6.0-src.tar.gz
        + /usr/bin/tar -xf -
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.dev'
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.ino'
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink'
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.dev'
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.ino'
        /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink'
        + STATUS=0
        + '[' 0 -ne 0 ']'
        
        + cd tez-0.6.0
        /var/tmp/rpm-tmp.iq6Eq4: line 37: cd: tez-0.6.0: No such file or directory
        
        
        Show
        YoungWoo Kim added a comment - Applied your patch and then I got an error like below: $ ./gradlew tez-rpm ... + cd /home/ywkim/workspace/bigtop/build/tez/rpm/BUILD + rm -rf tez-0.6.0 + /usr/bin/gzip -dc /home/ywkim/workspace/bigtop/build/tez/rpm/SOURCES/apache-tez-0.6.0-src.tar.gz + /usr/bin/tar -xf - /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.dev' /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.ino' /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink' /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.dev' /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.ino' /usr/bin/tar: Ignoring unknown extended header keyword `SCHILY.nlink' + STATUS=0 + '[' 0 -ne 0 ']' + cd tez-0.6.0 /var/tmp/rpm-tmp.iq6Eq4: line 37: cd: tez-0.6.0: No such file or directory
        Hide
        Olaf Flebbe added a comment -

        Thanks, fixed RPM

        Show
        Olaf Flebbe added a comment - Thanks, fixed RPM
        Hide
        YoungWoo Kim added a comment - - edited

        Olaf Flebbe Thanks. Right now packaging rpm works fine for me.

        Show
        YoungWoo Kim added a comment - - edited Olaf Flebbe Thanks. Right now packaging rpm works fine for me.
        Hide
        YoungWoo Kim added a comment - - edited

        Olaf Flebbe A few comments:
        1. How about adding TEZ_CONF_DIR and tez-site.xml? for instance, /usr/lib/tez/conf -> /etc/tez/conf and /etc/tez/conf/tez-site.xml
        2. Excluding war for tez-ui is intentional?
        3. slf4j*.jar are missing
        My tez-site.xml has a property:

          <property>
            <name>tez.lib.uris</name>
            <value>hdfs://mycluster/apps/tez/,hdfs://mycluster/apps/tez/lib</value>
          </property>
        

        I usually copy $TEZ_HOME/lib/*.jar to hdfs://mycluster/apps/tez/lib/. if slf4j*.jar are missing, tez apps will fail.

        Overall looks good to me. With some manual setup, an example for tez works fine for me on vagrant-puppet-vm

        [vagrant@bigtop1 ~]$ yarn jar /usr/lib/tez/tez-examples-0.6.0.jar orderedwordcount /tmp/passwd /tmp/out
        Running OrderedWordCount
        15/02/05 06:00:19 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.6.0, revision=42ac3a5c1ceca3605741674a7bda27579734880d, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150205-1158 ]
        ...
        15/02/05 06:00:35 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED
        
        [vagrant@bigtop1 ~]$ hdfs dfs -cat /tmp/out/*
        zookeeper:x:497:497:ZooKeeper:/var/lib/zookeeper:/sbin/nologin	1
        yarn:x:495:494:Hadoop	1
        ...
        HDFS:/var/lib/hadoop-hdfs:/bin/bash	1
        Daemon:/var/cache/rpcbind:/sbin/nologin	1
        User:/var/lib/nfs:/sbin/nologin	2
        
        Show
        YoungWoo Kim added a comment - - edited Olaf Flebbe A few comments: 1. How about adding TEZ_CONF_DIR and tez-site.xml? for instance, /usr/lib/tez/conf -> /etc/tez/conf and /etc/tez/conf/tez-site.xml 2. Excluding war for tez-ui is intentional? 3. slf4j*.jar are missing My tez-site.xml has a property: <property> <name>tez.lib.uris</name> <value>hdfs: //mycluster/apps/tez/,hdfs://mycluster/apps/tez/lib</value> </property> I usually copy $TEZ_HOME/lib/*.jar to hdfs://mycluster/apps/tez/lib/. if slf4j*.jar are missing, tez apps will fail. Overall looks good to me. With some manual setup, an example for tez works fine for me on vagrant-puppet-vm [vagrant@bigtop1 ~]$ yarn jar /usr/lib/tez/tez-examples-0.6.0.jar orderedwordcount /tmp/passwd /tmp/out Running OrderedWordCount 15/02/05 06:00:19 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.6.0, revision=42ac3a5c1ceca3605741674a7bda27579734880d, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150205-1158 ] ... 15/02/05 06:00:35 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED [vagrant@bigtop1 ~]$ hdfs dfs -cat /tmp/out/* zookeeper:x:497:497:ZooKeeper:/var/lib/zookeeper:/sbin/nologin 1 yarn:x:495:494:Hadoop 1 ... HDFS:/var/lib/hadoop-hdfs:/bin/bash 1 Daemon:/var/cache/rpcbind:/sbin/nologin 1 User:/var/lib/nfs:/sbin/nologin 2
        Hide
        Bruno Mahé added a comment -

        Nitpicks:

        • In a lot of places (descriptions, summaries, man pages...), I see the project being referenced as TEZ. But it should be Apache TEZ. Same for Bigtop, it should be Apache Bigtop (ex: maintainer section)
        • the rules files has a reference to zookeeper_home but does not seem to be used.
        Show
        Bruno Mahé added a comment - Nitpicks: In a lot of places (descriptions, summaries, man pages...), I see the project being referenced as TEZ. But it should be Apache TEZ. Same for Bigtop, it should be Apache Bigtop (ex: maintainer section) the rules files has a reference to zookeeper_home but does not seem to be used.
        Hide
        Olaf Flebbe added a comment -

        Ok, I streamlined all the tez references from the past to the text from the web page.

        Regarding Maintainer: Section If you like to have this changed, please use a separate JIRA, since this affects ALL Debian Packages.

        rules file removed zookeeper reference

        Show
        Olaf Flebbe added a comment - Ok, I streamlined all the tez references from the past to the text from the web page. Regarding Maintainer: Section If you like to have this changed, please use a separate JIRA, since this affects ALL Debian Packages. rules file removed zookeeper reference
        Hide
        Olaf Flebbe added a comment -

        Regarding (2). Since I did a update from 0.5.3 I simply missed it. YoungWoo Kim: I am thinking about creating a separate package tez-ui, including only the war file. What do you think?

        3) That's right tez AM does not run without sl4j in HDFS. However, it is not needed by tez on client side, AFAIK . There are already too many versions of sl4j in class path already (for instance shown in the beeline command prompt). I would recommend to copy sl4j to hdfs when configuring properly. What do you think?

        1) Ooch ... yes that should be part of the package.

        Show
        Olaf Flebbe added a comment - Regarding (2). Since I did a update from 0.5.3 I simply missed it. YoungWoo Kim : I am thinking about creating a separate package tez-ui, including only the war file. What do you think? 3) That's right tez AM does not run without sl4j in HDFS. However, it is not needed by tez on client side, AFAIK . There are already too many versions of sl4j in class path already (for instance shown in the beeline command prompt). I would recommend to copy sl4j to hdfs when configuring properly. What do you think? 1) Ooch ... yes that should be part of the package.
        Hide
        jay vyas added a comment -

        Olaf Flebbe

        • is this ready for a final review? If so - how do you want us to test it
        • Can you create a follo on JIRA for puppetization + Smoke test completion of TEZ?

        Thanks!

        Show
        jay vyas added a comment - Olaf Flebbe is this ready for a final review? If so - how do you want us to test it Can you create a follo on JIRA for puppetization + Smoke test completion of TEZ? Thanks!
        Hide
        Olaf Flebbe added a comment - - edited

        Tez -> Apache Tez
        More Love to RPM
        Fixed installation, stray directories etc
        Added /etc/tez/conf/tez-site.xml
        Used tez-minimal adapted tez-site.xml accordingly (Fixes Problem with too many jars in /apps/tez, missing sl4j)
        Added tez-ui.war
        That should be final.

        Notes Regarding installation:

        1)
        hadoop-env.sh has to be changed (Next JIRA for hadoop itself)

        export TEZ_CONF_DIR=/etc/tez/conf
        export TEZ_JARS="/usr/lib/tez/*:/usr/lib/tez/lib/*"
        
        export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
        

        2)

        hdfs dfs -mkdir /apps/tez
        hdfs dfs -mkdir /apps/tez/lib
        
        hdfs dfs -copyFromLocal  /usr/lib/tez/* /apps/tez
         hdfs dfs -copyFromLocal /usr/lib/tez/lib/* /apps/tez/lib
        

        3) Change mapred-site.xml

        <property>
               <name>mapreduce.framework.name</name>
               <value>yarn-tez</value>
        </property>
        

        4) Test hadoop-mapred-examples

        test tez-examples

        Use

        set hive.execution.engine=tez
        

        withing beeline to use hive or change hive-site.xml accordingly

        Show
        Olaf Flebbe added a comment - - edited Tez -> Apache Tez More Love to RPM Fixed installation, stray directories etc Added /etc/tez/conf/tez-site.xml Used tez-minimal adapted tez-site.xml accordingly (Fixes Problem with too many jars in /apps/tez, missing sl4j) Added tez-ui.war That should be final. Notes Regarding installation: 1) hadoop-env.sh has to be changed (Next JIRA for hadoop itself) export TEZ_CONF_DIR=/etc/tez/conf export TEZ_JARS= "/usr/lib/tez/*:/usr/lib/tez/lib/*" export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH 2) hdfs dfs -mkdir /apps/tez hdfs dfs -mkdir /apps/tez/lib hdfs dfs -copyFromLocal /usr/lib/tez/* /apps/tez hdfs dfs -copyFromLocal /usr/lib/tez/lib/* /apps/tez/lib 3) Change mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn-tez</value> </property> 4) Test hadoop-mapred-examples test tez-examples Use set hive.execution.engine=tez withing beeline to use hive or change hive-site.xml accordingly
        Hide
        Olaf Flebbe added a comment -

        Ready for Final review.

        Please read my notes, regarding configuration

        Show
        Olaf Flebbe added a comment - Ready for Final review. Please read my notes, regarding configuration
        Hide
        jay vyas added a comment -

        okay, thanks olaf. ill take a look at this patch now! it will be a day for me to test, but im starting tonite.

        Show
        jay vyas added a comment - okay, thanks olaf. ill take a look at this patch now! it will be a day for me to test, but im starting tonite.
        Hide
        jay vyas added a comment -

        I ran gradle tez-rpm.... it looks like the tez-ui step failed.

        Any thoughts Olaf Flebbe?

        [INFO] 
        [INFO] tez ................................................ SUCCESS [  9.100 s]
        [INFO] tez-api ............................................ SUCCESS [ 15.177 s]
        [INFO] tez-common ......................................... SUCCESS [  0.243 s]
        [INFO] tez-runtime-internals .............................. SUCCESS [  0.366 s]
        [INFO] tez-runtime-library ................................ SUCCESS [  1.918 s]
        [INFO] tez-mapreduce ...................................... SUCCESS [  0.716 s]
        [INFO] tez-examples ....................................... SUCCESS [  0.167 s]
        [INFO] tez-dag ............................................ SUCCESS [  2.668 s]
        [INFO] tez-tests .......................................... SUCCESS [  6.754 s]
        [INFO] tez-ui ............................................. FAILURE [11:50 min]
        [INFO] tez-plugins ........................................ SKIPPED
        [INFO] tez-yarn-timeline-history .......................... SKIPPED
        [INFO] tez-mbeans-resource-calculator ..................... SKIPPED
        [INFO] tez-dist ........................................... SKIPPED
        [INFO] Tez ................................................ SKIPPED
        
        
        Show
        jay vyas added a comment - I ran gradle tez-rpm .... it looks like the tez-ui step failed. Any thoughts Olaf Flebbe ? [INFO] [INFO] tez ................................................ SUCCESS [ 9.100 s] [INFO] tez-api ............................................ SUCCESS [ 15.177 s] [INFO] tez-common ......................................... SUCCESS [ 0.243 s] [INFO] tez-runtime-internals .............................. SUCCESS [ 0.366 s] [INFO] tez-runtime-library ................................ SUCCESS [ 1.918 s] [INFO] tez-mapreduce ...................................... SUCCESS [ 0.716 s] [INFO] tez-examples ....................................... SUCCESS [ 0.167 s] [INFO] tez-dag ............................................ SUCCESS [ 2.668 s] [INFO] tez-tests .......................................... SUCCESS [ 6.754 s] [INFO] tez-ui ............................................. FAILURE [11:50 min] [INFO] tez-plugins ........................................ SKIPPED [INFO] tez-yarn-timeline-history .......................... SKIPPED [INFO] tez-mbeans-resource-calculator ..................... SKIPPED [INFO] tez-dist ........................................... SKIPPED [INFO] Tez ................................................ SKIPPED
        Hide
        Olaf Flebbe added a comment -

        Could you please supply full error message?

        There may be same required component missing in your base os, since tez-ui is – ahem – a javascript beast. But I never had an issue with it before.

        I am away from my computer for today, btw.

        Show
        Olaf Flebbe added a comment - Could you please supply full error message? There may be same required component missing in your base os, since tez-ui is – ahem – a javascript beast. But I never had an issue with it before. I am away from my computer for today, btw.
        Hide
        jay vyas added a comment -

        Hi Olaf.

        I don't have any JavaScript specific Libs on this machine.

        If we need them let's make em explicit.

        I'll see if I can dig out more error messages

        Show
        jay vyas added a comment - Hi Olaf. I don't have any JavaScript specific Libs on this machine. If we need them let's make em explicit. I'll see if I can dig out more error messages
        Hide
        Olaf Flebbe added a comment -

        I doublechecked: A minimal cents with bigtop_toolchain applied has all necessary tools to recompile,

        jay vyas: You are compiling with userid 0 aka root, right ? May I ask you to use a non priviledged account?

        bower ESUDO         Cannot be run with sudo
        
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO] 
        [INFO] tez ............................................... SUCCESS [1:42.852s]
        [INFO] tez-api ........................................... SUCCESS [4:25.661s]
        [INFO] tez-common ........................................ SUCCESS [1.207s]
        [INFO] tez-runtime-internals ............................. SUCCESS [1.587s]
        [INFO] tez-runtime-library ............................... SUCCESS [9.434s]
        [INFO] tez-mapreduce ..................................... SUCCESS [10.137s]
        [INFO] tez-examples ...................................... SUCCESS [1.062s]
        [INFO] tez-dag ........................................... SUCCESS [10.116s]
        [INFO] tez-tests ......................................... SUCCESS [22.223s]
        [INFO] tez-ui ............................................ FAILURE [4:03.862s]
        [INFO] tez-plugins ....................................... SKIPPED
        [INFO] tez-yarn-timeline-history ......................... SKIPPED
        [INFO] tez-mbeans-resource-calculator .................... SKIPPED
        [INFO] tez-dist .......................................... SKIPPED
        [INFO] Tez ............................................... SKIPPED
        [INFO] ------------------------------------------------------------------------
        Additional error details:
        Since bower is a user command, there is no need to execute it with superuser permissions.
        If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs.
        
        Show
        Olaf Flebbe added a comment - I doublechecked: A minimal cents with bigtop_toolchain applied has all necessary tools to recompile, jay vyas : You are compiling with userid 0 aka root, right ? May I ask you to use a non priviledged account? bower ESUDO Cannot be run with sudo [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] tez ............................................... SUCCESS [1:42.852s] [INFO] tez-api ........................................... SUCCESS [4:25.661s] [INFO] tez-common ........................................ SUCCESS [1.207s] [INFO] tez-runtime-internals ............................. SUCCESS [1.587s] [INFO] tez-runtime-library ............................... SUCCESS [9.434s] [INFO] tez-mapreduce ..................................... SUCCESS [10.137s] [INFO] tez-examples ...................................... SUCCESS [1.062s] [INFO] tez-dag ........................................... SUCCESS [10.116s] [INFO] tez-tests ......................................... SUCCESS [22.223s] [INFO] tez-ui ............................................ FAILURE [4:03.862s] [INFO] tez-plugins ....................................... SKIPPED [INFO] tez-yarn-timeline-history ......................... SKIPPED [INFO] tez-mbeans-resource-calculator .................... SKIPPED [INFO] tez-dist .......................................... SKIPPED [INFO] Tez ............................................... SKIPPED [INFO] ------------------------------------------------------------------------ Additional error details: Since bower is a user command, there is no need to execute it with superuser permissions. If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs.
        Hide
        jay vyas added a comment -

        Thanks Olaf. Iirc I didn't run as root. Let me see if that fixes the issue..... I'll update tonite!

        Show
        jay vyas added a comment - Thanks Olaf. Iirc I didn't run as root. Let me see if that fixes the issue..... I'll update tonite!
        Hide
        jay vyas added a comment - - edited

        looks like the same issue.... i can try to re run bigtop toolchain to see...

        [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui ---
        bower ESUDO         Cannot be run with sudo
        Additional error details:
        Since bower is a user command, there is no need to execute it with superuser permissions.
        If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs.
        http://www.joyent.com/blog/installing-node-and-npm
        https://gist.github.com/isaacs/579814
        
        Show
        jay vyas added a comment - - edited looks like the same issue.... i can try to re run bigtop toolchain to see... [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui --- bower ESUDO Cannot be run with sudo Additional error details: Since bower is a user command, there is no need to execute it with superuser permissions. If you're having permission errors when using bower without sudo, please spend a few minutes learning more about how your system should work and make any necessary repairs. http://www.joyent.com/blog/installing-node-and-npm https://gist.github.com/isaacs/579814
        Hide
        Olaf Flebbe added a comment -

        jay vyas You did not understand me correctly. I do not have this issue. I only compiled it as root accidentally while preparing a minimal centos machine. Switching user fixed the problem,

        AFAIK you are running compile process definitely with root permissions, since the code triggering the message reads

        'use strict';
        module.exports = function ()

        { return process.getuid && process.getuid() === 0; }

        ;

        I.e. getuid() returned 0

        A few tickets ago I tripped myself over a piece of code of you which couldn't have worked if you are compiling as non-root. Please recheck.

        Show
        Olaf Flebbe added a comment - jay vyas You did not understand me correctly. I do not have this issue. I only compiled it as root accidentally while preparing a minimal centos machine. Switching user fixed the problem, AFAIK you are running compile process definitely with root permissions, since the code triggering the message reads 'use strict'; module.exports = function () { return process.getuid && process.getuid() === 0; } ; I.e. getuid() returned 0 A few tickets ago I tripped myself over a piece of code of you which couldn't have worked if you are compiling as non-root. Please recheck.
        Hide
        jay vyas added a comment -

        Actually, I tried this one first as non-root.

        Then as root...

        I can look some more, but other packages (spark, hadoop,... ) build ;
        Any other ideas (or if someone else wants to test thats fine to)?
        I'll check again today to see if anyone thing else is funny in my env.

        Show
        jay vyas added a comment - Actually, I tried this one first as non-root. Then as root... I can look some more, but other packages (spark, hadoop,... ) build ; Any other ideas (or if someone else wants to test thats fine to)? I'll check again today to see if anyone thing else is funny in my env.
        Hide
        Olaf Flebbe added a comment -

        Please try as non-root. And show the error message, if any. This non-root requirement is only for tez.

        Show
        Olaf Flebbe added a comment - Please try as non-root. And show the error message, if any. This non-root requirement is only for tez.
        Hide
        jay vyas added a comment -

        Hi Olaf. Ok let me retry as non root.

        Iirc I tried as non root originally but.. I'll try again and paste full output

        Show
        jay vyas added a comment - Hi Olaf. Ok let me retry as non root. Iirc I tried as non root originally but.. I'll try again and paste full output
        Hide
        jay vyas added a comment - - edited

        TL;DR, +1 and great work

        Olaf Flebbe Well, the issue actually wound up being my maven permissions, which seem to have had some corrupted dirs, I deleted those, started over, cloned the source from scratch, and it works on a clean VM, as non-root...
        next step will be

        • puppet recipes and
        • smoke tests.
          please coordinate those follow on JIRAs.
          +1, thanks !
        apache@localhost:~/Development/bigtop-jayunit100[0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch-Fri_Feb__6_21_36_06_EST_2015]$ rpm -qlp ./output/tez/noarch/tez-0.6.0-1.fc20.noarch.rpm 
        /etc/tez/conf/tez-site.xml
        /usr/lib/tez
        /usr/lib/tez/lib
        /usr/lib/tez/lib/commons-cli-1.2.jar
        /usr/lib/tez/lib/commons-codec-1.4.jar
        /usr/lib/tez/lib/commons-collections-3.2.1.jar
        /usr/lib/tez/lib/commons-collections4-4.0.jar
        /usr/lib/tez/lib/commons-io-2.4.jar
        /usr/lib/tez/lib/commons-lang-2.6.jar
        /usr/lib/tez/lib/commons-logging-1.1.3.jar
        /usr/lib/tez/lib/commons-math3-3.1.1.jar
        /usr/lib/tez/lib/guava-11.0.2.jar
        /usr/lib/tez/lib/hadoop-annotations-2.4.1.jar
        /usr/lib/tez/lib/hadoop-mapreduce-client-common-2.4.1.jar
        /usr/lib/tez/lib/hadoop-mapreduce-client-core-2.4.1.jar
        /usr/lib/tez/lib/jersey-client-1.9.jar
        /usr/lib/tez/lib/jersey-json-1.9.jar
        /usr/lib/tez/lib/jettison-1.3.4.jar
        /usr/lib/tez/lib/jsr305-2.0.3.jar
        /usr/lib/tez/lib/log4j-1.2.17.jar
        /usr/lib/tez/lib/protobuf-java-2.5.0.jar
        /usr/lib/tez/tez-api-0.6.0.jar
        /usr/lib/tez/tez-common-0.6.0.jar
        /usr/lib/tez/tez-dag-0.6.0.jar
        /usr/lib/tez/tez-examples-0.6.0.jar
        /usr/lib/tez/tez-mapreduce-0.6.0.jar
        /usr/lib/tez/tez-mbeans-resource-calculator-0.6.0.jar
        /usr/lib/tez/tez-runtime-internals-0.6.0.jar
        /usr/lib/tez/tez-runtime-library-0.6.0.jar
        /usr/lib/tez/tez-tests-0.6.0.jar
        /usr/lib/tez/tez-ui-0.6.0.war
        /usr/lib/tez/tez-yarn-timeline-history-0.6.0.jar
        /usr/share/doc/tez-0.6.0
        /usr/share/man/man1/tez.1.gz
        
        Show
        jay vyas added a comment - - edited TL;DR, +1 and great work Olaf Flebbe Well, the issue actually wound up being my maven permissions, which seem to have had some corrupted dirs, I deleted those, started over, cloned the source from scratch, and it works on a clean VM, as non-root... next step will be puppet recipes and smoke tests. please coordinate those follow on JIRAs. +1, thanks ! apache@localhost:~/Development/bigtop-jayunit100[0001-BIGTOP-1179.-Add-Apache-Tez-to-bigtop.patch-Fri_Feb__6_21_36_06_EST_2015]$ rpm -qlp ./output/tez/noarch/tez-0.6.0-1.fc20.noarch.rpm /etc/tez/conf/tez-site.xml /usr/lib/tez /usr/lib/tez/lib /usr/lib/tez/lib/commons-cli-1.2.jar /usr/lib/tez/lib/commons-codec-1.4.jar /usr/lib/tez/lib/commons-collections-3.2.1.jar /usr/lib/tez/lib/commons-collections4-4.0.jar /usr/lib/tez/lib/commons-io-2.4.jar /usr/lib/tez/lib/commons-lang-2.6.jar /usr/lib/tez/lib/commons-logging-1.1.3.jar /usr/lib/tez/lib/commons-math3-3.1.1.jar /usr/lib/tez/lib/guava-11.0.2.jar /usr/lib/tez/lib/hadoop-annotations-2.4.1.jar /usr/lib/tez/lib/hadoop-mapreduce-client-common-2.4.1.jar /usr/lib/tez/lib/hadoop-mapreduce-client-core-2.4.1.jar /usr/lib/tez/lib/jersey-client-1.9.jar /usr/lib/tez/lib/jersey-json-1.9.jar /usr/lib/tez/lib/jettison-1.3.4.jar /usr/lib/tez/lib/jsr305-2.0.3.jar /usr/lib/tez/lib/log4j-1.2.17.jar /usr/lib/tez/lib/protobuf-java-2.5.0.jar /usr/lib/tez/tez-api-0.6.0.jar /usr/lib/tez/tez-common-0.6.0.jar /usr/lib/tez/tez-dag-0.6.0.jar /usr/lib/tez/tez-examples-0.6.0.jar /usr/lib/tez/tez-mapreduce-0.6.0.jar /usr/lib/tez/tez-mbeans-resource-calculator-0.6.0.jar /usr/lib/tez/tez-runtime-internals-0.6.0.jar /usr/lib/tez/tez-runtime-library-0.6.0.jar /usr/lib/tez/tez-tests-0.6.0.jar /usr/lib/tez/tez-ui-0.6.0.war /usr/lib/tez/tez-yarn-timeline-history-0.6.0.jar /usr/share/doc/tez-0.6.0 /usr/share/man/man1/tez.1.gz
        Hide
        Olaf Flebbe added a comment -

        Will create JIRA's and will commit in a few minutes...

        Show
        Olaf Flebbe added a comment - Will create JIRA's and will commit in a few minutes...
        Hide
        Olaf Flebbe added a comment -

        pushd upstream. Added a bunch of JIRA for configuration and smoke testing

        Show
        Olaf Flebbe added a comment - pushd upstream. Added a bunch of JIRA for configuration and smoke testing

          People

          • Assignee:
            Olaf Flebbe
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development