Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Activity

      Hide
      plinnell Peter Linnell added a comment -

      +1

      Roman, can you hold this commit for a few hours ? It looks like we might get Bigtop-All to complete with all blue in the next few hours .

      Show
      plinnell Peter Linnell added a comment - +1 Roman, can you hold this commit for a few hours ? It looks like we might get Bigtop-All to complete with all blue in the next few hours .
      Hide
      cos Konstantin Boudnik added a comment -

      Damn Whirr is crapping out again with PermGen problem. Shall we just remove it from the stack?

      Show
      cos Konstantin Boudnik added a comment - Damn Whirr is crapping out again with PermGen problem. Shall we just remove it from the stack?
      Hide
      rguo Guo Ruijing added a comment -

      why not bump version to 2.4.1

      Show
      rguo Guo Ruijing added a comment - why not bump version to 2.4.1
      Hide
      rvs Roman Shaposhnik added a comment -

      Looking at 2.4.1 RELNOTES it does make sense: http://hadoop.apache.org/releases.html#30+June%2C+2014%3A+Release+2.4.1+available

      I think I'll do that (unless folks complain too loudly). Thanks for pointing it out.

      Show
      rvs Roman Shaposhnik added a comment - Looking at 2.4.1 RELNOTES it does make sense: http://hadoop.apache.org/releases.html#30+June%2C+2014%3A+Release+2.4.1+available I think I'll do that (unless folks complain too loudly). Thanks for pointing it out.
      Hide
      rvs Roman Shaposhnik added a comment -

      Konstantin Boudnik would you mind starting a thread on dev@bigtop.a.o about the Whirr's fate?

      Show
      rvs Roman Shaposhnik added a comment - Konstantin Boudnik would you mind starting a thread on dev@bigtop.a.o about the Whirr's fate?
      Hide
      plinnell Peter Linnell added a comment -

      Looking at the release notes, this looks good to take.. All fixes, no new features.

      Show
      plinnell Peter Linnell added a comment - Looking at the release notes, this looks good to take.. All fixes, no new features.
      Hide
      plinnell Peter Linnell added a comment -

      Roman Shaposhnik I just poked dev about Whirr.

      Show
      plinnell Peter Linnell added a comment - Roman Shaposhnik I just poked dev about Whirr.
      Hide
      cos Konstantin Boudnik added a comment -

      Thanks Peter Linnell!

      Show
      cos Konstantin Boudnik added a comment - Thanks Peter Linnell !
      Hide
      cos Konstantin Boudnik added a comment -

      So, from the last trunk build it looks like switching to Hadoop 2.4.0 breaks Hive, Oozie, and Spark: seemingly a bunch of incompatible API changes in the Hadoop (YARN) code. Again ;( Shall we still with the original plan for now?

      Show
      cos Konstantin Boudnik added a comment - So, from the last trunk build it looks like switching to Hadoop 2.4.0 breaks Hive, Oozie, and Spark: seemingly a bunch of incompatible API changes in the Hadoop (YARN) code. Again ;( Shall we still with the original plan for now?
      Hide
      rvs Roman Shaposhnik added a comment -

      Seem like 2.4.1 is actually a requirement, not an option. Thanks to Konstantin Boudnik for suggestion on how to deal with YARN failures. I'll be attaching 2.4.1 patch shortly.

      Show
      rvs Roman Shaposhnik added a comment - Seem like 2.4.1 is actually a requirement, not an option. Thanks to Konstantin Boudnik for suggestion on how to deal with YARN failures. I'll be attaching 2.4.1 patch shortly.
      Hide
      cos Konstantin Boudnik added a comment -

      Oh, great! Thanks for figuring out that 2.4.0 release was a dead-drop to start with.

      Show
      cos Konstantin Boudnik added a comment - Oh, great! Thanks for figuring out that 2.4.0 release was a dead-drop to start with.
      Hide
      jayunit100 jay vyas added a comment - - edited

      One other possible wrinkle: Is linux container executor supported on bigtop? If so kerberos will also now be a requirement FYI . you can see http://jayunit100.blogspot.com/2014/04/sally-is-nobody.html for details. this has a major impact on bigtop on alt FileSystems as well, because now to run a linux container (which posix file systems requires because of UID ), you actually must run kerberized security mode (actually, there are workarounds, like bundling an alternate container executor and using that, we can use that in bigtop also by creating a custom jar)... but either way you cant easily let yarn spin up arbitrary linux containers willy-nilly anymore wihtout kerberos.

      In any case, if this IS an issue we can use freeIPA recipes in the bigtop stack. They make kerberos alot easier imo.

      Show
      jayunit100 jay vyas added a comment - - edited One other possible wrinkle: Is linux container executor supported on bigtop? If so kerberos will also now be a requirement FYI . you can see http://jayunit100.blogspot.com/2014/04/sally-is-nobody.html for details. this has a major impact on bigtop on alt FileSystems as well, because now to run a linux container (which posix file systems requires because of UID ), you actually must run kerberized security mode (actually, there are workarounds, like bundling an alternate container executor and using that, we can use that in bigtop also by creating a custom jar)... but either way you cant easily let yarn spin up arbitrary linux containers willy-nilly anymore wihtout kerberos. In any case, if this IS an issue we can use freeIPA recipes in the bigtop stack. They make kerberos alot easier imo.
      Hide
      cos Konstantin Boudnik added a comment -

      you cant easily let yarn spin up arbitrary linux containers willy-nilly anymore wihtout kerberos

      Another solution - perhaps radical - would be to drop the support of YARN completely. After all, Spark works fine as a standalone system, hence a superior MapReduce model will be supported. When we add Gridgain, which doesn't care about YARN neither, we'll have in-memory OLTP and other things. Perhaps, this will help to finally produce a sane implementation of LinuxTaskController?

      Show
      cos Konstantin Boudnik added a comment - you cant easily let yarn spin up arbitrary linux containers willy-nilly anymore wihtout kerberos Another solution - perhaps radical - would be to drop the support of YARN completely. After all, Spark works fine as a standalone system, hence a superior MapReduce model will be supported. When we add Gridgain, which doesn't care about YARN neither, we'll have in-memory OLTP and other things. Perhaps, this will help to finally produce a sane implementation of LinuxTaskController?
      Hide
      rvs Roman Shaposhnik added a comment -

      Attaching another patch for 2.4.1. Tried Oozie/Spark/Hive

      Show
      rvs Roman Shaposhnik added a comment - Attaching another patch for 2.4.1. Tried Oozie/Spark/Hive
      Hide
      cos Konstantin Boudnik added a comment -

      +1

      Show
      cos Konstantin Boudnik added a comment - +1
      Hide
      plinnell Peter Linnell added a comment -

      jay vyas I do not think we explicitly enable container support yet, but I could be wrong. Thanks for digging up the info tho. We would need to consider that for the future for sure and I'd love to avoid having kerberos as mandatory.

      Show
      plinnell Peter Linnell added a comment - jay vyas I do not think we explicitly enable container support yet, but I could be wrong. Thanks for digging up the info tho. We would need to consider that for the future for sure and I'd love to avoid having kerberos as mandatory.
      Hide
      cos Konstantin Boudnik added a comment -

      Committed as 0a02b1a

      Show
      cos Konstantin Boudnik added a comment - Committed as 0a02b1a
      Hide
      rvs Roman Shaposhnik added a comment -

      jay vyas this was my doing with LCE The motivation back then was that you would either use native Hadoop executor without security or LCE with security on. I think you bring up a good point with non-HDFS HCFSs. Now that said, it comes with security implications described in a long thread here YARN-2424

      It now appears that YARN-2424 was committed and will be releases in 2.6. Still I'd be cautious about enabling this in Bigtop.

      At any rate, this type of topic would be better off on dev@ I think.

      Show
      rvs Roman Shaposhnik added a comment - jay vyas this was my doing with LCE The motivation back then was that you would either use native Hadoop executor without security or LCE with security on. I think you bring up a good point with non-HDFS HCFSs. Now that said, it comes with security implications described in a long thread here YARN-2424 It now appears that YARN-2424 was committed and will be releases in 2.6. Still I'd be cautious about enabling this in Bigtop. At any rate, this type of topic would be better off on dev@ I think.

        People

        • Assignee:
          rvs Roman Shaposhnik
          Reporter:
          rvs Roman Shaposhnik
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development