Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1235

Speed-up init-hdfs.sh for complete HCFS compliant provisioning

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment, general
    • Labels:
      None

      Description

      BIGTOP-1200 provides file schema in JSON format, for any HCFS related tool to read.

      BIGTOP-952 meanwhile give us fast provisioning by running in a single JVM (without multiple haoop fs -... commands_).

      Thus, when those two are finished, we can replace init-hdfs.sh (with the exception of a few extra shell commands in bottom of that script which still will need to be called).

      So this JIRA will:

      • create a new init-hcfs.groovy script which calls provision.groovy in BIGTOP-952, with init-hcfs.json as input.
      • add in the remaining straggler commands from init-hdfs.sh which dont fit into the generic json format as shell commands.
      • delete init-hdfs.sh entirely
      • update puppet scripts to use init-hcfs.groovy, calling them with the "hdfs" argument.

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment - - edited

          Update:

          • I've integrated the "oozie straggler" commands which arent in BIGTOP-1200 as pure shell calls. So no need for any extra commands in this JIRA.
          • Once those go in they can serve as a prototypical replacement for init-hdfs.sh,

          Sp the main task in this JIRA will be:

          • Update puppet provisioners
          • Make sure that groovy and is available at setup time .
          • Test it all by provisioning a hadoop system from scratch, without init-hdfs.sh
          • delete init-hdfs.sh
          • verify usage of "HCFS_SUPER_USER" parameter so that permissions can different depending on root_user value. the init-hcfs.json file which bigtop ships with needs to go through another round of validation.
          Show
          jayunit100 jay vyas added a comment - - edited Update: Now there are patches in for BIGTOP-1200 and BIGTOP-952 . I've integrated the "oozie straggler" commands which arent in BIGTOP-1200 as pure shell calls. So no need for any extra commands in this JIRA. Once those go in they can serve as a prototypical replacement for init-hdfs.sh, Sp the main task in this JIRA will be: Update puppet provisioners Make sure that groovy and is available at setup time . Test it all by provisioning a hadoop system from scratch, without init-hdfs.sh delete init-hdfs.sh verify usage of "HCFS_SUPER_USER" parameter so that permissions can different depending on root_user value. the init-hcfs.json file which bigtop ships with needs to go through another round of validation.
          Hide
          cos Konstantin Boudnik added a comment -

          Sounds like a good plan! Thanks Jay!

          Show
          cos Konstantin Boudnik added a comment - Sounds like a good plan! Thanks Jay!
          Hide
          jayunit100 jay vyas added a comment -

          Pinging the usual suspects who review my patches Konstantin Boudnik Bruno Mahé Mark Grover......... Can we push BIGTOP-1178 in ? I think its ready to roll and can be integral to the testing and validating this

          Show
          jayunit100 jay vyas added a comment - Pinging the usual suspects who review my patches Konstantin Boudnik Bruno Mahé Mark Grover ......... Can we push BIGTOP-1178 in ? I think its ready to roll and can be integral to the testing and validating this
          Hide
          cos Konstantin Boudnik added a comment -

          Jay, I will try to look into this later in the day. I am not a Vagrant expert - not by a long stretch, but I will try to

          Show
          cos Konstantin Boudnik added a comment - Jay, I will try to look into this later in the day. I am not a Vagrant expert - not by a long stretch, but I will try to
          Hide
          jayunit100 jay vyas added a comment - - edited

          Updated task list from mailing list:

          • Update puppet provisioners
          • Determine what path the provision.groovy script is written to in the real clusters `(/usr/lib/bigtop-utils/)` or somethhing like that?
          • Make sure that groovy is indeed available at setup time .
          • Test it all by provisioning a hadoop system from scratch, without init-hdfs.sh
          • delete init-hdfs.sh
          • rename provision.groovy to init-hcfs.groovy
          • verify usage of "HCFS_SUPER_USER" parameter so that permissions can different depending on root_user value. the init-hcfs.json file which bigtop ships with needs to go through another round of validation.
          Show
          jayunit100 jay vyas added a comment - - edited Updated task list from mailing list: Update puppet provisioners Determine what path the provision.groovy script is written to in the real clusters `(/usr/lib/bigtop-utils/)` or somethhing like that? Make sure that groovy is indeed available at setup time . Test it all by provisioning a hadoop system from scratch, without init-hdfs.sh delete init-hdfs.sh rename provision.groovy to init-hcfs.groovy verify usage of "HCFS_SUPER_USER" parameter so that permissions can different depending on root_user value. the init-hcfs.json file which bigtop ships with needs to go through another round of validation.
          Hide
          cos Konstantin Boudnik added a comment -

          Update puppet provisioners

          Determine what path the provision.groovy script is written to in the real clusters `(/usr/lib/bigtop-utils/)` or somethhing like that?

          I'd suggest to change hadoop packaging to include new provisioning script into it - it rather be more straight forward, no? This will also address the availability of the groovy runtime, because we'd be able to express it via dependency declaration.

          Show
          cos Konstantin Boudnik added a comment - Update puppet provisioners Determine what path the provision.groovy script is written to in the real clusters `(/usr/lib/bigtop-utils/)` or somethhing like that? I'd suggest to change hadoop packaging to include new provisioning script into it - it rather be more straight forward, no? This will also address the availability of the groovy runtime, because we'd be able to express it via dependency declaration.
          Hide
          jayunit100 jay vyas added a comment -

          Okay I willl map out how the hadoop packaging works . . . probably in the process will update the wiki page.

          Show
          jayunit100 jay vyas added a comment - Okay I willl map out how the hadoop packaging works . . . probably in the process will update the wiki page.
          Hide
          jayunit100 jay vyas added a comment -

          So, This patch will basically have two types of changes (1) to packaging and (2) to the puppet provisioning steps, and attached the first attempt at part(1).

          • removes init-hdfs.sh
          • renames provision.groovy to init-hcfs.groovy
          • updates bigtop-utils packaging, so that init-hcfs.groovy will be available in /usr/lib/bigtop-utils/

          I'll look again at this stuff later but just wanted to post an update about where this is at. Any feedback is welcome: Im not a packaging expert so might have mucked some stuff up. Either way ill test this all before submitting for a formal review.

          Show
          jayunit100 jay vyas added a comment - So, This patch will basically have two types of changes (1) to packaging and (2) to the puppet provisioning steps, and attached the first attempt at part(1). removes init-hdfs.sh renames provision.groovy to init-hcfs.groovy updates bigtop-utils packaging, so that init-hcfs.groovy will be available in /usr/lib/bigtop-utils/ I'll look again at this stuff later but just wanted to post an update about where this is at. Any feedback is welcome: Im not a packaging expert so might have mucked some stuff up. Either way ill test this all before submitting for a formal review.
          Hide
          mgrover Mark Grover added a comment -

          I would prefer if we didn't point blank remove init-hdfs. Perhaps, we could ship both simultaneously for a release or two (deprecating the old one) and then get rid of the older one.

          Show
          mgrover Mark Grover added a comment - I would prefer if we didn't point blank remove init-hdfs. Perhaps, we could ship both simultaneously for a release or two (deprecating the old one) and then get rid of the older one.
          Hide
          jayunit100 jay vyas added a comment -

          sure, we can still keep the script and it can be a facade to the new way of doing things, or even leave it unchanged for the time being

          Show
          jayunit100 jay vyas added a comment - sure, we can still keep the script and it can be a facade to the new way of doing things, or even leave it unchanged for the time being
          Hide
          mgrover Mark Grover added a comment -

          +1

          Show
          mgrover Mark Grover added a comment - +1
          Hide
          jayunit100 jay vyas added a comment - - edited

          I tried testing (my first attempt at any non trivial bigtop packaging mods), and get a failure. after trying to reproduce the command, it seems that the command gradle is running is like this :

           
          rpmbuild --define _topdir /home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/ --define bigtop_utils_base_version 0.9.0 --define bigtop_utils_version 0.9.0 --define bigtop_utils_release 1%{?dist} -bs --nodeps --buildroot=/home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/INSTALL /home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/SPECS/bigtop-utils.spec
          

          The error message i get when i run that command manually:

          error: Macro %_topdir has empty body
          error: Macro %_topdir has empty body
          error: Macro %_topdir has empty body
          error: Macro %bigtop_utils_base_version has empty body
          error: Macro %bigtop_utils_base_version has empty body
          error: Macro %bigtop_utils_version has empty body
          error: Macro %bigtop_utils_version has empty body
          error: Macro %bigtop_utils_release has empty body
          error: Macro %bigtop_utils_release has empty body
          

          will look more tomorrow. probably a dumb error if anyone can give me a hint what i messed up would appreciate it

          ...........

          nevermind, looks like it was just a m2_home issue on my VM ill carry on hacking with this . might have some cycles to finally get it done .

          Show
          jayunit100 jay vyas added a comment - - edited I tried testing ( my first attempt at any non trivial bigtop packaging mods ), and get a failure. after trying to reproduce the command, it seems that the command gradle is running is like this : rpmbuild --define _topdir /home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/ --define bigtop_utils_base_version 0.9.0 --define bigtop_utils_version 0.9.0 --define bigtop_utils_release 1%{?dist} -bs --nodeps --buildroot=/home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/INSTALL /home/apache/Development/bigtop-jayunit100/build/bigtop-utils/rpm/SPECS/bigtop-utils.spec The error message i get when i run that command manually: error: Macro %_topdir has empty body error: Macro %_topdir has empty body error: Macro %_topdir has empty body error: Macro %bigtop_utils_base_version has empty body error: Macro %bigtop_utils_base_version has empty body error: Macro %bigtop_utils_version has empty body error: Macro %bigtop_utils_version has empty body error: Macro %bigtop_utils_release has empty body error: Macro %bigtop_utils_release has empty body will look more tomorrow. probably a dumb error if anyone can give me a hint what i messed up would appreciate it ........... nevermind, looks like it was just a m2_home issue on my VM ill carry on hacking with this . might have some cycles to finally get it done .
          Hide
          jayunit100 jay vyas added a comment - - edited

          just an update of my WIP, not ready yet. now the init-hcfs.groovy is packaged in bigtop-utils . next have to verify that

          • the rpmified groovy provisioner , after being yum installed, can executed
          • ensure that the init-hcfs.groovy file is bundled as well
          • modify puppet to properly call init-hcfs.groovy
          Show
          jayunit100 jay vyas added a comment - - edited just an update of my WIP, not ready yet. now the init-hcfs.groovy is packaged in bigtop-utils . next have to verify that the rpmified groovy provisioner , after being yum installed, can executed ensure that the init-hcfs.groovy file is bundled as well modify puppet to properly call init-hcfs.groovy
          Hide
          jayunit100 jay vyas added a comment - - edited

          Konstantin Boudnik need some help : What way do you suggest to run the groovy hcfs file system setup ? right now the only way is to manually add on classpath elements .

          groovy -classpath /usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar
              :/usr/lib/hadoop/lib/guava-11.0.2.jar
              :/etc/hadoop/conf/:/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar
              :/usr/lib/hadoop/lib/commons-configuration-1.6.jar
              :/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/hadoop-auth.jar
              :/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar
              :/usr/lib/hadoop-hdfs/hadoop-hdfs.jar
              :/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar /vagrant/provision.groovy 
               init-hcfs.json
          

          any better ideas for the groovy classopath for file system provisioning?

          Show
          jayunit100 jay vyas added a comment - - edited Konstantin Boudnik need some help : What way do you suggest to run the groovy hcfs file system setup ? right now the only way is to manually add on classpath elements . groovy -classpath /usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar :/usr/lib/hadoop/lib/guava-11.0.2.jar :/etc/hadoop/conf/:/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar :/usr/lib/hadoop/lib/commons-configuration-1.6.jar :/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/hadoop-auth.jar :/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar :/usr/lib/hadoop-hdfs/hadoop-hdfs.jar :/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar /vagrant/provision.groovy init-hcfs.json any better ideas for the groovy classopath for file system provisioning?
          Hide
          cos Konstantin Boudnik added a comment -

          Do you think you can use Groovy RootLoader ?

          Show
          cos Konstantin Boudnik added a comment - Do you think you can use Groovy RootLoader ?
          Hide
          cos Konstantin Boudnik added a comment -

          Are we going anywhere with it? I really start hating init-hdfs.sh ;(

          Show
          cos Konstantin Boudnik added a comment - Are we going anywhere with it? I really start hating init-hdfs.sh ;(
          Hide
          cos Konstantin Boudnik added a comment -

          Also, for compatibly reasons, shall we convert init-hdfs.sh into a wrapper around new init-hcfs.groovy ?

          And BTW, I see that provision.groovy is referenced from a couple of places yet the patch seems to be deleting it actually?

          Show
          cos Konstantin Boudnik added a comment - Also, for compatibly reasons, shall we convert init-hdfs.sh into a wrapper around new init-hcfs.groovy ? And BTW, I see that provision.groovy is referenced from a couple of places yet the patch seems to be deleting it actually?
          Hide
          jayunit100 jay vyas added a comment - - edited

          hi Konstantin Boudnik ... this is very close to being done

          • replacing init-hdfs.sh can be done easily, just by using the results of BIGTOP-952, which is already in bigtop.
          • this patch, i was confused about the right way to to integrate groovy, modify spec files to package it, and so on. now that i understand bigtop better, i think its really close to being done.
          • you're right, we do delete provision.groovy but i think thats b/c we are copying it to init-hcfs.groovy for consistency
          • yes not a bad idea to have init-hdfs.sh wrap init-hcfs.groovy with hdfs semantics.
          Show
          jayunit100 jay vyas added a comment - - edited hi Konstantin Boudnik ... this is very close to being done replacing init-hdfs.sh can be done easily, just by using the results of BIGTOP-952 , which is already in bigtop. this patch, i was confused about the right way to to integrate groovy, modify spec files to package it, and so on. now that i understand bigtop better, i think its really close to being done. you're right, we do delete provision.groovy but i think thats b/c we are copying it to init-hcfs.groovy for consistency yes not a bad idea to have init-hdfs.sh wrap init-hcfs.groovy with hdfs semantics.
          Hide
          cos Konstantin Boudnik added a comment -

          you're right, we do delete provision.groovy but i think thats b/c we are copying it to init-hcfs.groovy for consistency

          then it makes sense to remove the references to provision.groovy in the comments, etc., no?

          Show
          cos Konstantin Boudnik added a comment - you're right, we do delete provision.groovy but i think thats b/c we are copying it to init-hcfs.groovy for consistency then it makes sense to remove the references to provision.groovy in the comments, etc., no?
          Hide
          jayunit100 jay vyas added a comment -

          yes definitely that should also be cleaned up... i would have to apply this patch and get my head back into it to really answer intelligibly.

          maybe i can try to get it working over the holidays ? or someone else can take it over as well by applying the patch . all the components are pretty much there... if nobody else jumps on it, i'll bump it up the stack a few notches

          Show
          jayunit100 jay vyas added a comment - yes definitely that should also be cleaned up... i would have to apply this patch and get my head back into it to really answer intelligibly. maybe i can try to get it working over the holidays ? or someone else can take it over as well by applying the patch . all the components are pretty much there... if nobody else jumps on it, i'll bump it up the stack a few notches
          Hide
          cos Konstantin Boudnik added a comment -

          lemme see if I can give it a spin in the next a couple of days... If not - your offer stands

          Show
          cos Konstantin Boudnik added a comment - lemme see if I can give it a spin in the next a couple of days... If not - your offer stands
          Hide
          cos Konstantin Boudnik added a comment -

          Here's the version that uses init-hdfs.sh to wrap up the settings for CLASSPATH and then kick-off the new Groovy-based file-system provisioning. The script part works ok.

          I am not sure if we should have make bigtop-utils to be dependent on bigtop-groovy at this point...? Opinions?

          Show
          cos Konstantin Boudnik added a comment - Here's the version that uses init-hdfs.sh to wrap up the settings for CLASSPATH and then kick-off the new Groovy-based file-system provisioning. The script part works ok. I am not sure if we should have make bigtop-utils to be dependent on bigtop-groovy at this point...? Opinions?
          Hide
          cos Konstantin Boudnik added a comment -

          An improvement to allow to work without a terminal

          Show
          cos Konstantin Boudnik added a comment - An improvement to allow to work without a terminal
          Hide
          jayunit100 jay vyas added a comment - - edited
          • sure, I like skipping a groovy dependency .
          • To be clear though, We need the groovy scripts in the rpms or else in the puppet folder itself. Right ? if there is a way to do this w/o rpm wrangling that might be nice. i didnt even look at the deb side yet
          Show
          jayunit100 jay vyas added a comment - - edited sure, I like skipping a groovy dependency . To be clear though, We need the groovy scripts in the rpms or else in the puppet folder itself. Right ? if there is a way to do this w/o rpm wrangling that might be nice. i didnt even look at the deb side yet
          Hide
          cos Konstantin Boudnik added a comment -

          I have added the groovy script to the RPM spec - totally blanked on deb side, will fix it shortly. And I need to also add the json definition as well, darn...

          IMO the package should be fully self-sufficient like it is right now. We are only calling init-hdfs.sh script from Puppet and it does the magic. I think preserving the same interface while improving on the performance will be nice. And that what I've tried to do with wrapping the classpath settings into init-hdfs.sh and then passing the control to the new groovy thing. Does it make sense?

          Show
          cos Konstantin Boudnik added a comment - I have added the groovy script to the RPM spec - totally blanked on deb side, will fix it shortly. And I need to also add the json definition as well, darn... IMO the package should be fully self-sufficient like it is right now. We are only calling init-hdfs.sh script from Puppet and it does the magic. I think preserving the same interface while improving on the performance will be nice. And that what I've tried to do with wrapping the classpath settings into init-hdfs.sh and then passing the control to the new groovy thing. Does it make sense?
          Hide
          jayunit100 jay vyas added a comment - - edited

          makes sense. now, just thinking out loud, testing will be quite difficult. ill try to outline steps here:

          1) enable yum output/ option for vagrant
          2) run gradle to build bigtop packages for yarn, hadoop, and so on
          3) spin up bigtop vms
          4) confirm that init-hfcs is running
          5) run smoke tests (4 and 5 will be automatically run from vagrant, but just listing them for clarity.)

          thanks for helping me with this. am quite distracted lots of snow and traveling this wk so the I'm not 100% sure i remember the details of how it all is working from the last time i worked on it.

          Show
          jayunit100 jay vyas added a comment - - edited makes sense. now, just thinking out loud, testing will be quite difficult. ill try to outline steps here: 1) enable yum output/ option for vagrant 2) run gradle to build bigtop packages for yarn, hadoop, and so on 3) spin up bigtop vms 4) confirm that init-hfcs is running 5) run smoke tests (4 and 5 will be automatically run from vagrant, but just listing them for clarity.) thanks for helping me with this. am quite distracted lots of snow and traveling this wk so the I'm not 100% sure i remember the details of how it all is working from the last time i worked on it.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Hi, would be nice if one can copy jar's to the HDFS as well. Need this for configuring tez.

          Show
          oflebbe Olaf Flebbe added a comment - Hi, would be nice if one can copy jar's to the HDFS as well. Need this for configuring tez.
          Hide
          jayunit100 jay vyas added a comment - - edited

          copy what jars to hfds? possibly modifying init-hfcs.sh ? we can do that in a sep lira. thats a good idea maybe there will be other jars we need to copy to pdfs

          Show
          jayunit100 jay vyas added a comment - - edited copy what jars to hfds? possibly modifying init-hfcs.sh ? we can do that in a sep lira. thats a good idea maybe there will be other jars we need to copy to pdfs
          Hide
          oflebbe Olaf Flebbe added a comment -

          Tez: resides in HDFS

          In our installation tez lives in /apps/tez in HDFS . Installation guide seems to recommend improved installation, still have to dig into it.
          http://tez.apache.org/install.html

          Show
          oflebbe Olaf Flebbe added a comment - Tez: resides in HDFS In our installation tez lives in /apps/tez in HDFS . Installation guide seems to recommend improved installation, still have to dig into it. http://tez.apache.org/install.html
          Hide
          cos Konstantin Boudnik added a comment -

          This new Groovy script does it for Oozie. There's a method copyJars that is controlled from json layout. So I guess your request will only need a change in init-hcfs.json file

          Show
          cos Konstantin Boudnik added a comment - This new Groovy script does it for Oozie. There's a method copyJars that is controlled from json layout. So I guess your request will only need a change in init-hcfs.json file
          Hide
          cos Konstantin Boudnik added a comment -

          jay vyas, shall I finish this or you plan to get it done soon?

          Show
          cos Konstantin Boudnik added a comment - jay vyas , shall I finish this or you plan to get it done soon?
          Hide
          jayunit100 jay vyas added a comment -

          Go for it ! I'm totally overcommited on a few things at work right now. I will test it for you if need be, and feel free to troll me in here or even call me if need be.

          Show
          jayunit100 jay vyas added a comment - Go for it ! I'm totally overcommited on a few things at work right now. I will test it for you if need be, and feel free to troll me in here or even call me if need be.
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, I will take it then.

          Show
          cos Konstantin Boudnik added a comment - Ok, I will take it then.
          Hide
          cos Konstantin Boudnik added a comment -

          Rebasing the patch after BIGTOP-1648.

          Show
          cos Konstantin Boudnik added a comment - Rebasing the patch after BIGTOP-1648 .
          Hide
          cos Konstantin Boudnik added a comment -

          Few more fixes in the groovy script itself and now it seems to be all working. I am doing fresh rebuild to have some final checks, but it looks good now and works FAST

          One new dependency: bigtop-utils is now requires bigtop-groovy for an obvious reason

          Show
          cos Konstantin Boudnik added a comment - Few more fixes in the groovy script itself and now it seems to be all working. I am doing fresh rebuild to have some final checks, but it looks good now and works FAST One new dependency: bigtop-utils is now requires bigtop-groovy for an obvious reason
          Hide
          jayunit100 jay vyas added a comment -

          cool. is this now a brand new patch or based on the original one? Do you need me to test it ? is it ready for putting into bigtop ?

          Show
          jayunit100 jay vyas added a comment - cool. is this now a brand new patch or based on the original one? Do you need me to test it ? is it ready for putting into bigtop ?
          Hide
          cos Konstantin Boudnik added a comment -

          The patch that essentially very close to the original one, but takes care about adding proper bits to the package, making sure that classpath is correct (just found another missed dependency), fixes a few implementation issues in the groovy script. I am rebuilding the stack right now for the final testing (the initial round was successful), but I am pretty confident it is ready for commit. And it surely ready for review.

          Show
          cos Konstantin Boudnik added a comment - The patch that essentially very close to the original one, but takes care about adding proper bits to the package, making sure that classpath is correct (just found another missed dependency), fixes a few implementation issues in the groovy script. I am rebuilding the stack right now for the final testing (the initial round was successful), but I am pretty confident it is ready for commit. And it surely ready for review.
          Hide
          cos Konstantin Boudnik added a comment -

          Found a typo, Fixed.

          So, yes - it is officially working now!

          Show
          cos Konstantin Boudnik added a comment - Found a typo, Fixed. So, yes - it is officially working now!
          Hide
          cos Konstantin Boudnik added a comment -

          Ah, one last: init-hdfs.sh needs to call bigtop-detect-javahome to set JAVA_HOME correcly, otherwise it might not work if that isn't set.

          Show
          cos Konstantin Boudnik added a comment - Ah, one last: init-hdfs.sh needs to call bigtop-detect-javahome to set JAVA_HOME correcly, otherwise it might not work if that isn't set.
          Hide
          jayunit100 jay vyas added a comment - - edited

          okay. BTW, I guess this now makes us (almost) decoupled from HDFS as a requirement for bigtop?

          Show
          jayunit100 jay vyas added a comment - - edited okay. BTW, I guess this now makes us (almost) decoupled from HDFS as a requirement for bigtop?
          Hide
          cos Konstantin Boudnik added a comment -

          Could you elaborate on this?

          And BTW - please review: it is ready to get committed.

          Show
          cos Konstantin Boudnik added a comment - Could you elaborate on this? And BTW - please review: it is ready to get committed.
          Hide
          evans_ye Evans Ye added a comment -

          Hi Konstantin Boudnik.

          I've tested this by Docker provisioner and it failed to deploy because of init-hcfs.json not found.
          The patch needs to add a line in bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec to include a default json file.

          622 %{lib_hadoop}/libexec/init-hdfs.json
          
          Show
          evans_ye Evans Ye added a comment - Hi Konstantin Boudnik . I've tested this by Docker provisioner and it failed to deploy because of init-hcfs.json not found. The patch needs to add a line in bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec to include a default json file. 622 %{lib_hadoop}/libexec/init-hdfs.json
          Hide
          cos Konstantin Boudnik added a comment -

          Actually, that's pretty weird as I have

          install -m 0755 ${DISTRO_DIR}/init-hcfs.json ${SYSTEM_LIBEXEC_DIR}/
          

          in the install-hadoop.sh

          Show
          cos Konstantin Boudnik added a comment - Actually, that's pretty weird as I have install -m 0755 ${DISTRO_DIR}/init-hcfs.json ${SYSTEM_LIBEXEC_DIR}/ in the install-hadoop.sh
          Hide
          cos Konstantin Boudnik added a comment -

          And what's even more weird, is that I have that line locally... ;( Looking into it - good catch.

          Show
          cos Konstantin Boudnik added a comment - And what's even more weird, is that I have that line locally... ;( Looking into it - good catch.
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, this should be correct. Also fixed missing file for deb package. Looks like something is going on with me local repo, as these two changes failed to get detected by git. I had to clone a new one to succeed. Thanks for great catch, Evans!

          Show
          cos Konstantin Boudnik added a comment - Ok, this should be correct. Also fixed missing file for deb package. Looks like something is going on with me local repo, as these two changes failed to get detected by git. I had to clone a new one to succeed. Thanks for great catch, Evans!
          Hide
          jayunit100 jay vyas added a comment -

          +1 ~ thanks for finsihing it off Konstantin Boudnik !
          I would suggest rename this JIRA to "Combine BIGTOP-1200 and BIGTOP-952 for complete HCFS compliant provisioning", since this is much more fundamental architecture improvement than just performance speed up, and ammend commit message before commiting.

          And you can bet there will be tweets going out.

          Show
          jayunit100 jay vyas added a comment - +1 ~ thanks for finsihing it off Konstantin Boudnik ! I would suggest rename this JIRA to "Combine BIGTOP-1200 and BIGTOP-952 for complete HCFS compliant provisioning", since this is much more fundamental architecture improvement than just performance speed up, and ammend commit message before commiting. And you can bet there will be tweets going out.
          Hide
          jayunit100 jay vyas added a comment -

          elaboration on the comment above; Now that we provision via init-hcfs.json, anyone can configure bigtop to use any file system, making it thus the first HCFS compliant distribution, in my mind.

          Show
          jayunit100 jay vyas added a comment - elaboration on the comment above; Now that we provision via init-hcfs.json , anyone can configure bigtop to use any file system, making it thus the first HCFS compliant distribution, in my mind.
          Hide
          evans_ye Evans Ye added a comment -

          Yup, +1. New patch works well on my Docker cluster. The deployment time has been reduced about 4~5min after this applied. Awesome!

          Show
          evans_ye Evans Ye added a comment - Yup, +1. New patch works well on my Docker cluster. The deployment time has been reduced about 4~5min after this applied. Awesome!
          Hide
          cos Konstantin Boudnik added a comment -

          Ah, I caught your drift now, Jay. Renamed the JIRA.
          Thanks for the review Evans Ye - I will commit it shortly.

          Show
          cos Konstantin Boudnik added a comment - Ah, I caught your drift now, Jay. Renamed the JIRA. Thanks for the review Evans Ye - I will commit it shortly.
          Hide
          cos Konstantin Boudnik added a comment -

          The patch is the same as before but reflects the JIRA's name change.

          Show
          cos Konstantin Boudnik added a comment - The patch is the same as before but reflects the JIRA's name change.
          Hide
          cos Konstantin Boudnik added a comment -

          Committed and pushed! Thanks Roman Shaposhnik, jay vyas for contribution of the initial bits!

          Show
          cos Konstantin Boudnik added a comment - Committed and pushed! Thanks Roman Shaposhnik , jay vyas for contribution of the initial bits!

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              jayunit100 jay vyas
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development