Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1502

Improve puppet deployment for new gridgain component

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Let's have puppet recipes that will automatically configure and bring up Gridgain caching layer

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Once the component is committed I will take care about the puppet part

          Show
          cos Konstantin Boudnik added a comment - Once the component is committed I will take care about the puppet part
          Hide
          rvs Roman Shaposhnik added a comment -

          Would love to see a patch for this one, since looking at Puppet code is the easiest way for me to understand the operational characteristics of a piece of software.

          Show
          rvs Roman Shaposhnik added a comment - Would love to see a patch for this one, since looking at Puppet code is the easiest way for me to understand the operational characteristics of a piece of software.
          Hide
          cos Konstantin Boudnik added a comment -

          Ilya Tikhonov, as I've asked here - could you provide more detailed info on the configuration of the daemons, so I can get this JIRA going?

          Thanks!

          Show
          cos Konstantin Boudnik added a comment - Ilya Tikhonov , as I've asked here - could you provide more detailed info on the configuration of the daemons, so I can get this JIRA going? Thanks!
          Hide
          pelya Ilya Tikhonov added a comment -

          Sorry for delay, Konstantin Boudnik.

          I attached document. It's draft and it's not been published yet. This describes top-level concept of using GGHA. Please read this first.

          Some additional comments:

          The both processors (MR and GGFS) are running by default. It doesn't need to configure it if you don't need to use complicated dual modes of GGFS.

          More detailed info about this read here:
          http://hadoop.gridgain.org/latest/GGFS
          http://hadoop.gridgain.org/latest/Configuration

          GridGain cluster nodes don't have master and slave roles. By default all nodes find each other in current subnetwork and each node can play master role.

          I think the main issue is the classpath for Hadoop components. These jars must be added to all client application and yarn nodes:
          gridgain-core-<version>.jar
          gridgain-hadoop-<version>.jar

          Outside of BigTop we usual recommend to add links to this jars into <hadoop_distribution>/share/hadoop/common/lib directory.

          And about memory usage.
          By default GGFS uses only heap memory is allocated in Xmx java option and MR generally uses off-heap memory. To change this value, you should export JVM_OPTS with all options are defined in bin/include/service.sh and change Xmx value if it's needed.

          Show
          pelya Ilya Tikhonov added a comment - Sorry for delay, Konstantin Boudnik . I attached document. It's draft and it's not been published yet. This describes top-level concept of using GGHA. Please read this first. Some additional comments: The both processors (MR and GGFS) are running by default. It doesn't need to configure it if you don't need to use complicated dual modes of GGFS. More detailed info about this read here: http://hadoop.gridgain.org/latest/GGFS http://hadoop.gridgain.org/latest/Configuration GridGain cluster nodes don't have master and slave roles. By default all nodes find each other in current subnetwork and each node can play master role. I think the main issue is the classpath for Hadoop components. These jars must be added to all client application and yarn nodes: gridgain-core-<version>.jar gridgain-hadoop-<version>.jar Outside of BigTop we usual recommend to add links to this jars into <hadoop_distribution>/share/hadoop/common/lib directory. And about memory usage. By default GGFS uses only heap memory is allocated in Xmx java option and MR generally uses off-heap memory. To change this value, you should export JVM_OPTS with all options are defined in bin/include/service.sh and change Xmx value if it's needed.
          Hide
          cos Konstantin Boudnik added a comment -

          Thanks Ilya - I will give it a try over the weekend.

          Show
          cos Konstantin Boudnik added a comment - Thanks Ilya - I will give it a try over the weekend.
          Hide
          jayunit100 jay vyas added a comment -

          hi Konstantin Boudnik , just a quick note - maybe you guys already know this, but I dont see a gridgain hook in cluster.pp .... So i dont think it conforms to our way of installing components. I assume that Is that what you are looking into ? but just figured id chime in

          Show
          jayunit100 jay vyas added a comment - hi Konstantin Boudnik , just a quick note - maybe you guys already know this, but I dont see a gridgain hook in cluster.pp .... So i dont think it conforms to our way of installing components. I assume that Is that what you are looking into ? but just figured id chime in
          Hide
          cos Konstantin Boudnik added a comment -

          That's why this ticket exists, Jay

          Show
          cos Konstantin Boudnik added a comment - That's why this ticket exists, Jay
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, I think I mostly have everything in place, but I would need to generate a separate core-site.xml and mapreduce-site.xml for the client applications, so that HDFS/MR components configs aren't affected.

          Show
          cos Konstantin Boudnik added a comment - Ok, I think I mostly have everything in place, but I would need to generate a separate core-site.xml and mapreduce-site.xml for the client applications, so that HDFS/MR components configs aren't affected.
          Hide
          plinnell Peter Linnell added a comment -

          +1

          Show
          plinnell Peter Linnell added a comment - +1
          Hide
          cos Konstantin Boudnik added a comment -

          Committed and pushed

          Show
          cos Konstantin Boudnik added a comment - Committed and pushed

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              cos Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development