Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1174

missing dependency setting on init-hdfs.sh in puppet recipes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: deployment
    • Labels:
      None

      Description

      There's no dependency setting to explicitly define init-hdfs.sh to be executed after hdfs get ready, so when running the puppet recipes to deploy a hadoop environment, the init-hdfs.sh script failed to setup folders and permissions on hdfs since it will be executed before namenode and datanode has been started.

      Sample output:

      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/data/1/hdfs]/mode: mode changed '0700' to '0755'
      notice: /Stage[main]/Hadoop_head_node/Exec[init hdfs]/returns: executed successfully
      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/data/2/hdfs]/mode: mode changed '0700' to '0755'
      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure: ensure changed 'stopped' to 'running'
      notice: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]/ensure: ensure changed 'stopped' to 'running'
      notice: /Stage[main]/Hadoop_head_node/Hadoop-hbase::Master[hbase master]/Service[hbase-master]/ensure: ensure changed 'stopped' to 'running'
      notice: Finished catalog run in 135.44 seconds
      

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          Yes i think this is a pretty important patch. i guess, if in place, would have prevented some of my confusion in spinning up the puppetized-vagrant recipe BIGTOP-1171 .

          Show
          jayunit100 jay vyas added a comment - Yes i think this is a pretty important patch. i guess, if in place, would have prevented some of my confusion in spinning up the puppetized-vagrant recipe BIGTOP-1171 .
          Hide
          mackrorysd Sean Mackrory added a comment -

          +1 - works for me. Committed and pushed. Note that I had never seen this syntax before - it's been too long since I touched Puppet - so I don't know for sure there are any negative consequences to the semantics here, so any additional review is more than welcome!

          Show
          mackrorysd Sean Mackrory added a comment - +1 - works for me. Committed and pushed. Note that I had never seen this syntax before - it's been too long since I touched Puppet - so I don't know for sure there are any negative consequences to the semantics here, so any additional review is more than welcome!
          Hide
          cos Konstantin Boudnik added a comment -

          The fix has been pushed to master on 1/7/2014, but the ticket was never resolved.

          Show
          cos Konstantin Boudnik added a comment - The fix has been pushed to master on 1/7/2014, but the ticket was never resolved.
          Hide
          evans_ye Evans Ye added a comment -

          Sean Mackrory
          To clarify my original thought when providing the patch, the code snippet

          Hadoop::Namenode<||> -> Hadoop::Datanode<||> -> Exec<| title == "init hdfs" |>
          

          chain collectors in the following executing order:
          first, run all the resources found in Hadoop::Namenode namespace
          second, run all the resources found in Hadoop::Datanode namespace
          thrid, run the Exec resource which has title equal to "init hdfs"

          Just In case if you would like to leave any comments on the patch

          Show
          evans_ye Evans Ye added a comment - Sean Mackrory To clarify my original thought when providing the patch, the code snippet Hadoop::Namenode<||> -> Hadoop::Datanode<||> -> Exec<| title == "init hdfs" |> chain collectors in the following executing order : first, run all the resources found in Hadoop::Namenode namespace second, run all the resources found in Hadoop::Datanode namespace thrid, run the Exec resource which has title equal to "init hdfs" Just In case if you would like to leave any comments on the patch

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development