Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1174

missing dependency setting on init-hdfs.sh in puppet recipes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: deployment
    • Labels:
      None

      Description

      There's no dependency setting to explicitly define init-hdfs.sh to be executed after hdfs get ready, so when running the puppet recipes to deploy a hadoop environment, the init-hdfs.sh script failed to setup folders and permissions on hdfs since it will be executed before namenode and datanode has been started.

      Sample output:

      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/data/1/hdfs]/mode: mode changed '0700' to '0755'
      notice: /Stage[main]/Hadoop_head_node/Exec[init hdfs]/returns: executed successfully
      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/data/2/hdfs]/mode: mode changed '0700' to '0755'
      notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure: ensure changed 'stopped' to 'running'
      notice: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]/ensure: ensure changed 'stopped' to 'running'
      notice: /Stage[main]/Hadoop_head_node/Hadoop-hbase::Master[hbase master]/Service[hbase-master]/ensure: ensure changed 'stopped' to 'running'
      notice: Finished catalog run in 135.44 seconds
      

        Issue Links

          Activity

          Hide
          evans_ye Evans Ye added a comment -

          Sean Mackrory
          To clarify my original thought when providing the patch, the code snippet

          Hadoop::Namenode<||> -> Hadoop::Datanode<||> -> Exec<| title == "init hdfs" |>
          

          chain collectors in the following executing order:
          first, run all the resources found in Hadoop::Namenode namespace
          second, run all the resources found in Hadoop::Datanode namespace
          thrid, run the Exec resource which has title equal to "init hdfs"

          Just In case if you would like to leave any comments on the patch

          Show
          evans_ye Evans Ye added a comment - Sean Mackrory To clarify my original thought when providing the patch, the code snippet Hadoop::Namenode<||> -> Hadoop::Datanode<||> -> Exec<| title == "init hdfs" |> chain collectors in the following executing order : first, run all the resources found in Hadoop::Namenode namespace second, run all the resources found in Hadoop::Datanode namespace thrid, run the Exec resource which has title equal to "init hdfs" Just In case if you would like to leave any comments on the patch
          Hide
          cos Konstantin Boudnik added a comment -

          The fix has been pushed to master on 1/7/2014, but the ticket was never resolved.

          Show
          cos Konstantin Boudnik added a comment - The fix has been pushed to master on 1/7/2014, but the ticket was never resolved.
          Hide
          mackrorysd Sean Mackrory added a comment -

          +1 - works for me. Committed and pushed. Note that I had never seen this syntax before - it's been too long since I touched Puppet - so I don't know for sure there are any negative consequences to the semantics here, so any additional review is more than welcome!

          Show
          mackrorysd Sean Mackrory added a comment - +1 - works for me. Committed and pushed. Note that I had never seen this syntax before - it's been too long since I touched Puppet - so I don't know for sure there are any negative consequences to the semantics here, so any additional review is more than welcome!
          Hide
          jayunit100 jay vyas added a comment -

          Yes i think this is a pretty important patch. i guess, if in place, would have prevented some of my confusion in spinning up the puppetized-vagrant recipe BIGTOP-1171 .

          Show
          jayunit100 jay vyas added a comment - Yes i think this is a pretty important patch. i guess, if in place, would have prevented some of my confusion in spinning up the puppetized-vagrant recipe BIGTOP-1171 .

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development