Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1686

puppet hadoop/mapred: Update and clean mapred template and namespace

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Update the mapred-site.xml template to current names for settings.
      Remove outdated settings. Make more settings adjustable and move their
      defaults into the Puppet class.

      Change variable names to consistently resemble the setting names.

        Activity

        Hide
        michaelweiser Michael Weiser added a comment -

        This is the first installment of the promised post-hiera-conversion cleanups:

        Olaf Flebbe has updated the mapred-site.xml template to mapred2 and I have cleaned up the variable names. They should all nicely reflect the setting they apply to and have lost their now redundant hadoop_ and hadoop_config_ prefixes.

        All the values should have stayed the same. We noticed however that some of those values differ from the Hadoop defaults and are unsure if they still make sense for mapred 2. In our environment we've tuned them quite differently. So a review of those values might be advised, perhaps as a follow-up Jira.

        Show
        michaelweiser Michael Weiser added a comment - This is the first installment of the promised post-hiera-conversion cleanups: Olaf Flebbe has updated the mapred-site.xml template to mapred2 and I have cleaned up the variable names. They should all nicely reflect the setting they apply to and have lost their now redundant hadoop_ and hadoop_config_ prefixes. All the values should have stayed the same. We noticed however that some of those values differ from the Hadoop defaults and are unsure if they still make sense for mapred 2. In our environment we've tuned them quite differently. So a review of those values might be advised, perhaps as a follow-up Jira.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Please review

        Show
        oflebbe Olaf Flebbe added a comment - Please review
        Hide
        cos Konstantin Boudnik added a comment -

        when a patch is added please change the state of the JIRA into "Patch available" to indicate that it is ready for a review.

        Show
        cos Konstantin Boudnik added a comment - when a patch is added please change the state of the JIRA into "Patch available" to indicate that it is ready for a review.
        Hide
        cos Konstantin Boudnik added a comment -

        and btw - Michael Weiser - I have added you to the list of contributors on the project, so you should be able to assign tickets to yourself. Thanks for all your work!

        Show
        cos Konstantin Boudnik added a comment - and btw - Michael Weiser - I have added you to the list of contributors on the project, so you should be able to assign tickets to yourself. Thanks for all your work!
        Hide
        cos Konstantin Boudnik added a comment -

        It looks all right and seems to simply the deployment code quite a bit. I assume it has been tested?

        Show
        cos Konstantin Boudnik added a comment - It looks all right and seems to simply the deployment code quite a bit. I assume it has been tested?
        Hide
        michaelweiser Michael Weiser added a comment -

        Konstantin Boudnik: It's been tested by Olaf Flebbe in our environment. He says, that because of the old settings names and missing settings for the new version, mapred was quite broken before and now does at least work for what we're doing with it. I have tested the variable renaming bit and to the best of my knowledge, settings that remain from the old template still get their values as before.

        But since it's quite a big change, it would certainly benefit from someone else testing that it still works for their use case.

        Show
        michaelweiser Michael Weiser added a comment - Konstantin Boudnik : It's been tested by Olaf Flebbe in our environment. He says, that because of the old settings names and missing settings for the new version, mapred was quite broken before and now does at least work for what we're doing with it. I have tested the variable renaming bit and to the best of my knowledge, settings that remain from the old template still get their values as before. But since it's quite a big change, it would certainly benefit from someone else testing that it still works for their use case.
        Hide
        oflebbe Olaf Flebbe added a comment -

        It is tested to give the expected XML files. However, some of the old settings are of dubious nature. Assuming that the author knows Hadoop much better, we didn't change the Bigtop defaults. Our Cluster is using Hadoop defaults however (and gives the correct XML files). We only increased Memory Limits a lot, since hive on tez needs much more Memory than standard Hadoop defaults.

        So it is tested: It gives the functional equivalent hadoop configs with Bigtop Defaults. And It runs very good with standard Hadoop Defaults.

        Show
        oflebbe Olaf Flebbe added a comment - It is tested to give the expected XML files. However, some of the old settings are of dubious nature. Assuming that the author knows Hadoop much better, we didn't change the Bigtop defaults. Our Cluster is using Hadoop defaults however (and gives the correct XML files). We only increased Memory Limits a lot, since hive on tez needs much more Memory than standard Hadoop defaults. So it is tested: It gives the functional equivalent hadoop configs with Bigtop Defaults. And It runs very good with standard Hadoop Defaults.
        Hide
        cos Konstantin Boudnik added a comment -

        Olaf Flebbe please do commit then, if you're confident with the change. Worst case scenario - we'll find some bugs and fix them in the second iteration. Thanks!

        Show
        cos Konstantin Boudnik added a comment - Olaf Flebbe please do commit then, if you're confident with the change. Worst case scenario - we'll find some bugs and fix them in the second iteration. Thanks!
        Hide
        oflebbe Olaf Flebbe added a comment -

        pushed

        Show
        oflebbe Olaf Flebbe added a comment - pushed

          People

          • Assignee:
            michaelweiser Michael Weiser
            Reporter:
            michaelweiser Michael Weiser
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development