Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-19702

Switch Pig default engine to tez in HDP 2.6

Details

    • Task
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • None
    • trunk
    • ambari-server
    • None

    Attachments

      1. AMBARI-19702.1.patch
        11 kB
        Daniel Dai
      2. AMBARI-19702.2.patch
        11 kB
        Daniel Dai
      3. AMBARI-19702.3.patch
        2 kB
        Daniel Dai
      4. AMBARI-19702.4.patch
        3 kB
        Daniel Dai

      Activity

        afernandez Alejandro Fernandez added a comment - - edited

        -1. Fix it and ship it.
        I think you meant to edit ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml instead of ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
        Within the same stack version, we don't apply config changes.

        Everything else looks good, thanks for making this change.

        cc daijy

        afernandez Alejandro Fernandez added a comment - - edited -1. Fix it and ship it. I think you meant to edit ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml instead of ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml Within the same stack version, we don't apply config changes. Everything else looks good, thanks for making this change. cc daijy
        daijy Daniel Dai added a comment -

        Thanks for pointing out. Patch updated.

        daijy Daniel Dai added a comment - Thanks for pointing out. Patch updated.
        afernandez Alejandro Fernandez added a comment - +1 for AMBARI-19702.2.patch

        Pushed to trunk, commit 2f06c1cefac30e0a53d631cf2a2a6aef64a89e0c
        branch-2.5, commit 84c680a9ef14581463dc333b36ebcd3603cdaa78

        afernandez Alejandro Fernandez added a comment - Pushed to trunk, commit 2f06c1cefac30e0a53d631cf2a2a6aef64a89e0c branch-2.5, commit 84c680a9ef14581463dc333b36ebcd3603cdaa78
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Ambari-branch-2.5 #849 (See https://builds.apache.org/job/Ambari-branch-2.5/849/)
        AMBARI-19702. Switch Pig default engine to tez in HDP 2.6 (Daniel Dai (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=84c680a9ef14581463dc333b36ebcd3603cdaa78)

        • (add) ambari-server/src/main/resources/stacks/HDP/2.6/services/PIG/configuration/pig-properties.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Ambari-branch-2.5 #849 (See https://builds.apache.org/job/Ambari-branch-2.5/849/ ) AMBARI-19702 . Switch Pig default engine to tez in HDP 2.6 (Daniel Dai (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=84c680a9ef14581463dc333b36ebcd3603cdaa78 ) (add) ambari-server/src/main/resources/stacks/HDP/2.6/services/PIG/configuration/pig-properties.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #6599 (See https://builds.apache.org/job/Ambari-trunk-Commit/6599/)
        AMBARI-19702. Switch Pig default engine to tez in HDP 2.6 (Daniel Dai (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=2f06c1cefac30e0a53d631cf2a2a6aef64a89e0c)

        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml
        • (add) ambari-server/src/main/resources/stacks/HDP/2.6/services/PIG/configuration/pig-properties.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #6599 (See https://builds.apache.org/job/Ambari-trunk-Commit/6599/ ) AMBARI-19702 . Switch Pig default engine to tez in HDP 2.6 (Daniel Dai (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=2f06c1cefac30e0a53d631cf2a2a6aef64a89e0c ) (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml (add) ambari-server/src/main/resources/stacks/HDP/2.6/services/PIG/configuration/pig-properties.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
        daijy Daniel Dai added a comment -

        Thanks afernandez!

        daijy Daniel Dai added a comment - Thanks afernandez !

        daijy, afernandez, this prevents Ambari Server start, and also causes unit test failures:

        ERROR [main] ModuleFileUnmarshaller:141 - Cannot parse /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        ERROR [main] ModuleFileUnmarshaller:143 - Upgrade pack must contain pre-downgrade elements if pre-upgrade exists for processing component PIG/PIG
        

        Could you please check?

        adoroszlai Attila Doroszlai added a comment - daijy , afernandez , this prevents Ambari Server start, and also causes unit test failures: ERROR [main] ModuleFileUnmarshaller:141 - Cannot parse /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml ERROR [main] ModuleFileUnmarshaller:143 - Upgrade pack must contain pre-downgrade elements if pre-upgrade exists for processing component PIG/PIG Could you please check?
        daijy Daniel Dai added a comment -

        Sorry about this. Attach AMBARI-19702.3.patch. Can you check if this fixes the issue? Or if you want me to create a separate Jira, let me know.

        daijy Daniel Dai added a comment - Sorry about this. Attach AMBARI-19702 .3.patch. Can you check if this fixes the issue? Or if you want me to create a separate Jira, let me know.

        daijy, yes, AMBARI-19702.3.patch fixes the issues. Thanks for the quick response.

        adoroszlai Attila Doroszlai added a comment - daijy , yes, AMBARI-19702.3.patch fixes the issues. Thanks for the quick response.
        daijy Daniel Dai added a comment -

        Thanks adoroszlai!

        daijy Daniel Dai added a comment - Thanks adoroszlai !

        afernandez, could you please commit AMBARI-19702.3.patch, too, to resolve this issue? Thanks.

        adoroszlai Attila Doroszlai added a comment - afernandez , could you please commit AMBARI-19702.3.patch , too, to resolve this issue? Thanks.
        afernandez Alejandro Fernandez added a comment - - edited

        daijy, I pushed addendum AMBARI-19702.3.patch to
        trunk, commit 772c093073b31fb0887d7072622e4c37c227e28d
        branch-2.5, commit 4116c4092d1e32f685dbf379f9ca2ec89718d89d

        afernandez Alejandro Fernandez added a comment - - edited daijy , I pushed addendum AMBARI-19702.3.patch to trunk, commit 772c093073b31fb0887d7072622e4c37c227e28d branch-2.5, commit 4116c4092d1e32f685dbf379f9ca2ec89718d89d
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Ambari-branch-2.5 #856 (See https://builds.apache.org/job/Ambari-branch-2.5/856/)
        AMBARI-19702. ADDENDUM. Switch Pig default engine to tez in HDP 2.6 (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=4116c4092d1e32f685dbf379f9ca2ec89718d89d)

        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Ambari-branch-2.5 #856 (See https://builds.apache.org/job/Ambari-branch-2.5/856/ ) AMBARI-19702 . ADDENDUM. Switch Pig default engine to tez in HDP 2.6 (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=4116c4092d1e32f685dbf379f9ca2ec89718d89d ) (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #6605 (See https://builds.apache.org/job/Ambari-trunk-Commit/6605/)
        AMBARI-19702. ADDENDUM. Switch Pig default engine to tez in HDP 2.6 (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=772c093073b31fb0887d7072622e4c37c227e28d)

        • (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml
        • (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #6605 (See https://builds.apache.org/job/Ambari-trunk-Commit/6605/ ) AMBARI-19702 . ADDENDUM. Switch Pig default engine to tez in HDP 2.6 (afernandez: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=772c093073b31fb0887d7072622e4c37c227e28d ) (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml (edit) ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
        jonathanhurley Jonathan Hurley added a comment - - edited

        I just saw the review for this and I believe the code is incorrect. You wanted to change a value inside of pig-properties/content:

        # exectype local|mapreduce|tez, mapreduce is default
        exectype=mapreduce
        

        However, the upgrade work you did specifies:

            <service name="PIG">
              <component name="PIG">
                <changes>
                  <definition xsi:type="configure" id="hdp_2_6_0_0_pig_use_tez">
                    <type>pig-properties</type>
                    <replace key="exectype" find="exectype=mapreduce" replace-with="exectype=tez" />
                  </definition>
                </changes>
              </component>
            </service>
        

        The key is not exectype ... the key would need to be content since this is a part of pig-properties/content ...

        cc afernandez since you committed this on behalf of the author...

        jonathanhurley Jonathan Hurley added a comment - - edited I just saw the review for this and I believe the code is incorrect. You wanted to change a value inside of pig-properties/content: # exectype local|mapreduce|tez, mapreduce is default exectype=mapreduce However, the upgrade work you did specifies: <service name= "PIG" > <component name= "PIG" > <changes> <definition xsi:type= "configure" id= "hdp_2_6_0_0_pig_use_tez" > <type>pig-properties</type> <replace key= "exectype" find= "exectype=mapreduce" replace-with= "exectype=tez" /> </definition> </changes> </component> </service> The key is not exectype ... the key would need to be content since this is a part of pig-properties/content ... cc afernandez since you committed this on behalf of the author...
        daijy Daniel Dai added a comment - - edited

        Thanks jonathanhurley, attach AMBARI-19702.4.patch to fix it. Also change the comment "mapreduce is default" to "tez is default" in the patch.

        daijy Daniel Dai added a comment - - edited Thanks jonathanhurley , attach AMBARI-19702 .4.patch to fix it. Also change the comment "mapreduce is default" to "tez is default" in the patch.

        That looks better - is there a Reviewboard for this patch?

        jonathanhurley Jonathan Hurley added a comment - That looks better - is there a Reviewboard for this patch?
        daijy Daniel Dai added a comment -
        daijy Daniel Dai added a comment - Here is the RB link: https://reviews.apache.org/r/56183/

        Thanks - afernandez Can you commit this on behalf of Daniel?

        jonathanhurley Jonathan Hurley added a comment - Thanks - afernandez Can you commit this on behalf of Daniel?

        Ping afernandez Can we commit this on Daniel's behalf?

        avijayan Aravindan Vijayan added a comment - Ping afernandez Can we commit this on Daniel's behalf?

        Bulk moving of issues out of 2.5.0.

        avijayan Aravindan Vijayan added a comment - Bulk moving of issues out of 2.5.0.
        daijy Daniel Dai added a comment -

        avijayan, afernandez, can you help commit to 2.5.0 branch? Otherwise, we leave it in a half done status.

        daijy Daniel Dai added a comment - avijayan , afernandez , can you help commit to 2.5.0 branch? Otherwise, we leave it in a half done status.

        People

          daijy Daniel Dai
          daijy Daniel Dai
          Votes:
          0 Vote for this issue
          Watchers:
          7 Start watching this issue

          Dates

            Created:
            Updated: