Details

    • Type: Epic
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 2.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Ambari should support rack awareness to drive Hadoop operations.

      • Ability to set/edit Rack ID on hosts (On the hosts details page and prob need a dialog "Manage Rack Assignment..." on the Hosts Action menu to allow you to add Rack IDs and bulk move hosts in/out of a rack – similar to host config groups dialog)
      • Adjust heatmap to layout hosts by Rack ID (or default rack if not set)
      • Adjust hosts table to include Rack ID as a column, which is sortable and searchable

      Ambari will generate a topology script that echos this Rack ID, and sets that topology script in the config.

      1. AMBARI-6646.patch
        49 kB
        Tom Beerbower

        Issue Links

          Issues in Epic

          There are no issues in this epic.

            Activity

            Hide
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12706713/AMBARI-6646.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 11 new or modified test files.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The test build failed in ambari-server

            Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/2118//testReport/
            Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/2118//console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706713/AMBARI-6646.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 11 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in ambari-server Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/2118//testReport/ Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/2118//console This message is automatically generated.
            Hide
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12707417/AMBARI-6646.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 5 new or modified test files.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The test build failed in ambari-server

            Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/2143//testReport/
            Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/2143//console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707417/AMBARI-6646.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The test build failed in ambari-server Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/2143//testReport/ Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/2143//console This message is automatically generated.
            Hide
            hudson Hudson added a comment -

            SUCCESS: Integrated in Ambari-trunk-Commit #2121 (See https://builds.apache.org/job/Ambari-trunk-Commit/2121/)
            AMBARI-6646 - Ability to set rack awareness (tbeerbower) (tbeerbower: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=5746ee6d99c4a71b1c1c87e5aabeb966dfe47e87)

            • ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java
            • ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml
            • ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
            • ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py
            • ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
            • ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
            • ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
            • ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py
            • ambari-server/src/test/python/stacks/2.0.6/configs/default.json
            • ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py
            • ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
            • ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
            • ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
            • ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml
            • ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2
            • ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml
            • ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
            • ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java
            • ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
            • ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py
            • ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
            • ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
            • ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
            • ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml
            • ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java
            Show
            hudson Hudson added a comment - SUCCESS: Integrated in Ambari-trunk-Commit #2121 (See https://builds.apache.org/job/Ambari-trunk-Commit/2121/ ) AMBARI-6646 - Ability to set rack awareness (tbeerbower) (tbeerbower: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=5746ee6d99c4a71b1c1c87e5aabeb966dfe47e87 ) ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py ambari-server/src/test/python/stacks/2.0.6/configs/default.json ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java
            Hide
            neetagarimella Neeta Garimella added a comment -

            I'm interested in exploiting Rack awareness/rack ID in laying down Spectrum Scale(GPFS) file system. How can I get next level of details? Of course code can be downloaded but is there a detailed documentation of this feature. Thanks!

            Show
            neetagarimella Neeta Garimella added a comment - I'm interested in exploiting Rack awareness/rack ID in laying down Spectrum Scale(GPFS) file system. How can I get next level of details? Of course code can be downloaded but is there a detailed documentation of this feature. Thanks!
            Hide
            tbeerbower Tom Beerbower added a comment -

            Neeta Garimella, The Ambari docs for 2.1.0 are still being developed, but here is some more information on manually setting up rack awareness ...

            http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_system-admin-guide/content/admin_configure_rack_awareness.html

            With this patch, Ambari should take care of these steps for you. If you set the Hosts/rack_info property on a host, Ambari should mark the HDFS service for restart. You will see a restart flag on the service in the Ambari UI. Change the rack info for as many hosts as you need to then restart the service through the UI. As the service is restarted, the topology scripts are updated to reflect the changes in rack information for the hosts. You can verify that hadoop sees the rack changes by running the report in step 4 of the instructions above...

            sudo -u hdfs hadoop dfsadmin -report
            

            With the latest Ambari UI code you should see an option to set the rack info for each host. You can also set the rack info property through the API ...

            PUT api/v1/clusters/c1/hosts/[host_name]
            
            {
              "Hosts" : {
                "rack_info" : "/rack-01"
              }
            }
            

            Hope this helps. Thanks!

            Show
            tbeerbower Tom Beerbower added a comment - Neeta Garimella , The Ambari docs for 2.1.0 are still being developed, but here is some more information on manually setting up rack awareness ... http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_system-admin-guide/content/admin_configure_rack_awareness.html With this patch, Ambari should take care of these steps for you. If you set the Hosts/rack_info property on a host, Ambari should mark the HDFS service for restart. You will see a restart flag on the service in the Ambari UI. Change the rack info for as many hosts as you need to then restart the service through the UI. As the service is restarted, the topology scripts are updated to reflect the changes in rack information for the hosts. You can verify that hadoop sees the rack changes by running the report in step 4 of the instructions above... sudo -u hdfs hadoop dfsadmin -report With the latest Ambari UI code you should see an option to set the rack info for each host. You can also set the rack info property through the API ... PUT api/v1/clusters/c1/hosts/[host_name] { "Hosts" : { "rack_info" : "/rack-01" } } Hope this helps. Thanks!
            Hide
            hasonhai Ha Son Hai added a comment -

            Hi,
            I have just upgraded to Ambari 2.1 from 2.0.1 after the announcement of HDP 2.3 ready for enterprise.
            The upgrading of Ambari and Ambari Metrics is very easy and smooth. However, after the upgrading, Ambari display the icon to refresh the services with stale config. I tried to restart the services, but all of them failed with the error:

            Caught an exception while executing custom service command: <class ‘ambari_agent.AgentException.AgentException’>: ‘Broken data in given range, expected – m-n or m, got : /default-rack:0-21′; ‘Broken data in given range, expected – m-n or m, got : /default-rack:0-21′

            I wonder if the error is come from this new feature “rack-awareness”?

            P.S. I have 22 servers, all of them running HDP version 2.2.4.4-16.

            Show
            hasonhai Ha Son Hai added a comment - Hi, I have just upgraded to Ambari 2.1 from 2.0.1 after the announcement of HDP 2.3 ready for enterprise. The upgrading of Ambari and Ambari Metrics is very easy and smooth. However, after the upgrading, Ambari display the icon to refresh the services with stale config. I tried to restart the services, but all of them failed with the error: Caught an exception while executing custom service command: <class ‘ambari_agent.AgentException.AgentException’>: ‘Broken data in given range, expected – m-n or m, got : /default-rack:0-21′; ‘Broken data in given range, expected – m-n or m, got : /default-rack:0-21′ I wonder if the error is come from this new feature “rack-awareness”? P.S. I have 22 servers, all of them running HDP version 2.2.4.4-16.

              People

              • Assignee:
                tbeerbower Tom Beerbower
                Reporter:
                shivanigupta Shivani Gupta
              • Votes:
                3 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development