[AMBARI-666] Ambari architectural changes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.2.0
• Labels: AMBARI-666
• Assignee: Jitendra Nath Pandey
• Reporter: Jitendra Nath Pandey
• Created: 14/Aug/12 21:57
• Updated: 07/May/13 18:14

Description

• First release of Ambari is underway and this jira is to facilitate next set of major features in Ambari. This jira proposes a few significant architectural changes towards following goals:
• Webservice APIs
• Pluggability of components: Particularly third party tools like puppet.
• Reduce dependencies
• Flexibility to add features.
• Maintainability of code, unit testing, modular design.
• Improve performance.

A design proposal will be published soon.

Issue Links

• is related to: AMBARI-875 Support cluster blueprint in Ambari

Sub-Tasks

• Delete old directories/code: Resolved
• Seperate directory for ambari-server.: Resolved
• Fix pom structure: Resolved
• Remove non-required dependencies from pom files: Resolved
• Basic object structure/design for the server: Resolved
• More basic classes for new design: Resolved
• Classes for request objects.: Resolved
• Class to encapsulate stage: Resolved
• More basic class re-structuring: Resolved
• Add interface for ActionManager to access live state: Resolved
• Ambari Web (browser-based UI) skeleton: Resolved
• Add a simple server and artifact generation to run a server with a simple api check: Resolved
• Fix mvn build broken by AMBARI-698 changes.: Resolved
• Add skeleton for Ambari agent that talks to the server and collects information for host.: Resolved
• Heartbeat handler classes.: Resolved
• Initial work on node-fsm: Resolved
• Add basic tests for Node FSM: Resolved
• More work on Node FSM and additional tests/cleanup: Resolved
• Whitespace cleanup in files.: Resolved
• Getting hardware info on disks/cpu/others using facter and using it during registeration.: Resolved
• Basic registration and heartbeat protocol implementation between the server and the agent.: Resolved
• Create utility functions related to localStorage for first two steps: cluster name and Install options. Also develop view logic with preliminary validations for these two steps.: Resolved
• Action manager skeleton: Resolved
• Initial work for Job FSM: Resolved
• Job FSM Impl and tests: Resolved
• Integrate domain objects and Rest serialized objects.: Resolved
• Add back TestNodeImpl and fix memory types and disk info serialization: Resolved
• Starter implementation for Installer Customize Services page. Stylize top nav and implement static footer. Stylize login page.: Resolved
• Fix installer navigation: Resolved
• Enable Responsive Design: Resolved
• Tweak basic styles for Installer: Resolved
• Remove Hardwareprofile class since its not needed anymore.: Resolved
• Action scheduler implementation: Resolved
• Implement Installer Welcome page and Install Options page: Resolved
• Add tabs, dynamic form generation, validation errors, and info popovers for Customize Services page in Installer: Resolved
• Add commandstatus/result/error objects into the rest API between server and agent.: Resolved
• ORM-based data access layer for new design: Resolved
• Fix rat excludes as mvn rat:check fails at the top level: Open
• Initial work on ServiceComponentNode FSM: Resolved
• Add dynamic validation based on data type, error count display, and submission block for Installer Customize Services page: Resolved
• Add unit tests for jersey apis on the server.: Resolved
• Action manager and fsm interaction: Resolved
• Unit tests for action scheduler.: Resolved
• Add Jersey Resource for BootStrapping and JAXB elements for API entities.: Resolved
• Initial work for adding DataNode/TaskTracker/RegionServer configuration overrides on groups of hosts in Installer Customize Services page: Resolved
• Clean up Installer Welcome page and Install Options page: Resolved
• Initial work on Cluster Management pages: Resolved
• SvcComponent-Node FSM: Resolved
• s/Node/Host/g: Resolved
• Initial cluster fsm implementation: Resolved
• Hook FSMs into ORM layer: Resolved
• Use ember-i18n to externalize string resources in Ambari Web: Resolved
• Implement running a bootstrap api, keep a fifo queue and logic for getting a request.: Resolved
• Add unit testing framework for Ambari Web: Resolved
• Add definition for service config properties: Resolved
• Add unit tests for Installer Step 1 (Welcome page): Resolved
• Integrate configuration properties and custom configuration file overrides in Customize Services page: Resolved
• Add unit tests for step2 (Install option page) of installer.: Resolved
• Implement DataNode/TaskTracker/RegionServer overrides based on groups of hosts: Open
• Complete Java side implementation of bootstrapping agent hosts.: Resolved
• Fix build compilation issue.: Resolved
• Re-structure servicecomponenthost fsm layout: Resolved
• Add missing license header to TestServiceComponentHostState: Resolved
• Fix broken compile as a result of re-factor of FSM layout: Resolved
• Heartbeat handler: Registration response should query component status.: Resolved
• Heartbeat handler: Update state as reported in heartbeat.: Resolved
• Heartbeat handler: Handle heartbeat timeout.: Resolved
• Implement Installer Step 4 (Select Services): Resolved
• Cluster name and ambari version should be persisted with any state persisted by Ambari or its component.: Open
• Add puppet scripts to the agent for configuring/installing various services and add security aspects to api's and server/agent.: Resolved
• Fix injection in data access objects to use guice provider.: Resolved
• Fix broken build for adding guice servlet dependency.: Resolved
• Implement Confirm Hosts page for Ambari installer: Resolved
• Implement Installer Step 6 (Assign Slaves): Resolved
• Integrate REST API: Resolved
• Add bootstrap script to ssh in parallel and setup agents on a list of hosts.: Resolved
• Implement step 5 of installer wizard (Assign Masters): Resolved
• Implement step 9 (Install,start and test) of installer wizard: Resolved
• Cluster Management pages for Ambari Web: Resolved
• Add security between the server and agent and authentication for the API.: Resolved
• Stylize main nav: Resolved
• Change Host FSM as per new requirements of heartbeat handler: Resolved
• Renable and fix AgentResourceTest.: Resolved
• In memory implementation for ActionDBAccessor: Resolved
• Puppet scripts for all the modules to install/configure the stack.: Resolved
• Add NOTICE file for copyright of stdlib from puppetlabs.: Resolved
• Ensure data flows across all steps in installer wizard: Resolved
• Introduce ManagementController interface: Resolved
• Make FSM related changes for heartbeat handler: Resolved
• Registration unit test: Resolved
• Fix guice injection in the server.: Resolved
• Add Resource download API on the server.: Resolved
• Action response unit test.: Resolved
• Utility to initialize in memory action db for testing.: Open
• Registration throws HostNotFoundException for new hosts.: Resolved
• Fix server and agent startup for end to end testing.: Resolved
• Prototype for management spi interface: Resolved
• Registration response: Resolved
• Add unit tests and java docs for SPI code.: Resolved
• Make MapReduce, Nagios, and Ganglia optional during cluster install: Resolved
• Add log4j properties for logging at the server.: Resolved
• Fix failing tests for AgentResource and BootStrap.: Resolved
• Remove ambari-server dependency on ambari-api: Resolved
• Prototype for management spi interface continued: Resolved
• Fix import issue due to move of Predicate class: Resolved
• Prototype for management spi part 3: Resolved
• Hack to add a stage for testing in in-memory db.: Resolved
• Fix heartbeat message from the agent which is causing NPE at the server.: Resolved
• Enhance Cluster Management pages: Resolved
• FSM initial refactoring for eventual live/desired objects: Resolved
• Fix routing issues with Admin pages: Resolved
• Add requestId tracking objects for management spi: Resolved
• Remove State object as configs/stack version/running state are handled as standalone entities: Resolved
• Fix Action scheduler tests because of fsm interface changes.: Resolved
• Handle appropriate start/stop/install/.. events at their respective failed states: Resolved
• Continue refactoring of management interface and ResourceProvider: Resolved
• Controller layer implementation part 1: Resolved
• Bug fix in jaxb serialization for maps.: Open
• Controller layer implementation part 2: Resolved
• Refactoring ClusterController to allow pluggable ResourceProviders: Resolved
• Ensure data persistence during navigation, refresh and ensure necessary restrictions: Resolved
• In API , improve partial response support to drill down n levels: Resolved
• Update Cluster Management -> Services pages to use a left nav rather than tabs for service selection: Resolved
• Fix import dependencies so that unit test suites can be run again for Ambari Web: Resolved
• Add host names to cluster request and response objects: Resolved
• Management controller implemenation work: Resolved
• Remove JAXB dependencies in Server Agent protocol and move to POJO based jackson serializer.: Resolved
• Implement basic service state update and verify flow to ActionManager: Resolved
• Implement an agent simulator for unit testing.: Resolved
• Fix security filter on the server agent ports and remove duplication on servlet contexts for certs signing.: Resolved
• Controller layer implementation part 3: Resolved
• Bug in processing command reports.: Resolved
• Add clusterName to the status of the commands run by the agent.: Resolved
• Manifest generation for various actions from the server.: Resolved
• Add unit tests for ResourceProviderImpl: Resolved
• Various fixes and tests for controller implementation: Resolved
• Move manifest generation into the ambari agent directory.: Resolved
• Merge ambari-api with ambari-server: Resolved
• Use RoleCommand instead of ServiceComponentHostEventType for HostAction in Controller: Resolved
• Update JMXPropertyProvider: Resolved
• Fix generation of requestId to be unique across restarts: Resolved
• Fix basic injection issues for controller impl: Resolved
• HACK: Add a thread in server to inject requests for testing.: Resolved
• Temporary fix for server start order: Resolved
• hitting browser refresh should not clear present step data that had already been persisted to local DB: Resolved
• Fix comparison predicates in case where resource does not have property value.: Resolved
• Fix null point exception during adding of hosts to cluster: Resolved
• Fix more null pointers for partial request objects: Resolved
• Mock JMX provider for manual tests: Resolved
• Fix NPE in the server to be able to run the server api's.: Resolved
• Select Masters Page: make zooKeeper addition/removal UI more organized: Resolved
• Run pyunit tests from maven test target and also autogenerated openssl passphrase.: Resolved
• Various tests for FSM and Controller impl: Resolved
• Stage planner implementation: Resolved
• Flatten the ExecutionCommand: Resolved
• Hook up Ganglia property provider: Resolved
• Add more complete JMX metrics: Resolved
• Serve ambari-web from jetty: Resolved
• Improve REST API functionality regarding query and partial response: Resolved
• Create the skeleton for a custom data adapter in Ambari Web: Resolved
• Add cluster heatmap: Resolved
• Refactor Install Options page for the Install Wizard: Resolved
• Installer -> Select Services page: warning popups are no longer appearing: Resolved
• Tighten up the layout for the Install page of the Installer: Resolved
• Remove code that adds fake stages for testing.: Resolved
• API query against /clusters doesn't return any data: Resolved
• Fix mvn tests to be able to run the python tests cleanly.: Resolved
• Add Hadoop stack selection to the Installer: Open
• Add unit test for action queue on the agent.: Resolved
• Add ORM layer for the FSM's in the server.: Resolved
• Improve user interactions in Confirm Hosts page of the Installer: Open
• Clean up site.pp generation on the agent and remove the imports in the sample site.pp.: Resolved
• Util to deserialize ExecutionCommand: Resolved
• Support metric types other than string: Resolved
• Integrate basic set of rest APIs with ambari-web installer wizard: Resolved
• Hookup Nagios alerts section in Ambari UI to backend server: Resolved
• Fix hostinfo reporting at the server and add a unit test for deserilization for the host information from agent.: Resolved
• Support for multiple objects' updates in controller: Resolved
• Put metrics under metrics category: Resolved
• Refactor resource provider implementation for changes to management interface.: Resolved
• Various tests for FSM, controller and state objects: Resolved
• Installer skips Confirm Hosts page of the wizard when testing locally without Ambari Server: Resolved
• Implement step8 'Review page' for the ambari installer wizard.: Resolved
• Group-based DataNode/TaskTracker/RegionServer overrides: Resolved
• Implement Add Hosts Wizard: Resolved
• Improve user interactions on Confirm Hosts page of the Installer: Resolved
• Implement Dashboard/Service Summary: Resolved
• Fix miscellaneous issues related to Ambari Web: Resolved
• Support filters in controller get* apis: Resolved
• Ability to save configuration: Resolved
• Add more tests for controller implementation: Resolved
• Provide cluster metric graphs on Ambari main dashboard: Resolved
• Add client library option to step6 (Assign slaves) of installer wizard. Also add indicator and popovers for hosts with master component.: Resolved
• Initial work to refactor the Wizards in Ambari Web: Resolved
• Add puppet executor at the agent to be able to run various commands from the server.: Resolved
• TestHeartBeatMonitor fails intermittently.: Resolved
• Need name consistency for metrics: Resolved
• Resolve all navigation related issues for Step6 (Slave and Client component) of installer wizard.: Resolved
• Request object for Operations.: Resolved
• Request object for Operations.: Resolved
• Use slf4j-api 1.5.5 rather than 1.6.6: Resolved
• Add configuration mapping support: Resolved
• Improve support for jobs/actions on cluster: Resolved
• Fix zooKeeper badge allocation logic for single host and muliple hosts installation in step5 installer wizard.: Resolved
• Various fixes for config handling integration: Resolved
• Ensure state changes only happen after actionmanager persists actions: Resolved
• Fix puppet site creation with flattening of execution commands send from the server.: Resolved
• Util to extract hosts for various components: Resolved
• Add support for getting multiple objects in controller: Resolved
• Add ui option to either create a Postgres database for Hive and Oozie or choose existing database.: Resolved
• Pass correct cluster info to Action Manager: Resolved
• Implement an efficient way to provide Ganglia data for heatmap and other cluster visualization tools: Resolved
• Test case for ConfigurationService: Resolved
• Fix issues found when integrating server and agent for install of HDFS.: Open
• provide api support for temporal queries: Resolved
• Fix all apis to return correctly filled status response.: Resolved
• Fix issues related to Slave Component Group in Installer: Resolved
• Implement App Browser for displaying and navigating Pig/Hive workflows: Resolved
• Provide HDFS metrics graphs in HDFS services page: Resolved
• Update layout and flow for App Browser: Resolved
• Update styles in Cluster Management: Resolved
• Partial refactoring and consolidation of code for various wizards: Resolved
• APIs for Actions.: Resolved
• Bug fixes for starting various services at the agent.: Open
• Use stack name and version in fsm layer: Resolved
• ResourceProvider changes for Request and Task resources: Resolved
• rename job to action: Resolved
• Return RequestStatus through ClusterController: Resolved
• Provide API support for asynchronous requests: Resolved
• Provide metrics graphs on the MapReduce services page: Resolved
• Enable end to end testing of Request and Task resources: Resolved
• Show HBASE_REGIONSERVER metrics: Resolved
• Support for /hosts endpoint: Resolved
• Add initial hooks to trigger smoke tests on service starts: Resolved
• Provide service specific alerts in the service summary pages: Resolved
• Map update to PUT and create to POST: Resolved
• Add id for zookeeper servers.: Open
• Provide HBase service specific graphs: Resolved
• Hardcode service name and client component mapping while awaiting meta data integration: Resolved
• Integration fixes for the ORM layer.: Open
• Adding more tests and more api's for getting service info and component info.: Open
• More refactoring of Wizards in Ambari Web: Resolved
• Integrate Install and start APIs with the installer wizard.: Resolved
• Add Host metrics: Resolved
• Fixes for meta info layer: Resolved
• Fix 500 errors on get resources api: Resolved
• Support retrieving information for multiple requests: Resolved
• Make it easier to test Deploy (Install, Start + Test) step of the wizard: Resolved
• Invoke service starts after services are installed in the wizard: Resolved
• Provide metric graphs for individual hosts: Resolved
• Integrate meta data to be able to run service checks after a start of a service: Resolved
• Fix navigation issues in installer wizard introduced due to regression: Resolved
• Support installation of Ganglia master and slaves via Ambari Web: Resolved
• Provide API support for 'OR' predicate: Resolved
• Bug Fixes for fixing ganglia install and start using the API's.: Open
• Make API authentication configurable at the server. For testing its useful to switch off authentication for the API's: Open
• Provide API support for updates of multiple resources in a single request: Resolved
• HDFS_CLIENT installation failure.: Resolved
• On unavailability of non-master components, host with least number of master components should install all slave and client components.: Resolved
• Adding a host whose hostname is the same as the one the user is accessing Ambari Web with breaks the Installer: Resolved
• Create a key value get/put API for web ui to use for tmp storage.: Open
• Handle passing repo information from server to agent: Resolved
• Meta data api describing stacks and configs for various services and there version.: Open
• Sub-resources and metrics through API are broken.: Resolved
• Update of host components fail when service name is not passed in: Resolved
• Dependency order when stopping services.: Resolved
• Implement summary page of installer wizard.: Resolved
• Stop events should be handled at all valid points for safe recovery: Resolved
• Write controller unit tests to verify that config updates are allowed only in certain states like init/installed/start-failed: Open
• Enhance predicate comparison: Resolved
• Task resources not returned under request resources: Resolved
• GET on temporal data throws 500: Resolved
• Add additional Ganglia metrics and JMX properties: Resolved
• Refactor resource provider implementation to move inner classes.: Resolved
• Ensure zookeeper service check run after zookeeper start: Resolved
• Decommissioning of datanodes: Resolved
• Fix support for cascading updates to configs: Resolved
• Hook HDFS/MapReduce/HBase/Host graphs to backend API: Resolved
• Refactor Wizard and Cluster Management code: Resolved
• Refactor App Browser Code: Resolved
• More refactoring of App Browser code: Resolved
• Allow installation of various service components: Resolved
• Add configuration to host_component request: Resolved
• Add ability to set rack info for a host: Resolved
• Enhance hosts api to return more information like rack info, reg/heartbeat times: Resolved
• Add support for exposing filesystem type for mount points at host level: Resolved
• Support OR in API query: Resolved
• Support oozie-3.2: Open
• Fix issues with deploying stack: Resolved
• Update resource drops property values.: Resolved
• Show task logs for each host in the Deploy step of the wizard: Resolved
• Refactor App Browser: Resolved
• Initial work to integrate Ambari Web with Ambari Server: Resolved
• Logout does not clean application state properly: Resolved
• Hook up login with server authentication: Resolved
• Host metrics API servlet not filtering on given fields: Resolved
• Deploy logs not shown for failed tasks: Resolved
• Expose metrics and properties for UI: Resolved
• Fix HBASE JMX properties: Resolved
• Fix partial response for root level host query: Resolved
• RUBYLIB env variable expanding forever: Resolved
• Use FQDN instead of hostname when registering an agent with Ambari server: Resolved
• Cluster level Network, Load, CPU and Memory metrics in API needed for dashboard page: Resolved
• Integrate Installer with config APIs: Resolved
• Nagios sections should use backend API to populate: Resolved
• Allow properties entered in custom config (ex: hdfs-site.xml) to override existing or create new properties.: Resolved
• No Ganglia/JMX metrics data are coming through: Resolved
• Populate dashboard>HDFS section with API data: Resolved
• Add aggregate IO stats to workflow data web service: Resolved
• Populate dashboard>MapReduce section with API data: Resolved
• Cluster level graphs need to use API for data: Resolved
• Fix extra comma in rrd.py output: Resolved
• Create 2 missing HDFS service graphs: Resolved
• Fix race condition in DefaultProviderModule: Resolved
• Host metrics charts should use live data: Resolved
• Hook service summary sections in service pages to API: Resolved
• Create HBase summary section in Dashboard & Service pages: Resolved
• Initial API calls after Ambari Web install resulted in 500 (ArrayIndexOutOfBoundsException); Ambari Web stuck at "Loading...": Resolved
• Alerts not showing up in Ambari UI due to model refactoring: Resolved
• Add methods to get postgres user name/password or any regex validated string input from user: Resolved
• Start time, end time and resolution not set correctly for rrd.py call in Ganglia property provider.: Resolved
• Ambari-agent init script needs to be aware of already running/not running agent process: Resolved
• Integrate Heatmap UI to backend API: Resolved
• Dashboard page should handle API sending JSON as strings and objects: Resolved
• Reconfiguration of service for cluster management: Resolved
• Display total install and start services time on summary page and polish summary page ui.: Resolved
• Resolve overlap between JMX and Ganglia metrics: Resolved
• Fix missing block metrics for NAMENODE: Resolved
• MapReduce & HDFS summaries should use ServiceComponentInfo values: Resolved
• Add api support for updating multiple host_component resources for multiple hosts in a single request: Resolved
• Metrics links in web ui should link to Ganglia UI: Resolved
• Check for host registration at step3 of installer wizard and retrieve information for RAM and no. of cores.: Resolved
• Host hover in cluster heatmap showing mock data: Resolved
• Nagios and Ganglia links should use public host names in URLs: Resolved
• Metric Charts - display local time rather than UTC: Resolved
• Aggregate creation of multiple services and assignment of host to cluster.: Resolved
• Service Info/Quick Links do not display external hostnames: Resolved
• Implement an efficient way to provide Ganglia data for heatmap and other cluster visualization tools: Resolved
• Ganglia setup does not allow for subsequent changes to the cluster.: Resolved
• Improve Nagios alerts time display: Resolved
• Cluster heatmap: green should always mean "good": Resolved
• Additional metrics need to be added to Heatmap UI: Resolved
• Heatmap UI fluctuates between white and green colors intermittently: Resolved
• Updates with queries that contain non primary key fields may update resources that don't satisfy the query: Resolved
• API is not returning Ganglia metrics for one of the hosts in the cluster: Resolved
• Service summary sections have incorrect values displayed: Resolved
• Heatmap with no numbers on the hover: Resolved
• Create Configuration API call is throwing 500: UnsupportedPropertyException: Resolved
• Integrate slave configuration parameters with respective service on step7 of installer wizard: Resolved
• Tune Ganglia request: Resolved
• Host metric values coming in with 0 values: Resolved
• Dashboard page takes long time to load: Resolved
• UnsupportedPropertyException thrown from update: Resolved
• Dashboard page loads very slow due to hosts?fields=* API call: Resolved
• Implement retrying of bootstrap on confirm host page: Resolved
• Refactor SPI Request interface to remove PropertyId: Resolved
• Expose CapacityRemaining JMX metric to NAMENODE ServiceComponentInfo: Resolved
• Can't reset ambari-server due to fail drop/create database DDLs: Resolved
• Implement data loading: Resolved
• Refactor cluster management: Resolved
• Provide Datanode decommission & recommission capability in UI: Resolved
• Rename Charts section to Heatmaps: Resolved
• Service > MapReduce map slots reserved / occupied are backwards: Resolved
• Dashboard cluster level graphs showing only 45 minutes of data: Resolved
• HDFS Disk Capacity in HDFS Service Summary is totally off: Resolved
• App browser fixes: Resolved
• Workflow Web Service potentially leaks DB connections upon exceptions: Resolved
• Convert Apache license header comment style in Handlebars files to Handlebars comments rather than JavaScript comments: Resolved
• Data loading refactoring for cluster management: Resolved
• Data loading for App Browser: Resolved
• Ambari API: return proper response codes and provide a meaningful error response body: Resolved