[OOZIE-561] Integrate Oozie with HCatalog

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: 4.0.0
• Assignee: Mona Chitnis
• Reporter: Santhosh Muthur Srinivasan
• Created: 15/Sep/11 01:51
• Updated: 31/Aug/13 00:02
• Resolved: 06/Jun/13 23:07

Description

With the incubation of HCatalog, we have a mechanism to abstract data and storage on HDFS. A natural progression for Oozie is to interact with HCatalog to facilitate the interplay between MapReduce, Pig and Hive. In addition, the support for notification in HCatalog will alleviate (and not eliminate) the need to poll HDFS for data sets represented as tables and partitions.

Sub-Tasks

• Create JMSService used for any JMS compliant product: Closed
• Generic utility class to register/unregister a JMS message handler: Closed
• Utility class to parse HCat URI: Closed
• Implement the Missing Dependency structure for HCat partitions: Closed
• Coordinator action table schema change: Closed
• Add logic to register to Missing Dependency Structure in coord action materialization: Closed
• Parameterize <uris> tag currently hardcoded: Closed
• Implement logic to update dependencies via push JMS message: Closed
• Add configurable 'max_size' for total number of Partition dependency map entries: Resolved
• Command to update push-based dependency: Closed
• Add static method to create URI String in HCatURI: Closed
• Add new EL function to retrieve HCatalog server, DB and table name: Closed
• Add EL functions to get HCat dataIn and dataOut: Resolved
• Metadata Accessor service for HCatalog: Closed
• Update dataIn and dataOut EL functions to support partitions: Closed
• Create general scheme handler: Closed
• Command to check the missing partitions directly against HCatalog server: Closed
• Add HCatalog jar as resource for building: Closed
• Revert OOZIE-1095 once dependent HCat jar mavenized: Closed
• Resolve issues found in integration: Closed
• Change default done-flag from _SUCCESS to empty for Hcat: Closed
• Fix JMS message consumer to maintain single session per topic registration: Closed
• The size of the map cache in PartitionDependencyManagerService should be configurable: Closed
• change HCatURI to specify partitions in path instead of query parameter: Closed
• EL Functions for hcatalog: Closed
• Prepare actions for hcat: Closed
• Display missing partition dependencies via job -info command on CLI: Closed
• Provide rule based mechanism to allow multiple hcatalog servers to connect to JMS server: Closed
• Escaping Ampersand in the HCat URI to bypass XML validation: Closed
• Modify Recovery Service to handle push missing dependencies: Closed
• DB upgrade scripts for hcat changes: Closed
• Make all the latest/future instances as pull dependences: Closed
• EL function hcat:exists for decision making: Closed
• Add hcataloglib sub-module: Closed
• Fix and rework PartitionDependency Management: Closed
• Reliability and retry for JMS connections: Closed
• Dependency cache with configurations for eviction, ttl and max elements in memory: Closed
• Retry jms connections on failure: Closed
• HCat EL functions for database and table should be modified: Closed
• Create a hcat sharelib which can be included in pig, hive and java actions: Closed
• Review of HCAT integration branch: Resolved
• Rework uri handling for Prepare actions and jms server mapping: Closed
• Address review comments in OOZIE-1210: Closed
• Create a HCatalog Integration Guide: Closed
• CoordPushCheck doesn't evaluate the configuration section which is propogated to workflow: Closed
• CoordActionInputCheck shouldn't queue CoordPushInputCheck: Closed
• Coord action timeout not happening when there is a exception: Closed
• Log messages for DependencyChecker class show wrong jobid and actionid: Closed
• latest() gets resolved before all push dependencies are resolved: Closed
• latest/future check for hcat can cause shutdown to hang: Closed
• Registered push dependencies are not removed on Coord Kill command: Closed
• Fix few HCat dependency check issues: Closed
• Dryrun option for push missing deps: Closed
• Exception in push dependency check when there is also a pull dependency leaves it in waiting till timeout: Closed
• Drop partition while reruning a job doesn't work as Hcatalog doesn't have DoAs support: Open
• CoordActionInputCheck requeues itself even if only push missing dependencies exist: Closed
• CoordPushDependencyCheck queued by Recovery Services doesn't remove dependencies from cache: Closed
• URIHandlerService not allowing relative path for URI's: Closed