[FALCON-2182] Add support for Falcon user extensions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: trunk, 0.11
• Component/s: extensions
• Release Note: Resolving this as most of the sub-tasks are resolved. The ones remaining are nice to haves, which I'll move out of this task.
• Assignee: Unassigned
• Reporter: Pallavi Rao
• Created: 14/Nov/16 07:26
• Updated: 26/Oct/17 09:23
• Resolved: 26/Oct/17 09:23

Description

Currently server-side trusted extensions are supported (see FALCON-634 for more details).

Need to extend the Falcon Extensions to support extensions built by users. For example, lets say a user wants to add an extension for "data-quality check". This shouldn't require it to be compiled and tested with Falcon build. User should be able to add it on-the-fly (while the Falcon Server is running) and other users should be able to discover it, and "instantiate" this new extension.

Issue Links

• is blocked by: FALCON-2266 Non-intuitive response while registering/unregistering/submitting extension via server
• is blocked by: FALCON-2267 Definition api fails if resources are empty
• is blocked by: FALCON-2268 Extension list api does not follow default ASC order
• is blocked by: FALCON-2270 Extension job details returns incomplete response
• is blocked by: FALCON-2271 Scheduling non existing job fails with non intuitive response
• is blocked by: FALCON-2272 Extension update succeeds while it is disabled
• is blocked by: FALCON-2274 Job list in extension is failing with IllegalArgumentException
• is blocked by: FALCON-2275 Unable to list instances of an extension
• is blocked by: FALCON-2276 FalconCLIException while submitting trusted extension job
• is blocked by: FALCON-2277 Incorrect error message on registering trusted exceptions
• is blocked by: FALCON-2283 Internal Server Error : On trying to submit a trusted extension job
• is blocked by: FALCON-2284 Error while listing extension jobs
• is blocked by: FALCON-2286 Falcon upgradation fails to create new tables
• is blocked by: FALCON-2287 Extension list fails with MIME type error on server
• is blocked by: FALCON-2292 User Extension submission is failing because of the validate late inputs
• is blocked by: FALCON-2269 Extension api fails via server in distributed mode
• is related to: FALCON-2245 Improve error message for submitting job during extension jobName check
• is related to: FALCON-2213 Falcon logs does not contain response message when recipe operations are performed.
• is related to: FALCON-2214 Falcon extension registered without fully qualified hdfs path
• is related to: FALCON-2215 Extension job submission throws non intuitive error
• is related to: FALCON-2217 Job submission should fail with extension not registered.
• is related to: FALCON-2224 Change return type of extension APIs to APIResult
• is related to: FALCON-2239 Falcon extension definition feature is broken
• is related to: FALCON-2240 Enable/disable feature is broken
• is related to: FALCON-2241 Job submission for registered extension is broken
• is related to: FALCON-2242 Extension validate feature is broken
• is related to: FALCON-2243 Extension validation does not check job existance
• is related to: FALCON-2261 Job submission fails with IllegalArgumentException
• is related to: FALCON-2264 Error response returns object instead of response-message
• is related to: FALCON-2221 Submitting an existing job fails with MySQLIntegrityConstraintViolationException
• is related to: FALCON-2246 Api to list all the jobs in the system (and what extensions do they refer to)
• is related to: FALCON-2247 Include status of the entities of the extensionJob in getDetailExtensionJobs
• is related to: FALCON-2216 Recipe job submission api is non intutive
• relates to: FALCON-2238 Falcon extension describe feature is broken

Sub-Tasks

• Extension Builder changes to support new user extensions: Resolved
• Support for storing metadata of non trusted recipe: Resolved
• Falcon Client changes for Falcon user extensions: Resolved
• Support to delete a non trusted recipe metadata from server: Resolved
• Rest api to get details of an extension: Resolved
• Refactor modules to create a new module for entity xsd specifications and jaxb bindings: Resolved
• Rest api to register extension: Resolved
• Add server side checks to not let any entity mutations on entities generated by extensions: Resolved
• Extension Rest Api changes to support new user extensions and extension job metastore changes: Resolved
• Fix test case failure caused by FALCON-2188: Resolved
• Falcon build is failing for falcon jpa service to be initialized before extension servivce: Resolved
• Enhance validate API to support config validation for user extensions: Resolved
• Add API to get details of an extension job: Resolved
• Extensions User documentation: Resolved
• Add enable and disable option for an extension: Resolved
• Delete API support for extension job (user extension): Resolved
• Update API support for extension job (user extension): Resolved
• Falcon Unit changes to support extensions: Resolved
• Only the owner of the extension, should be able to unregister, disable, enable.: Closed
• Falcon CLI(client) need hadoop in the classpath: Resolved
• Registering user extensions in embedded/distributed mode: Resolved
• describe for non trusted extension: Resolved
• Server side changes for submit and submitAndSchedule for user extensions: Resolved
• Falcon Fails to start in distributed mode: Resolved
• Add a profile for lgpl: Resolved
• Distributed mode support for User Extension: Resolved
• Falcon fails to start with ExtensionService: Closed
• Submit ,Schedule and submitAndSchedule API for extension in distributed mode: Resolved
• Intitutive error handling in Falcon Client: Resolved
• Support Schedule for User Extensions: Resolved
• Make changes in falcon unit to enable/disable extensions: Resolved
• Suspend/Resume API support for extension job (user extension): Resolved
• Improve error message for definition/describe api in case extension does not exist: Resolved
• Enhance extension List api to support for user extensions: Resolved