[HBASE-17732] Coprocessor Design Improvements

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0-alpha-4, 2.0.0
• Component/s: Coprocessors
• Labels: incompatible
• Hadoop Flags: Incompatible change

Release Note

We are moving from Inheritence
- Observer *is* Coprocessor
- FooService *is* CoprocessorService
To Composition
- Coprocessor *has* Observer
- Coprocessor *has* Service
------------------------------------------------------
Summary
------------------------------------------------------
- Adds four new interfaces - MasterCoprocessor, RegionCoprocessor, RegionServierCoprocessor,
  WALCoprocessor
- These new *Coprocessor interfaces have a get*Observer() function for each observer type
  supported by them.
- Added Coprocessor#getService() to base interface. All extending *Coprocessor interfaces will
  get it from the base interface.
- Added BulkLoadObserver hooks to RegionCoprocessorHost instad of SecureBulkLoadManager doing its
  own trickery.
- CoprocessorHost#find*() fuctions: Too many testing hooks digging into CP internals.
  Deleted if can, else marked @VisibleForTesting.
------------------------------------------------------
Backward Compatibility
------------------------------------------------------
- Old coprocessors implementing *Observer won't get loaded (no backward compatibility guarantees).
- Third party coprocessors only implementing Coprocessor will not get loaded (just like Observers).
- Old coprocessors implementing CoprocessorService (for master/region host)
  /SingletonCoprocessorService (for RegionServer host) will continue to work with 2.0.
- Added test to ensure backward compatibility of CoprocessorService/SingletonCoprocessorService
- Note that if a coprocessor implements both observer and service in same class, its service
  component will continue to work but it's observer component won't work.

• Assignee: Apekshit Sharma
• Reporter: Apekshit Sharma
• Created: 04/Mar/17 00:57
• Updated: 17/Jun/22 05:20
• Resolved: 27/Sep/17 19:50

Description

The two main changes are:

• Adding template for coprocessor type to CoprocessorEnvironment i.e. interface CoprocessorEnvironment<C extends Coprocessor>
  • Enables us to load only relevant coprocessors in hosts. Right now each type of host loads all types of coprocs and it's only during execOperation that it checks if the coproc is of correct type i.e. XCoprocessorHost will load XObserver, YObserver, and all others, and will check in execOperation if coproc instanceOf XObserver and ignore the rest.
  • Allow sharing of a bunch functions/classes which are currently duplicated in each host. For eg. CoprocessorOperations, CoprocessorOperationWithResult, execOperations().
• Introduce 4 coprocessor classes and use composition between these new classes and and old observers
  • The real gold here is, moving forward, we'll be able to break down giant everything-in-one observers (masterobserver has 100+ functions) into smaller, more focused observers. These smaller observer can then have different compat guarantees!!

Here's a more detailed design doc: https://docs.google.com/document/d/1mPkM1CRRvBMZL4dBQzrus8obyvNnHhR5it2yyhiFXTg/edit?usp=sharing

Issue Links

• blocks: YARN-7213 [Umbrella] Test and validate HBase-2.0.x with Atsv2
• breaks: HBASE-21001 ReplicationObserver fails to load in HBase 2.0.0
• causes: HBASE-24329 Remove deprecated CoprocessorService
• is related to: HBASE-18776 Update coprocessor docs after HBASE-17732
• is related to: HBASE-18884 Coprocessor Design Improvements follow up of HBASE-17732
• is related to: HBASE-18898 Provide way for the core flow to know whether CP implemented each of the hooks
• relates to: HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
• links to: Old code review. Design/code changed substantially after that
• links to: Review Board (master)