Uploaded image for project: 'HCatalog'
  1. HCatalog
  2. HCATALOG-546

Rework HCatalog's JMS Notifications



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.4.1
    • 0.5
    • notification
    • None


      In 0.4.1, the NotificationListener listens for metastore operations and emits JMS notifications containing the entire metastore-objects (Database/Table/Partitions) in Java-serialized form. The assumption at the time was that consumers might need access to the whole object. This policy poses a couple of problems:

      1. The notifications are verbose, since it conveys a bunch of information that's available from querying the metastore anyway.

      2. Consumers of these JMS notifications (e.g. Oozie) would now be dependent on the Java class definitions of metastore-objects. If they change, Oozie would also need to be restarted (with updated libs), to consume the notifications.

      Ideally, the notifications should convey only the minimum information that identifies the metastore-change unambiguously. (Everything else can be queried for.) They should be backward compatible. If new fields are added, existing consumers shouldn't break (unless they intend to consume the new fields). Also, the notification-format ought to be pluggable.

      For the initial rework, we're proposing to use a JSON-string to represent the notification-content. We're also proposing a helper-class for the likes of Oozie to use, that converts the strings to POJOs, in a backward-compatible fashion.

      I'll attach sample notifications and a tentative patch shortly.


        1. sample.Add.Drop.Table.json
          0.2 kB
          Mithun Radhakrishnan
        2. sample.Add.Drop.Partition.json
          0.3 kB
          Mithun Radhakrishnan
        3. sample.Add.Drop.Database.json
          0.1 kB
          Mithun Radhakrishnan
        4. HCATALOG-546.trunk.rebased.patch
          78 kB
          Mithun Radhakrishnan
        5. HCATALOG-546.branch5.patch
          78 kB
          Mithun Radhakrishnan
        6. HCATALOG-546.branch4.rebased.patch
          87 kB
          Mithun Radhakrishnan

        Issue Links



              mithun Mithun Radhakrishnan
              mithun Mithun Radhakrishnan
              0 Vote for this issue
              5 Start watching this issue