[ATLAS-4878] utility to analyze hook notifications

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: atlas-core
• Assignee: Madhan Neethiraj
• Reporter: Madhan Neethiraj
• Created: 10/Jun/24 01:25
• Updated: 15/Nov/24 03:06
• Resolved: 18/Jun/24 06:42

Description

A utility to analyze notifications received from hooks to gather following details will be useful in troubleshooting:

1. number of notifications per notification type (CREATE, UPDATE, PARTIAL_UPDATE, DELETE, ..)
2. number of entities referenced in notifications per entity type
3. number of entity operations performed while processing the notifications (create/update/delete)

 

For example, following details by analyzing 114k notifications from Hive hook show that 94% of entities processed are of type hive_column and hive_column_lineage :

{
  "notifications":         114755,
  "entities":              598435,
  "notificationEntities": 2575347,
  "notificationByType": {
    "ENTITY_CREATE_V2":         49428,
    "ENTITY_FULL_UPDATE_V2":     1597,
    "ENTITY_PARTIAL_UPDATE_V2": 36561,
    "ENTITY_DELETE_V2":         27169
  },
  "notificationEntityByType": {
    "hdfs_path":             16417,
    "hive_db":               20471,
    "hive_table":            57143,
    "hive_storagedesc":      30018,
    "hive_column":          685384,
    "hive_process":          41512
    "hive_column_lineage": 1724402,
  },
  "entityOperations": {
    "CREATE":         598435,
    "UPDATE":        1913182
    "PARTIAL_UPDATE":  36561,
    "DELETE":          27169
  },
  "entityOperationsByType": {
    "CREATE": {
      "hdfs_path":            10940,
      "hive_db":                224,
      "hive_table":           22154,
      "hive_storagedesc":     15280,
      "hive_column":         332332,
      "hive_process":         23462,
      "hive_column_lineage": 194043
    },
    "UPDATE" {
      "hdfs_path":              5477,
      "hive_column":          319559,
      "hive_column_lineage": 1530359,
      "hive_db":               20203,
      "hive_process":          18050,
      "hive_storagedesc":      13204,
      "hive_table":             6330
    },
   "PARTIAL_UPDATE": {
     "hive_column":      33493,
     "hive_storagedesc":  1534,
     "hive_table":        1534
    },
    "DELETE": {
      "hive_db":       44,
      "hive_table": 27125
    }
  }
}