Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-1111

Data loss is observed when atlas is restarted while hive_table metadata ingestion into kafka topic is in-progress

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 0.8-incubating
    • 0.8-incubating, 0.7.1-incubating
    • None
    • None

    Description

      During atlas stop, the graph is shutdown first and then the services like NotificationHookConsumer is shutdown. After graph is shutdown and before NotificationHookConsumer is shutdown, the message handling fails as the graph is down, but the NotificationHookConsumer commits the message(updating the offset). So, the messages during this time are lost

      Attachments

        1. ATLAS-1111.1.patch
          17 kB
          Suma Shivaprasad
        2. ATLAS-1111.2.patch
          24 kB
          Shwetha GS
        3. ATLAS-1111.3.patch
          51 kB
          Shwetha GS
        4. ATLAS-1111.4.patch
          51 kB
          Shwetha GS
        5. ATLAS-1111.5.patch
          55 kB
          Suma Shivaprasad
        6. ATLAS-1111.patch
          12 kB
          Shwetha GS

        Activity

          People

            shwethags Shwetha GS
            sharmadhas Sharmadha S
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: