Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26937

Batch events during incremental replication to avoid O.O.M

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • Hive

    Description

      • Currently incremental replication flow of hive dumps all events read from notification logs sequentially in staging directory.
      • Repl Load loads all the event directories present in staging directory to a list and processes them.
      • This has caused O.O.M issues when number of events are large.

      Hence introducing batching of events where Repl Dump dumps events in batches and Repl Load processes events batch by batch to avoid O.O.M.

      Attachments

        Issue Links

          Activity

            People

              rakshithc Rakshith C
              rakshithc Rakshith C
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h