Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12353

When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.0.0
    • 1.3.0, 2.0.0
    • Transactions

    Description

      One of the things that this method does is delete entries from TXN_COMPONENTS for partition that it was trying to compact.
      This causes Aborted transactions in TXNS to become empty according to
      CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be deleted.
      Once they are deleted, data that belongs to these txns is deemed committed...

      We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) states. We should also not delete then entry from markedCleaned()
      We'll have separate process that cleans 'f' and 's' records after X minutes (or after > N records for a given partition exist).
      This allows SHOW COMPACTIONS to show some history info and how many times compaction failed on a given partition (subject to retention interval) so that we don't have to call markCleaned() on Compactor failures at the same time preventing Compactor to constantly getting stuck on the same bad partition/table.

      Ideally we'd want to include END_TIME field.

      Attachments

        1. HIVE-12353.2.patch
          27 kB
          Eugene Koifman
        2. HIVE-12353.3.patch
          26 kB
          Eugene Koifman
        3. HIVE-12353.4.patch
          62 kB
          Eugene Koifman
        4. HIVE-12353.6.patch
          72 kB
          Eugene Koifman
        5. HIVE-12353.7.patch
          72 kB
          Eugene Koifman
        6. HIVE-12353.8.patch
          72 kB
          Eugene Koifman
        7. HIVE-12353.patch
          23 kB
          Eugene Koifman

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ekoifman Eugene Koifman Assign to me
            ekoifman Eugene Koifman
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment