Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17403

Fail concatenation for unmanaged and transactional tables

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.3.0, 2.4.0, 3.0.0
    • 2.4.0, 3.0.0
    • None
    • None

    Description

      ALTER TABLE .. CONCATENATE should fail if the table is not managed by hive.

      For unmanaged tables, file names can be anything. Hive has some assumptions about file names which can result in data loss for unmanaged tables.

      Example of this is a table/partition having 2 different files files (part-m-00000_1417075294718 and part-m-00018_1417075294718). Although both are completely different files, hive thinks these are files generated by separate instances of same task (because of failure or speculative execution). Hive will end up removing this file

      2017-08-28T18:19:29,516 WARN  [b27f10d5-d957-4695-ab2a-1453401793df main]: exec.Utilities (:()) - Duplicate taskid file removed: file:/Users/table/part=20141120/.hive-staging_hive_2017-08-28_18-19-27_210_3381701454205724533-1/_tmp.-ext-10000/part-m-00018__1417075294718 with length 958510. Existing file: file:/Users/table/part=20141120/.hive-staging_hive_2017-08-28_18-19-27_210_3381701454205724533-1/_tmp.-ext-10000/part-m-00000__1417075294718 with length 1123116
      

      DDL should restrict concatenation for unmanaged tables.

      Attachments

        1. HIVE-17403.1.patch
          25 kB
          Prasanth Jayachandran
        2. HIVE-17403.2.patch
          29 kB
          Prasanth Jayachandran
        3. HIVE-17403.2.patch
          32 kB
          Prasanth Jayachandran
        4. HIVE-17403.3.patch
          28 kB
          Prasanth Jayachandran

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            prasanth_j Prasanth Jayachandran Assign to me
            prasanth_j Prasanth Jayachandran
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment