Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: File Formats, Transactions
    • Labels:
      None

      Description

      A new interface ql.io.RecordUpdater was added as part of the ACID work in 0.13. This interface should extend RecordWriter because:

      1. If it does not significant portions of FileSinkOperator will have to be reworked to handle both RecordWriter and RecordUpdater
      2. Once a file format accepts transactions, it should not generally be possible to write using RecordWriter.write as that will write old style records without transaction information.

        Activity

        Alan Gates created issue -
        Alan Gates made changes -
        Field Original Value New Value
        Attachment HIVE-7098.patch [ 12645858 ]
        Alan Gates made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12645858/HIVE-7098.patch

        ERROR: -1 due to 13 failed/errored test(s), 5526 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-253/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 13 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12645858

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12645858/HIVE-7098.patch ERROR: -1 due to 13 failed/errored test(s), 5526 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/253/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-253/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12645858
        Hide
        Alan Gates added a comment -

        Ran the failing tests locally, all of which pass except root_dir_external_table which fails on trunk as well. So I conclude none of these are issues for this patch.

        Show
        Alan Gates added a comment - Ran the failing tests locally, all of which pass except root_dir_external_table which fails on trunk as well. So I conclude none of these are issues for this patch.
        Hide
        Alan Gates added a comment -

        After looking more at this I'm not sure this is the best way forward. I want to review the design further before committing this patch, so moving it from patch available to open.

        Show
        Alan Gates added a comment - After looking more at this I'm not sure this is the best way forward. I want to review the design further before committing this patch, so moving it from patch available to open.
        Alan Gates made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Alan Gates added a comment -

        After further investigation and work I don't think this makes sense. FileSinkOperator has to distinguish between whether to use write(), insert(), update(), or delete() anyway at the end, so mashing these two types together makes no sense.

        Show
        Alan Gates added a comment - After further investigation and work I don't think this makes sense. FileSinkOperator has to distinguish between whether to use write(), insert(), update(), or delete() anyway at the end, so mashing these two types together makes no sense.
        Alan Gates made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        5h 36m 1 Alan Gates 20/May/14 21:45
        Patch Available Patch Available Open Open
        42d 23h 10m 1 Alan Gates 02/Jul/14 20:55
        Open Open Resolved Resolved
        28d 19h 51m 1 Alan Gates 31/Jul/14 16:46

          People

          • Assignee:
            Alan Gates
            Reporter:
            Alan Gates
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development