Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.94.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Import currently create a new Delete object for each delete KV found in a result object.
      This can be improved with the new Delete API that allows adding a delete KV to a Delete object.

      1. 5431.txt
        3 kB
        Lars Hofhansl

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #117 (See https://builds.apache.org/job/HBase-TRUNK-security/117/)
        HBASE-5431 Improve delete marker handling in Import M/R jobs (Revision 1290955)

        Result = FAILURE
        larsh :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #117 (See https://builds.apache.org/job/HBase-TRUNK-security/117/ ) HBASE-5431 Improve delete marker handling in Import M/R jobs (Revision 1290955) Result = FAILURE larsh : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
        Hide
        Lars Hofhansl added a comment -

        Committed to trunk, thanks for the review stack.

        Show
        Lars Hofhansl added a comment - Committed to trunk, thanks for the review stack.
        Hide
        Lars Hofhansl added a comment -

        Yeah, we'd need to export/import memstoreTS as well. Amit has one more thing to worry about.

        Show
        Lars Hofhansl added a comment - Yeah, we'd need to export/import memstoreTS as well. Amit has one more thing to worry about.
        Hide
        stack added a comment -

        In fact the only "correct" ordering would be to create a Put or Delete for each KV.

        Yeah. I was wondering about this.

        OK. +1.

        For Amit's patch, if we switched on his facility, then we'd export with memstorets? Though I suppose that'd be no good at import time?

        Show
        stack added a comment - In fact the only "correct" ordering would be to create a Put or Delete for each KV. Yeah. I was wondering about this. OK. +1. For Amit's patch, if we switched on his facility, then we'd export with memstorets? Though I suppose that'd be no good at import time?
        Hide
        Lars Hofhansl added a comment -

        That is to say, because all KV are timestamped they can be collected and applied in bulk (in the same way that Replication does it).

        Show
        Lars Hofhansl added a comment - That is to say, because all KV are timestamped they can be collected and applied in bulk (in the same way that Replication does it).
        Hide
        Lars Hofhansl added a comment -

        You mean Puts, then Deletes I assume ...
        Unless Amit messes things up this is correct w.r.t. timestamp ordering

        In fact the only "correct" ordering would be to create a Put or Delete for each KV.

        Show
        Lars Hofhansl added a comment - You mean Puts, then Deletes I assume ... Unless Amit messes things up this is correct w.r.t. timestamp ordering In fact the only "correct" ordering would be to create a Put or Delete for each KV.
        Hide
        stack added a comment -

        So, we output Deletes and then we output Deletes? We'll be changing the order of kvs that came in in the Result? Thats ok?

        Show
        stack added a comment - So, we output Deletes and then we output Deletes? We'll be changing the order of kvs that came in in the Result? Thats ok?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515067/5431.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 158 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/987//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/987//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/987//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515067/5431.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 158 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/987//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/987//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/987//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        Simple patch.
        The removed Delete constructor was added in 0.94 just for this case, so it's safe to remove it now.

        Show
        Lars Hofhansl added a comment - Simple patch. The removed Delete constructor was added in 0.94 just for this case, so it's safe to remove it now.

          People

          • Assignee:
            Lars Hofhansl
            Reporter:
            Lars Hofhansl
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development