Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Tool to replay WAL files using a M/R job.

      The WAL can be replayed for a set of tables or all tables, and a timerange can be provided (in milliseconds).
      The WAL is filtered to this set of tables, the output can optionally be mapped to another set of tables.

      WAL replay can also generate HFiles for later bulk importing, in that case the WAL is replayed for a single table only.
      Show
      Tool to replay WAL files using a M/R job. The WAL can be replayed for a set of tables or all tables, and a timerange can be provided (in milliseconds). The WAL is filtered to this set of tables, the output can optionally be mapped to another set of tables. WAL replay can also generate HFiles for later bulk importing, in that case the WAL is replayed for a single table only.

      Description

      Just an idea I had. Might be useful for restore of a backup using the HLogs.
      This could an M/R (with a mapper per HLog file).

      The tool would get a timerange and a (set of) table(s). We'd pick the right HLogs based on time before the M/R job is started and then have a mapper per HLog file.
      The mapper would then go through the HLog, filter all WALEdits that didn't fit into the time range or are not any of the tables and then uses HFileOutputFormat to generate HFiles.
      Would need to indicate the splits we want, probably from a live table.

      1. 5604-v10.txt
        36 kB
        Lars Hofhansl
      2. 5604-v11.txt
        37 kB
        Lars Hofhansl
      3. 5604-v4.txt
        24 kB
        Lars Hofhansl
      4. 5604-v6.txt
        29 kB
        Lars Hofhansl
      5. 5604-v7.txt
        29 kB
        Lars Hofhansl
      6. 5604-v8.txt
        34 kB
        Lars Hofhansl
      7. 5604-v9.txt
        36 kB
        Lars Hofhansl
      8. HLog-5604-v3.txt
        16 kB
        Lars Hofhansl

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #171 (See https://builds.apache.org/job/HBase-TRUNK-security/171/)
          HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326002)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #171 (See https://builds.apache.org/job/HBase-TRUNK-security/171/ ) HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326002) Result = FAILURE larsh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2757 (See https://builds.apache.org/job/HBase-TRUNK/2757/)
          HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326002)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2757 (See https://builds.apache.org/job/HBase-TRUNK/2757/ ) HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326002) Result = FAILURE larsh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #114 (See https://builds.apache.org/job/HBase-0.94/114/)
          HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326001)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #114 (See https://builds.apache.org/job/HBase-0.94/114/ ) HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326001) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #10 (See https://builds.apache.org/job/HBase-0.94-security/10/)
          HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326001)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #10 (See https://builds.apache.org/job/HBase-0.94-security/10/ ) HBASE-5604 Addendum. Remove test as per discussion on jira. (Revision 1326001) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Lars Hofhansl added a comment -

          OK. Done. I'll use that build as RC1.

          Show
          Lars Hofhansl added a comment - OK. Done. I'll use that build as RC1.
          Hide
          Lars Hofhansl added a comment -

          @Stack and Ted: I'll just remove that test. The HFile pretty printer will show time in the same TZ as WALPlayer would parse it, which is still useful.
          I'll not file an extra jira but post an addendum to this one.

          Show
          Lars Hofhansl added a comment - @Stack and Ted: I'll just remove that test. The HFile pretty printer will show time in the same TZ as WALPlayer would parse it, which is still useful. I'll not file an extra jira but post an addendum to this one.
          Hide
          Ted Yu added a comment -

          See the following code from http://stackoverflow.com/questions/2375222/java-simpledateformat-for-time-zone-with-a-colon-seperator:

          String dateString = "2010-03-01T00:00:00-08:00";
          String pattern = "yyyy-MM-dd'T'HH:mm:ss";
          SimpleDateFormat sdf = new SimpleDateFormat(pattern);
          Date date = sdf.parse(dateString);
          
          Show
          Ted Yu added a comment - See the following code from http://stackoverflow.com/questions/2375222/java-simpledateformat-for-time-zone-with-a-colon-seperator: String dateString = "2010-03-01T00:00:00-08:00" ; String pattern = "yyyy-MM-dd'T'HH:mm:ss" ; SimpleDateFormat sdf = new SimpleDateFormat(pattern); Date date = sdf.parse(dateString);
          Hide
          stack added a comment -

          Remove the Date stuff. Just do basic ms.

          Show
          stack added a comment - Remove the Date stuff. Just do basic ms.
          Hide
          Lars Hofhansl added a comment -

          Sigh... That's exactly what I was afraid about when adding data parsing.
          What timezone is used to store the WAL edit? Is it GMT? If so, the date should be interpreted as GMT.
          If the data in the WAL is always the local TZ, I should remove the date verification test (which in a sense is pointless anyway, because it just tests whether SimpleDateFormat works).

          Show
          Lars Hofhansl added a comment - Sigh... That's exactly what I was afraid about when adding data parsing. What timezone is used to store the WAL edit? Is it GMT? If so, the date should be interpreted as GMT. If the data in the WAL is always the local TZ, I should remove the date verification test (which in a sense is pointless anyway, because it just tests whether SimpleDateFormat works).
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #9 (See https://builds.apache.org/job/HBase-0.94-security/9/)
          HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325608)
          HBASE-5604 M/R tool to replay WAL files (Revision 1325562)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java

          larsh :
          Files :

          • /hbase/branches/0.94/src/docbkx/ops_mgt.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #9 (See https://builds.apache.org/job/HBase-0.94-security/9/ ) HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325608) HBASE-5604 M/R tool to replay WAL files (Revision 1325562) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java larsh : Files : /hbase/branches/0.94/src/docbkx/ops_mgt.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Ted Yu added a comment -

          Looking at test failure reported by Hadoop QA: https://builds.apache.org/job/PreCommit-HBASE-Build/1514//testReport/org.apache.hadoop.hbase.mapreduce/TestWALPlayer/testTimeFormat/

          java.lang.AssertionError: expected:<1334092861001> but was:<1334067661001>
          

          I wonder if timezone could be an issue here - the difference is 7 hours.

          If you don't want to involve call such as setTimeZone(TimeZone.getTimeZone(“America/Los_Angeles”)), please comment out:

              assertEquals(1334092861001L, conf.getLong(HLogInputFormat.END_TIME_KEY, 0));
          
          Show
          Ted Yu added a comment - Looking at test failure reported by Hadoop QA: https://builds.apache.org/job/PreCommit-HBASE-Build/1514//testReport/org.apache.hadoop.hbase.mapreduce/TestWALPlayer/testTimeFormat/ java.lang.AssertionError: expected:<1334092861001> but was:<1334067661001> I wonder if timezone could be an issue here - the difference is 7 hours. If you don't want to involve call such as setTimeZone(TimeZone.getTimeZone(“America/Los_Angeles”)), please comment out: assertEquals(1334092861001L, conf.getLong(HLogInputFormat.END_TIME_KEY, 0));
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #169 (See https://builds.apache.org/job/HBase-TRUNK-security/169/)
          HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325609)
          HBASE-5604 M/R tool to replay WAL files (Revision 1325555)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java

          larsh :
          Files :

          • /hbase/trunk/src/docbkx/ops_mgt.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #169 (See https://builds.apache.org/job/HBase-TRUNK-security/169/ ) HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325609) HBASE-5604 M/R tool to replay WAL files (Revision 1325555) Result = FAILURE larsh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java larsh : Files : /hbase/trunk/src/docbkx/ops_mgt.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2750 (See https://builds.apache.org/job/HBase-TRUNK/2750/)
          HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325609)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2750 (See https://builds.apache.org/job/HBase-TRUNK/2750/ ) HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325609) Result = SUCCESS larsh : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #111 (See https://builds.apache.org/job/HBase-0.94/111/)
          HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325608)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #111 (See https://builds.apache.org/job/HBase-0.94/111/ ) HBASE-5604 addendum, mark TestWALPlayer as LargeTest (Revision 1325608) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Lars Hofhansl added a comment -

          Done. Thanks Ted.

          Show
          Lars Hofhansl added a comment - Done. Thanks Ted.
          Hide
          Lars Hofhansl added a comment -

          Yes, should do an addendum. Will mark as large.

          Show
          Lars Hofhansl added a comment - Yes, should do an addendum. Will mark as large.
          Hide
          Ted Yu added a comment -

          TestWALPlayer.java doesn't have test category.

          Show
          Ted Yu added a comment - TestWALPlayer.java doesn't have test category.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #109 (See https://builds.apache.org/job/HBase-0.94/109/)
          HBASE-5604 M/R tool to replay WAL files (Revision 1325562)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/docbkx/ops_mgt.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #109 (See https://builds.apache.org/job/HBase-0.94/109/ ) HBASE-5604 M/R tool to replay WAL files (Revision 1325562) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/docbkx/ops_mgt.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2749 (See https://builds.apache.org/job/HBase-TRUNK/2749/)
          HBASE-5604 M/R tool to replay WAL files (Revision 1325555)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/src/docbkx/ops_mgt.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2749 (See https://builds.apache.org/job/HBase-TRUNK/2749/ ) HBASE-5604 M/R tool to replay WAL files (Revision 1325555) Result = FAILURE larsh : Files : /hbase/trunk/src/docbkx/ops_mgt.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
          Hide
          Lars Hofhansl added a comment -

          And I did insert the spaces before commit

          Show
          Lars Hofhansl added a comment - And I did insert the spaces before commit
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.94 and 0.96.
          Thanks for the reviews Stack and Ted!

          Show
          Lars Hofhansl added a comment - Committed to 0.94 and 0.96. Thanks for the reviews Stack and Ted!
          Hide
          Ted Yu added a comment -

          Patch looks good.
          Minor comment:

          +      HLog.Entry temp;
          +      long i=-1;
          

          Insert spaces around equal sign above.

          Show
          Ted Yu added a comment - Patch looks good. Minor comment: + HLog.Entry temp; + long i=-1; Insert spaces around equal sign above.
          Hide
          Lars Hofhansl added a comment -

          @Ted: Are you OK with latest patch? If so, I'll commit today.

          Show
          Lars Hofhansl added a comment - @Ted: Are you OK with latest patch? If so, I'll commit today.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522473/5604-v11.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplication
          org.apache.hadoop.hbase.client.TestFromClientSide
          org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522473/5604-v11.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1498//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Patch that includes the documentation from HBASE-5774.

          Show
          Lars Hofhansl added a comment - Patch that includes the documentation from HBASE-5774 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522454/5604-v10.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522454/5604-v10.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1497//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          all arc lint problems fixed.

          Show
          Lars Hofhansl added a comment - all arc lint problems fixed.
          Hide
          Ted Yu added a comment -

          If you run the patch through 'arc lint', you would see (just excerpt):

             Warning  (TXT3) Line Too Long
              This line is 106 characters long, but the convention is 100 characters.
          
                       274     System.err.println("  -D" + BULK_OUTPUT_CONF_KEY + "=/path/for/output");
                       275     System.err.println("  (Exactly one table must be specified for this option, and not mapping can be specified!)");
                       276     System.err.println("Other options:");
              >>>      277     System.err.println("  -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)");
                       278     System.err.println("  -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)");
                       279     System.err.println("For performance also consider the following options:\n"
                       280         + "  -Dmapred.map.tasks.speculative.execution=false\n"
          
             Warning  (TXT3) Line Too Long
              This line is 105 characters long, but the convention is 100 characters.
          
                       275     System.err.println("  (Exactly one table must be specified for this option, and not mapping can be specified!)");
                       276     System.err.println("Other options:");
                       277     System.err.println("  -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)");
              >>>      278     System.err.println("  -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)");
                       279     System.err.println("For performance also consider the following options:\n"
                       280         + "  -Dmapred.map.tasks.speculative.execution=false\n"
                       281         + "  -Dmapred.reduce.tasks.speculative.execution=false");
          

          Below is a long line:

          +        throw new IOException(option + " must be specified either in the form 2001-02-20T16:35:06.99 or as a number of milliseconds");
          

          Minor comment: the 'a' in 'as a number' should be omitted.

          Show
          Ted Yu added a comment - If you run the patch through 'arc lint', you would see (just excerpt): Warning (TXT3) Line Too Long This line is 106 characters long , but the convention is 100 characters. 274 System .err.println( " -D" + BULK_OUTPUT_CONF_KEY + "=/path/ for /output" ); 275 System .err.println( " (Exactly one table must be specified for this option, and not mapping can be specified!)" ); 276 System .err.println( "Other options:" ); >>> 277 System .err.println( " -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)" ); 278 System .err.println( " -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)" ); 279 System .err.println( "For performance also consider the following options:\n" 280 + " -Dmapred.map.tasks.speculative.execution= false \n" Warning (TXT3) Line Too Long This line is 105 characters long , but the convention is 100 characters. 275 System .err.println( " (Exactly one table must be specified for this option, and not mapping can be specified!)" ); 276 System .err.println( "Other options:" ); 277 System .err.println( " -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)" ); >>> 278 System .err.println( " -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)" ); 279 System .err.println( "For performance also consider the following options:\n" 280 + " -Dmapred.map.tasks.speculative.execution= false \n" 281 + " -Dmapred.reduce.tasks.speculative.execution= false " ); Below is a long line: + throw new IOException(option + " must be specified either in the form 2001-02-20T16:35:06.99 or as a number of milliseconds" ); Minor comment: the 'a' in 'as a number' should be omitted.
          Hide
          Lars Hofhansl added a comment -

          Since this is only new code I propose this for 0.94.0 as well.

          Show
          Lars Hofhansl added a comment - Since this is only new code I propose this for 0.94.0 as well.
          Hide
          stack added a comment -

          None +1 on commit.

          Show
          stack added a comment - None +1 on commit.
          Hide
          Lars Hofhansl added a comment -

          Any objections to the latest patch?

          Show
          Lars Hofhansl added a comment - Any objections to the latest patch?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522168/5604-v9.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522168/5604-v9.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1470//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Added data parsing, if you hate it, I'll pull. Might be useful, as the date is human readable so the operator can double check.
          Includes a simple data parsing test.

          Show
          Lars Hofhansl added a comment - Added data parsing, if you hate it, I'll pull. Might be useful, as the date is human readable so the operator can double check. Includes a simple data parsing test.
          Hide
          Lars Hofhansl added a comment -

          @Stack: Yeah, probably. You'd vote for leaving it the way it is?

          Show
          Lars Hofhansl added a comment - @Stack: Yeah, probably. You'd vote for leaving it the way it is?
          Hide
          stack added a comment -

          The parsing would be done with SimpleDateFormat.

          If user entered data exactly right (they probably will have looked at hbase, done math w/ ms's, then converted to date to pass your tool only to have it convert back to ms).

          Show
          stack added a comment - The parsing would be done with SimpleDateFormat. If user entered data exactly right (they probably will have looked at hbase, done math w/ ms's, then converted to date to pass your tool only to have it convert back to ms).
          Hide
          Lars Hofhansl added a comment -

          The parsing would be done with SimpleDateFormat. I was going a bit back and forth about this. Could accept both actually, try SimpleDateFormat first, if there's a parse error use ms.
          Not sure which part would do that parsing. Can be done in WALPlayer, or down at HLogInputFormat.

          Show
          Lars Hofhansl added a comment - The parsing would be done with SimpleDateFormat. I was going a bit back and forth about this. Could accept both actually, try SimpleDateFormat first, if there's a parse error use ms. Not sure which part would do that parsing. Can be done in WALPlayer, or down at HLogInputFormat.
          Hide
          stack added a comment -

          Thought about using SimpleDateFormat, then punted. I guess you're right, should make it bit more user friendly.

          Passing dates instead of ms will be a pain doing the parse. Outputting a date instead of ms will be of little use when hbase only shows ms.

          Show
          stack added a comment - Thought about using SimpleDateFormat, then punted. I guess you're right, should make it bit more user friendly. Passing dates instead of ms will be a pain doing the parse. Outputting a date instead of ms will be of little use when hbase only shows ms.
          Hide
          Lars Hofhansl added a comment -
          • Let's not overkill on the exceptions. This is an inner class of WALPlayer, WALPlayer will always pass the correct arguments. Maybe I'll make the mapper classed private and remove all exception handling.
          • Thought about using SimpleDateFormat, then punted. I guess you're right, should make it bit more user friendly.
          • The TODO was left over. Not sure that even makes sense.
          Show
          Lars Hofhansl added a comment - Let's not overkill on the exceptions. This is an inner class of WALPlayer, WALPlayer will always pass the correct arguments. Maybe I'll make the mapper classed private and remove all exception handling. Thought about using SimpleDateFormat, then punted. I guess you're right, should make it bit more user friendly. The TODO was left over. Not sure that even makes sense.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522068/5604-v8.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestFromClientSide

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522068/5604-v8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1460//console This message is automatically generated.
          Hide
          Ted Yu added a comment -
          +      if (tablesToUse == null || tableMap == null || tablesToUse.length != tableMap.length) {
          +        // this can only happen when HLogMapper is used directly by a class other than WALPlayer
          +        throw new IOException("No tables or incorrect table mapping specified.");
          

          I think if we provide separate exceptions for the first two checks and the last check, it would be easier for user to understand.

          +    System.err.println("  -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)");
          +    System.err.println("  -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)");
          

          User would have to resort to conversion tool in order to find out the ms readings for desired date / time. Can we make this more user-friendly ?
          e.g. in TimeStampingFileContext.java we have:

              this.sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
          

          See also http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html#parse%28java.lang.String,%20java.text.ParsePosition%29

          +    public String[] getLocations() throws IOException, InterruptedException {
          +      // TODO: Find the data node with the most blocks for this HLog?
          

          Would the above be addressed in a separate JIRA ?

          +        if (i>0) LOG.info("Skipped " + i + " entries.");
          

          Minor: add spaces around '>'

          Show
          Ted Yu added a comment - + if (tablesToUse == null || tableMap == null || tablesToUse.length != tableMap.length) { + // this can only happen when HLogMapper is used directly by a class other than WALPlayer + throw new IOException( "No tables or incorrect table mapping specified." ); I think if we provide separate exceptions for the first two checks and the last check, it would be easier for user to understand. + System .err.println( " -D" + HLogInputFormat.START_TIME_KEY + "=ms (only apply edit after this time)" ); + System .err.println( " -D" + HLogInputFormat.END_TIME_KEY + "=ms (only apply edit before this time)" ); User would have to resort to conversion tool in order to find out the ms readings for desired date / time. Can we make this more user-friendly ? e.g. in TimeStampingFileContext.java we have: this .sdf = new SimpleDateFormat( "yyyy-MM-dd'T'HH:mm:ss" ); See also http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html#parse%28java.lang.String,%20java.text.ParsePosition%29 + public String [] getLocations() throws IOException, InterruptedException { + // TODO: Find the data node with the most blocks for this HLog? Would the above be addressed in a separate JIRA ? + if (i>0) LOG.info( "Skipped " + i + " entries." ); Minor: add spaces around '>'
          Hide
          Lars Hofhansl added a comment -

          Addressed Ted's comment (except for Put.heapSize()). Also actually included the WALPlayer end-to-end test this time.

          Show
          Lars Hofhansl added a comment - Addressed Ted's comment (except for Put.heapSize()). Also actually included the WALPlayer end-to-end test this time.
          Hide
          Lars Hofhansl added a comment -

          Thanks Ted.

          • I'll add Javadoc.
          • index of 0 is used here, since when creating HFiles for bulk import only a single table is currently allowed (that is also documented in usage(), but perhaps not clearly enough...?). I'll add a comment to that extent.
          • That the two arrays are of the same size if guaranteed in WALPlayer.createSubmittableJob(), but perhaps it is better to double check here.
          • Checking heapSize() seems unnecessary. After all, this is single WALEdit.
          Show
          Lars Hofhansl added a comment - Thanks Ted. I'll add Javadoc. index of 0 is used here, since when creating HFiles for bulk import only a single table is currently allowed (that is also documented in usage(), but perhaps not clearly enough...?). I'll add a comment to that extent. That the two arrays are of the same size if guaranteed in WALPlayer.createSubmittableJob(), but perhaps it is better to double check here. Checking heapSize() seems unnecessary. After all, this is single WALEdit.
          Hide
          Ted Yu added a comment -
          +public class WALPlayer extends Configured implements Tool {
          

          Javadoc for the above new class is desirable.

          +    public void setup(Context context) {
          +      table = Bytes.toBytes(context.getConfiguration().getStrings(TABLES_KEY)[0]);
          +    }
          

          Why index of 0 is always used above ?

          +    public void setup(Context context) {
          +      String[] tableMap = context.getConfiguration().getStrings(TABLE_MAP_KEY);
          +      int i = 0;
          +      for (String table : context.getConfiguration().getStrings(TABLES_KEY)) {
          +        tables.put(Bytes.toBytes(table), Bytes.toBytes(tableMap[i++]));
          

          I think validation on the lengths of the two String[] should be performed. If they don't match, bail out early.

          +            // Aggregate as much as possible into a single Put/Delete
          +            // operation before writing to the context.
          

          Shall we utilize Put.heapSize() and remember the aggregate size of the Put so that we can write to context when certain threshold is reached ?

          Show
          Ted Yu added a comment - + public class WALPlayer extends Configured implements Tool { Javadoc for the above new class is desirable. + public void setup(Context context) { + table = Bytes.toBytes(context.getConfiguration().getStrings(TABLES_KEY)[0]); + } Why index of 0 is always used above ? + public void setup(Context context) { + String [] tableMap = context.getConfiguration().getStrings(TABLE_MAP_KEY); + int i = 0; + for ( String table : context.getConfiguration().getStrings(TABLES_KEY)) { + tables.put(Bytes.toBytes(table), Bytes.toBytes(tableMap[i++])); I think validation on the lengths of the two String[] should be performed. If they don't match, bail out early. + // Aggregate as much as possible into a single Put/Delete + // operation before writing to the context. Shall we utilize Put.heapSize() and remember the aggregate size of the Put so that we can write to context when certain threshold is reached ?
          Hide
          Lars Hofhansl added a comment -

          TestForceCacheImportantBlocks passes locally. I'm ready to commit.

          Show
          Lars Hofhansl added a comment - TestForceCacheImportantBlocks passes locally. I'm ready to commit.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522035/5604-v7.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522035/5604-v7.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1458//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          This should be close to what I would like to commit.
          Added a simple end-to-end test for WALPlayer.

          Show
          Lars Hofhansl added a comment - This should be close to what I would like to commit. Added a simple end-to-end test for WALPlayer.
          Hide
          stack added a comment -

          All sounds good Lars.

          Show
          stack added a comment - All sounds good Lars.
          Hide
          Lars Hofhansl added a comment -

          Supports multiple tables or all tables now (unless bulk output is used).
          Added more tests.

          This is still just for parking, I plan to add some high level end-to-end tests.

          Show
          Lars Hofhansl added a comment - Supports multiple tables or all tables now (unless bulk output is used). Added more tests. This is still just for parking, I plan to add some high level end-to-end tests.
          Hide
          Lars Hofhansl added a comment -

          I also think it is still important to be able to import log entries to a different table. So I'll allow providing two comma separated lists of table names. The 2nd one is optional, and if provided it will be used to derive the respective target tables.

          If the case of BulkExport only a single table name and no mapping will be allowed.
          Or maybe I should get rid of that option now...?

          Show
          Lars Hofhansl added a comment - I also think it is still important to be able to import log entries to a different table. So I'll allow providing two comma separated lists of table names. The 2nd one is optional, and if provided it will be used to derive the respective target tables. If the case of BulkExport only a single table name and no mapping will be allowed. Or maybe I should get rid of that option now...?
          Hide
          Lars Hofhansl added a comment -

          Thanks Stack.

          The meta check is actually for "meta" entries in the WAL (see HLog.completeCacheFlushLogEdit), not edits to the ROOT or META. I'll add a comment to that extend. ROOT and META could be passed as <tablename> and it would do the right thing.

          The printStackTrace I took from Import. Just means that the message will show up in the Jobs stderr rather then syslog. I think that is what we want? Can change, though.

          Playing all edits or edit from multiple tables does not allow me to use TableOutputFormat or HFileOutputFormat. I can use MultiTableOutputFormat for the live cluster case, and change the mappers to output the tablename as key.
          For bulk output to HFiles that would tricky. +1 on doing that, though, for a full backup option.

          Maybe allow all or a set of tables for the live cluster case and only allow a single table for the Bulk output case. I'll do that.

          Re: coding style... I thought I followed the standard Will use FBs lint.

          Show
          Lars Hofhansl added a comment - Thanks Stack. The meta check is actually for "meta" entries in the WAL (see HLog.completeCacheFlushLogEdit), not edits to the ROOT or META. I'll add a comment to that extend. ROOT and META could be passed as <tablename> and it would do the right thing. The printStackTrace I took from Import. Just means that the message will show up in the Jobs stderr rather then syslog. I think that is what we want? Can change, though. Playing all edits or edit from multiple tables does not allow me to use TableOutputFormat or HFileOutputFormat. I can use MultiTableOutputFormat for the live cluster case, and change the mappers to output the tablename as key. For bulk output to HFiles that would tricky. +1 on doing that, though, for a full backup option. Maybe allow all or a set of tables for the live cluster case and only allow a single table for the Bulk output case. I'll do that. Re: coding style... I thought I followed the standard Will use FBs lint.
          Hide
          stack added a comment -

          On the patch, can we play all edits from all passed in WALs or does it have to be by a specific table only? I think it'd be nice to be able to pass a list of tables but thats just a 'nice-to-have'. I think it more important we play edits for one table or all (Could do the 'all' in another JIRA. Could add time range to include in another issue too)

          We check if a meta table and if it is one, we just skip? Whats up w/ that? Maybe I want to use this tool to restore meta table? If you don't want it used on meta, fail earlier if we are passed a meta table to use as table.

          Make this a LOG rather than a stdout + e.printStackTrace();

          Make it a versionedwritable instead in the below?

          + static class HLogSplit extends InputSplit implements Writable {

          Usual style is to have space around operators. Try the fb lint on your patch?

          Patch looks good. +1 on commit. Add a release note.

          Show
          stack added a comment - On the patch, can we play all edits from all passed in WALs or does it have to be by a specific table only? I think it'd be nice to be able to pass a list of tables but thats just a 'nice-to-have'. I think it more important we play edits for one table or all (Could do the 'all' in another JIRA. Could add time range to include in another issue too) We check if a meta table and if it is one, we just skip? Whats up w/ that? Maybe I want to use this tool to restore meta table? If you don't want it used on meta, fail earlier if we are passed a meta table to use as table. Make this a LOG rather than a stdout + e.printStackTrace(); Make it a versionedwritable instead in the below? + static class HLogSplit extends InputSplit implements Writable { Usual style is to have space around operators. Try the fb lint on your patch? Patch looks good. +1 on commit. Add a release note.
          Hide
          Lars Hofhansl added a comment -

          Maybe we could do it only when the file is renamed anyway, such as when it is moved to .oldlogs. It's not really needed for this to work correctly.

          Show
          Lars Hofhansl added a comment - Maybe we could do it only when the file is renamed anyway, such as when it is moved to .oldlogs. It's not really needed for this to work correctly.
          Hide
          stack added a comment -

          Its kinda ugly doing the rename when I think on it. Would be nice having the metadata but the rename could take a little time to do going against the namenode. We could do it offline out of line w/ the edits but it could always fail. Should never really happen but that it could means that we should be able to handle both name formats.

          Show
          stack added a comment - Its kinda ugly doing the rename when I think on it. Would be nice having the metadata but the rename could take a little time to do going against the namenode. We could do it offline out of line w/ the edits but it could always fail. Should never really happen but that it could means that we should be able to handle both name formats.
          Hide
          Lars Hofhansl added a comment -

          New version. Should hold up to some amount of scrutiny now.
          Added an initial test to verify basic HLogInputFormat/HLogReader functionality.

          Show
          Lars Hofhansl added a comment - New version. Should hold up to some amount of scrutiny now. Added an initial test to verify basic HLogInputFormat/HLogReader functionality.
          Hide
          Lars Hofhansl added a comment -

          Related to this, when I say point in time (PIT) restore, I mean a PIT w.r.t. to the write time of the WAL edits (not the timestamps of the KeyValues).
          Although I can see a usecase for the latter as well, especially when the timestamps are used for snapshot isolation.

          Show
          Lars Hofhansl added a comment - Related to this, when I say point in time (PIT) restore, I mean a PIT w.r.t. to the write time of the WAL edits (not the timestamps of the KeyValues). Although I can see a usecase for the latter as well, especially when the timestamps are used for snapshot isolation.
          Hide
          Lars Hofhansl added a comment -

          Should we rename WALs on close so they have the start and end time as their name?

          Thinking more about this. It would be quite useful, possibly for other cases in the future, if one could tell by the file the exact time range of the edits.
          Do you have a feeling for what (if anything) this would break?

          Show
          Lars Hofhansl added a comment - Should we rename WALs on close so they have the start and end time as their name? Thinking more about this. It would be quite useful, possibly for other cases in the future, if one could tell by the file the exact time range of the edits. Do you have a feeling for what (if anything) this would break?
          Hide
          Lars Hofhansl added a comment -

          This is what I have so far. Pretty simple.

          It's not finished, and I only sporadically get to work on this.

          Did some testing, both playing directly into a table and writing to HFiles worked fine.
          I have not tested time based filtering, yet.

          Repeat: This is not ready, I just need to store this somewhere

          Show
          Lars Hofhansl added a comment - This is what I have so far. Pretty simple. It's not finished, and I only sporadically get to work on this. Did some testing, both playing directly into a table and writing to HFiles worked fine. I have not tested time based filtering, yet. Repeat: This is not ready, I just need to store this somewhere
          Hide
          Lars Hofhansl added a comment -

          Ok... I'll do a separate tool called WALPlayer. The Import rationale would be that indeed right now it can import HLog files. I added HFiles output to Import in HBASE-5440.
          HDFS would probably not work, as these files would be copied around for archiving.
          Changing the names of WALs is interesting. I'd be worried about side-effects to other tools.

          Maybe it's not even an issue. A reader of the HLog can tell after looking at the first log entries whether it can ignore the rest of the file.

          Show
          Lars Hofhansl added a comment - Ok... I'll do a separate tool called WALPlayer. The Import rationale would be that indeed right now it can import HLog files. I added HFiles output to Import in HBASE-5440 . HDFS would probably not work, as these files would be copied around for archiving. Changing the names of WALs is interesting. I'd be worried about side-effects to other tools. Maybe it's not even an issue. A reader of the HLog can tell after looking at the first log entries whether it can ignore the rest of the file.
          Hide
          stack added a comment -

          I like the name WALPlayer.

          How is this related to Import at all? (Import imports WAL files?)

          Does Import write HFiles?

          Does the hdfs mod time get updated when you close the files? If so, that might work for when the file is in postion but won't work if file gets moved to .oldlogs or to archive? Should we add a tail on a WAL with metadata of fixed size so you know where to start reading to pick it up? I suppose you can't rely on the fact that all WALs are present? If they were, you could use the start date of the next WAL (after sorting them by date) as the ending date of the current file.

          Should we rename WALs on close so they have the start and end time as their name?

          Show
          stack added a comment - I like the name WALPlayer. How is this related to Import at all? (Import imports WAL files?) Does Import write HFiles? Does the hdfs mod time get updated when you close the files? If so, that might work for when the file is in postion but won't work if file gets moved to .oldlogs or to archive? Should we add a tail on a WAL with metadata of fixed size so you know where to start reading to pick it up? I suppose you can't rely on the fact that all WALs are present? If they were, you could use the start date of the next WAL (after sorting them by date) as the ending date of the current file. Should we rename WALs on close so they have the start and end time as their name?
          Hide
          Lars Hofhansl added a comment -

          I have something basic working now. Needs a bunch of polishing, but it works in principle.

          Right now I have this hooked up with Import. I.e. Import can optionally import from a directory that contains HLog files (at any depth).

          Does it make sense to keep this with Import? The advantage is that there is one place for Importing stuff, and things like CF mapping exist already. On the other hand Import is becoming a bit overloaded with options now and something PlayLogs might be better along ImportTsv and Import. Can still work out how to share some of the code.

          Also from the name of an HLog file I can tell when the first entry was written to it, but not what the last entry is. Is there a way to find this out? It would allow me to filter HLog files of it is known that they do not fall in the requested time range.

          Show
          Lars Hofhansl added a comment - I have something basic working now. Needs a bunch of polishing, but it works in principle. Right now I have this hooked up with Import. I.e. Import can optionally import from a directory that contains HLog files (at any depth). Does it make sense to keep this with Import? The advantage is that there is one place for Importing stuff, and things like CF mapping exist already. On the other hand Import is becoming a bit overloaded with options now and something PlayLogs might be better along ImportTsv and Import. Can still work out how to share some of the code. Also from the name of an HLog file I can tell when the first entry was written to it, but not what the last entry is. Is there a way to find this out? It would allow me to filter HLog files of it is known that they do not fall in the requested time range.
          Hide
          stack added a comment -

          So, just need to write a smart mapper that takes filtering params and a WALInputFormat?

          Show
          stack added a comment - So, just need to write a smart mapper that takes filtering params and a WALInputFormat?
          Hide
          Lars Hofhansl added a comment -

          I went the distributed log splitting route first a while back. Had it all working in fact.
          Or so I thought. Then I tried playing logs to a new table and realized that distributed log splitting only works for crash recovery (before the regions could split further) because it splits using the region name in the log.
          That makes sense, because otherwise each region server participating in log splitting would need to look up the current region for each encountered row otherwise (the region could have split and the row in question needs to go one of the daugthers). That is essentially what the highlevel API does anyway.

          Yes, definitely need a reducer.

          Similar to what I did for Import, I can see this working in two modes:

          1. The mappers directly apply changes to a running HBase cluster (TableOutputformat). No reducers needed in this case.
          2. Create HFiles via HFileOutputFormat in the reduce phase.

          In fact this tool would probably be very much like Import, "just" with a different InputFormat.

          Show
          Lars Hofhansl added a comment - I went the distributed log splitting route first a while back. Had it all working in fact. Or so I thought. Then I tried playing logs to a new table and realized that distributed log splitting only works for crash recovery (before the regions could split further) because it splits using the region name in the log. That makes sense, because otherwise each region server participating in log splitting would need to look up the current region for each encountered row otherwise (the region could have split and the row in question needs to go one of the daugthers). That is essentially what the highlevel API does anyway. Yes, definitely need a reducer. Similar to what I did for Import, I can see this working in two modes: The mappers directly apply changes to a running HBase cluster (TableOutputformat). No reducers needed in this case. Create HFiles via HFileOutputFormat in the reduce phase. In fact this tool would probably be very much like Import, "just" with a different InputFormat.
          Hide
          stack added a comment -

          I like the postgres link. Should point at that when we doc this tool.

          This could an M/R (with a mapper per HLog file).

          You'll need a reduce (I think you deduce this yourself above but saying it for completeness). If a map-only MR job, if many WALs, say 100s, you could end up w/ the same amount of hfiles per region (if each WAL had at least one edit for this region). You'd need a reducer to coalesce by region.

          This tool would not apply the edits in the order in which we received them. We'd be reliant on sort order only which should be fine I think since this is what happens if they were instead inserted via the memstore anyways.

          Hmm... Maybe this is only useful when we have a lot of logs ....maybe there would be no advantage here turning this in an M/R job, but maybe it should just be a standalone client...?

          Well, even if tens of files only, you'd want to //ize it to do the filtering, etc., so MR sounds right.

          Or you could hack on the distributed split code to add a 'filtering' facility... so it dropped edits that were outside of a range – e.g. not one of the specified tables or not of a time range. The output of distributed log splitting is only replayed on region open so you'd need to figure how to get the region to load the edits (An MR job to write hfiles sounds way more straightforward relatively).

          Show
          stack added a comment - I like the postgres link. Should point at that when we doc this tool. This could an M/R (with a mapper per HLog file). You'll need a reduce (I think you deduce this yourself above but saying it for completeness). If a map-only MR job, if many WALs, say 100s, you could end up w/ the same amount of hfiles per region (if each WAL had at least one edit for this region). You'd need a reducer to coalesce by region. This tool would not apply the edits in the order in which we received them. We'd be reliant on sort order only which should be fine I think since this is what happens if they were instead inserted via the memstore anyways. Hmm... Maybe this is only useful when we have a lot of logs ....maybe there would be no advantage here turning this in an M/R job, but maybe it should just be a standalone client...? Well, even if tens of files only, you'd want to //ize it to do the filtering, etc., so MR sounds right. Or you could hack on the distributed split code to add a 'filtering' facility... so it dropped edits that were outside of a range – e.g. not one of the specified tables or not of a time range. The output of distributed log splitting is only replayed on region open so you'd need to figure how to get the region to load the edits (An MR job to write hfiles sounds way more straightforward relatively).
          Hide
          Lars Hofhansl added a comment -

          Yep, that's what I have in mind. Can do PITR within HBase's time resolution.

          We have the requirement to keep historical data around for our customers.

          Replication is for disaster recovery (main cluster died, we will do that too).
          Backpup with PITR would be for "Sh*t, we messed up some data can you restore the data as of X?" type scenarios.

          Show
          Lars Hofhansl added a comment - Yep, that's what I have in mind. Can do PITR within HBase's time resolution. We have the requirement to keep historical data around for our customers. Replication is for disaster recovery (main cluster died, we will do that too). Backpup with PITR would be for "Sh*t, we messed up some data can you restore the data as of X?" type scenarios.
          Hide
          Jonathan Hsieh added a comment -

          Hm, so this is similar to this http://www.postgresql.org/docs/8.2/static/continuous-archiving.html? For hbase, this would to enable a "consistent" warm backup (though no strong guarantees across regions) that would be cheaper than the full replication mechanism?

          Show
          Jonathan Hsieh added a comment - Hm, so this is similar to this http://www.postgresql.org/docs/8.2/static/continuous-archiving.html? For hbase, this would to enable a "consistent" warm backup (though no strong guarantees across regions) that would be cheaper than the full replication mechanism?
          Hide
          Lars Hofhansl added a comment -

          This is definitely not to replace distributed log splitting as result of a crash but for dealing with accidentally deleted data.

          Relational databases usually support point in time recovery from a backup by taking periodic baseline backups and archiving the WAL. Upon recovery the base backup closest before the PIT is used and then the logs are replayed to the desired to PIT.

          Since HBase has not snapshotting, yet, any backup solution will necessary lead to an inconsistent copy that can only be made consistent by replaying some of the logs (to cover the duration the backup took).

          Log replay in HBase is either slow (standalone client using the highlevel API) or can only be used for crash recovery (log splitting, because the logs are split by region names, wouldn't be able to deal with split regions).

          This would take the part of log replaying for a thje log replay part in a PITR scenario.
          Look at this as an M/R version of HBASE-3752.

          Show
          Lars Hofhansl added a comment - This is definitely not to replace distributed log splitting as result of a crash but for dealing with accidentally deleted data. Relational databases usually support point in time recovery from a backup by taking periodic baseline backups and archiving the WAL. Upon recovery the base backup closest before the PIT is used and then the logs are replayed to the desired to PIT. Since HBase has not snapshotting, yet, any backup solution will necessary lead to an inconsistent copy that can only be made consistent by replaying some of the logs (to cover the duration the backup took). Log replay in HBase is either slow (standalone client using the highlevel API) or can only be used for crash recovery (log splitting, because the logs are split by region names, wouldn't be able to deal with split regions). This would take the part of log replaying for a thje log replay part in a PITR scenario. Look at this as an M/R version of HBASE-3752 .
          Hide
          Jonathan Hsieh added a comment -

          What is the use case for filtering the HLogs? Would you want to do a "partial" recovery?

          I've encountered a situation where an entire large cluster went out and every RS's WAL needed to do log splitting. The nn went down under hbase. Since this was before distributed log splitting it took an overnight to restore. Distributed log splitting would have really helped here (roughly divide by 100) but its not clear if there was enough data to make the bulk load overcome the extra writes required with a MR job.

          I'd guess that distributed log splitting is probably faster – with an MR job you'd potentially need to materialize after map, do a shuffle (needed?), and materialize again after reduce before bulk loading (which may split the generated hfiles) (multiple writes per put/delete). Distributed log splitting, assuming there is no WAL writes on replay may not incur disk cost except for regular memstore flushes (which means single write per put/delete).

          Show
          Jonathan Hsieh added a comment - What is the use case for filtering the HLogs? Would you want to do a "partial" recovery? I've encountered a situation where an entire large cluster went out and every RS's WAL needed to do log splitting. The nn went down under hbase. Since this was before distributed log splitting it took an overnight to restore. Distributed log splitting would have really helped here (roughly divide by 100) but its not clear if there was enough data to make the bulk load overcome the extra writes required with a MR job. I'd guess that distributed log splitting is probably faster – with an MR job you'd potentially need to materialize after map, do a shuffle (needed?), and materialize again after reduce before bulk loading (which may split the generated hfiles) (multiple writes per put/delete). Distributed log splitting, assuming there is no WAL writes on replay may not incur disk cost except for regular memstore flushes (which means single write per put/delete).
          Hide
          Lars Hofhansl added a comment -

          This is a part of HBase I not that familiar with.
          Why is this in principle different from ImportTsv?
          I guess it is because each mapper can encounter WALEdits for many tables in the HLog file(s) that it works on...?
          In the end, though, it would a reducer writing the HFiles, so the distribution of HLogs to mappers should not matter. I think.

          Hmm... Maybe this is only useful when we have a lot of logs to replay such as in a point in time recovery scenario using HLogs.
          Or maybe there would be no advantage here turning this in an M/R job, but maybe it should just be a standalone client...?

          Show
          Lars Hofhansl added a comment - This is a part of HBase I not that familiar with. Why is this in principle different from ImportTsv? I guess it is because each mapper can encounter WALEdits for many tables in the HLog file(s) that it works on...? In the end, though, it would a reducer writing the HFiles, so the distribution of HLogs to mappers should not matter. I think. Hmm... Maybe this is only useful when we have a lot of logs to replay such as in a point in time recovery scenario using HLogs. Or maybe there would be no advantage here turning this in an M/R job, but maybe it should just be a standalone client...?
          Hide
          stack added a comment -

          Oh, hfile has to be sorted too. At log splitting time, would probably end up writing lots of small hfiles... one per WAL split say. Could make for more i/o though might in long run though it could also get region back online faster.

          Show
          stack added a comment - Oh, hfile has to be sorted too. At log splitting time, would probably end up writing lots of small hfiles... one per WAL split say. Could make for more i/o though might in long run though it could also get region back online faster.
          Hide
          Lars Hofhansl added a comment -

          Interesting.
          Does the order matter in this case? Here we'd replay the entire set of WALEdits for the duration specified and then store the collective outcome in HFiles.
          All operations in the HLogs are idempotent, right?

          Show
          Lars Hofhansl added a comment - Interesting. Does the order matter in this case? Here we'd replay the entire set of WALEdits for the duration specified and then store the collective outcome in HFiles. All operations in the HLogs are idempotent, right?
          Hide
          stack added a comment -

          I considered that instead of writing sequence files when we did log splitting, that instead we write hfiles and then load those instead of replaying the edits on region open as we currently do now. I think I balked because we'd no longer respect the order in which the edits arrived; i.e. we'd not apply the edits in sequenceid order.

          Show
          stack added a comment - I considered that instead of writing sequence files when we did log splitting, that instead we write hfiles and then load those instead of replaying the edits on region open as we currently do now. I think I balked because we'd no longer respect the order in which the edits arrived; i.e. we'd not apply the edits in sequenceid order.

            People

            • Assignee:
              Lars Hofhansl
              Reporter:
              Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development