Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Tool to replay WAL files using a M/R job.

      The WAL can be replayed for a set of tables or all tables, and a timerange can be provided (in milliseconds).
      The WAL is filtered to this set of tables, the output can optionally be mapped to another set of tables.

      WAL replay can also generate HFiles for later bulk importing, in that case the WAL is replayed for a single table only.
      Show
      Tool to replay WAL files using a M/R job. The WAL can be replayed for a set of tables or all tables, and a timerange can be provided (in milliseconds). The WAL is filtered to this set of tables, the output can optionally be mapped to another set of tables. WAL replay can also generate HFiles for later bulk importing, in that case the WAL is replayed for a single table only.

      Description

      Just an idea I had. Might be useful for restore of a backup using the HLogs.
      This could an M/R (with a mapper per HLog file).

      The tool would get a timerange and a (set of) table(s). We'd pick the right HLogs based on time before the M/R job is started and then have a mapper per HLog file.
      The mapper would then go through the HLog, filter all WALEdits that didn't fit into the time range or are not any of the tables and then uses HFileOutputFormat to generate HFiles.
      Would need to indicate the splits we want, probably from a live table.

      1. HLog-5604-v3.txt
        16 kB
        Lars Hofhansl
      2. 5604-v4.txt
        24 kB
        Lars Hofhansl
      3. 5604-v6.txt
        29 kB
        Lars Hofhansl
      4. 5604-v7.txt
        29 kB
        Lars Hofhansl
      5. 5604-v8.txt
        34 kB
        Lars Hofhansl
      6. 5604-v9.txt
        36 kB
        Lars Hofhansl
      7. 5604-v10.txt
        36 kB
        Lars Hofhansl
      8. 5604-v11.txt
        37 kB
        Lars Hofhansl

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Lars Hofhansl
              Reporter:
              Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development