Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3812

Use HBase snapshots in async index building M/R job

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.10.0
    • Fix Version/s: 4.11.0
    • Labels:
      None

      Description

      As per discussion with James, HBase snapshots makes it lot easier and faster to operate on existing data.

      So explore using HBase snapshots in index building M/R job for async index.

      1. PHOENIX-3812.patch
        12 kB
        Akshita Malhotra
      2. PHOENIX-3812-4.x-0.98.patch
        12 kB
        Akshita Malhotra

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user akshita-malhotra opened a pull request:

          https://github.com/apache/phoenix/pull/260

          PHOENIX-3812: Use HBase snapshots in async index building M/R job

          • Index tool creates a snapshot and uses it as a configuration parameter to run index M/R job using HBase snapshot.
          • Add option to configure use of snapshots in IndexTool

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/akshita-malhotra/phoenix PHOENIX-3812

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/phoenix/pull/260.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #260


          commit 57fb264ba76e849db3bc4375f87091499cbce618
          Author: Akshita <akshita.malhotra@salesforce.com>
          Date: 2017-06-07T23:14:47Z

          PHOENIX-3812: Use HBase snapshots in async index building M/R job


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user akshita-malhotra opened a pull request: https://github.com/apache/phoenix/pull/260 PHOENIX-3812 : Use HBase snapshots in async index building M/R job Index tool creates a snapshot and uses it as a configuration parameter to run index M/R job using HBase snapshot. Add option to configure use of snapshots in IndexTool You can merge this pull request into a Git repository by running: $ git pull https://github.com/akshita-malhotra/phoenix PHOENIX-3812 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/260.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #260 commit 57fb264ba76e849db3bc4375f87091499cbce618 Author: Akshita <akshita.malhotra@salesforce.com> Date: 2017-06-07T23:14:47Z PHOENIX-3812 : Use HBase snapshots in async index building M/R job
          Hide
          jamestaylor James Taylor added a comment -

          Patch looks good, but make sure to close admin in configureJobForAysncIndex method, Akshita Malhotra.

          Show
          jamestaylor James Taylor added a comment - Patch looks good, but make sure to close admin in configureJobForAysncIndex method, Akshita Malhotra .
          Hide
          akshita.malhotra Akshita Malhotra added a comment - - edited

          James Taylor Thanks for the comment.
          I have updated and uploaded two patches:
          PHOENIX-3812.patch applies cleanly to master and 1.1 branch.
          PHOENIX-3812-4.x-0.98.patch is for 4.x-0.98 branch.

          Show
          akshita.malhotra Akshita Malhotra added a comment - - edited James Taylor Thanks for the comment. I have updated and uploaded two patches: PHOENIX-3812 .patch applies cleanly to master and 1.1 branch. PHOENIX-3812 -4.x-0.98.patch is for 4.x-0.98 branch.
          Hide
          jamestaylor James Taylor added a comment -

          +1. Thanks, Akshita Malhotra!

          Show
          jamestaylor James Taylor added a comment - +1. Thanks, Akshita Malhotra !
          Hide
          jamestaylor James Taylor added a comment -

          Pushed to master and 4.x branches. Thanks for the contribution, Akshita Malhotra.

          Show
          jamestaylor James Taylor added a comment - Pushed to master and 4.x branches. Thanks for the contribution, Akshita Malhotra .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Phoenix-master #1657 (See https://builds.apache.org/job/Phoenix-master/1657/)
          PHOENIX-3812 Use HBase snapshots in async index building M/R job (jamestaylor: rev 59f1a2839480cdc77ffb0c6742986ffff3b91bba)

          • (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexTool.java
          • (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java
          • (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexExtendedIT.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Phoenix-master #1657 (See https://builds.apache.org/job/Phoenix-master/1657/ ) PHOENIX-3812 Use HBase snapshots in async index building M/R job (jamestaylor: rev 59f1a2839480cdc77ffb0c6742986ffff3b91bba) (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexTool.java (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexExtendedIT.java

            People

            • Assignee:
              akshita.malhotra Akshita Malhotra
              Reporter:
              sukunaidu@gmail.com Maddineni Sukumar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development