HBase
  1. HBase
  2. HBASE-4618 HBase backups
  3. HBASE-4661

Ability to export the list of files for a some or all column families for a given region

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: documentation, regionserver
    • Labels:
      None

      Description

      Should be able to query the regionservers to figure out the list of files in one/some/all column families for a given regions to determine which files to copy for a backup.

        Issue Links

          Activity

          Show
          Madhuwanti Vaidya added a comment - https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4661 get list of store files.".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 This interface already has a flushRegion that takes an HRegionInterface. Should we deprecate it? Is this regionName the full name or is it encoded Name?
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 Is region the right granularity? We'll have to make a bunch of calls. Should we rather allow instead asking the regionserver return all storefiles for a given table (with optional columnfamily qualifier supplied)?
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2169 Does the flush happen inline or are we just queuing a flush? If former, perhaps return boolean on whether we flushed or no (not important)
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 How do I express region has not been flushed at all? -1?

          REVISION DETAIL
          https://reviews.facebook.net/D729

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4661 get list of store files.". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 This interface already has a flushRegion that takes an HRegionInterface. Should we deprecate it? Is this regionName the full name or is it encoded Name? src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 Is region the right granularity? We'll have to make a bunch of calls. Should we rather allow instead asking the regionserver return all storefiles for a given table (with optional columnfamily qualifier supplied)? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2169 Does the flush happen inline or are we just queuing a flush? If former, perhaps return boolean on whether we flushed or no (not important) src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 How do I express region has not been flushed at all? -1? REVISION DETAIL https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          madhuvaidya has commented on the revision "[jira] HBASE-4661 get list of store files.".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 In the backup code we used this API (with regionName which is the full name). Not sure - maybe we could deprecate the other API.
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 We used it at this granularity since we scheduled one map task per region for taking a backup of the table.
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2169 The flush happens inline. (It's a blocking call)
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 The lastFlushTime is set when the region is initialized which is equivalent to a flush for our purpose. In each map task - we flush a region before asking for the list of store files to backup. We want to protect against the case when a flush goes through, but the map task fails, in which case a flush would be scheduled again if we didn't check lastFlushTime > backupStartTime.

          REVISION DETAIL
          https://reviews.facebook.net/D729

          Show
          Phabricator added a comment - madhuvaidya has commented on the revision " [jira] HBASE-4661 get list of store files.". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 In the backup code we used this API (with regionName which is the full name). Not sure - maybe we could deprecate the other API. src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 We used it at this granularity since we scheduled one map task per region for taking a backup of the table. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2169 The flush happens inline. (It's a blocking call) src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 The lastFlushTime is set when the region is initialized which is equivalent to a flush for our purpose. In each map task - we flush a region before asking for the list of store files to backup. We want to protect against the case when a flush goes through, but the map task fails, in which case a flush would be scheduled again if we didn't check lastFlushTime > backupStartTime. REVISION DETAIL https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4661 get list of store files.".

          +1 Would suggest adding deprecate to old flushRegion on commit.

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 Yeah, I'd say deprecate the other. Its kinda a pain anyways taking a whole HRI.
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 Ok. Sounds good.
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 Ok. Sounds good.

          REVISION DETAIL
          https://reviews.facebook.net/D729

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4661 get list of store files.". +1 Would suggest adding deprecate to old flushRegion on commit. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:88 Yeah, I'd say deprecate the other. Its kinda a pain anyways taking a whole HRI. src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java:118 Ok. Sounds good. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:2189 Ok. Sounds good. REVISION DETAIL https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          madhuvaidya updated the revision "[jira] HBASE-4661 get list of store files.".
          Reviewers: Karthik, Kannan, nspiegelberg, JIRA

          Added deprecated to old flushRegion.

          REVISION DETAIL
          https://reviews.facebook.net/D729

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java

          Show
          Phabricator added a comment - madhuvaidya updated the revision " [jira] HBASE-4661 get list of store files.". Reviewers: Karthik, Kannan, nspiegelberg, JIRA Added deprecated to old flushRegion. REVISION DETAIL https://reviews.facebook.net/D729 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4661 get list of store files.".

          +1

          If none of you lads are going to commit it, stick it up on JIRA and I will... good stuff.

          REVISION DETAIL
          https://reviews.facebook.net/D729

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4661 get list of store files.". +1 If none of you lads are going to commit it, stick it up on JIRA and I will... good stuff. REVISION DETAIL https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          Karthik has commented on the revision "[jira] HBASE-4661 get list of store files.".

          I can commit - left it up accepted and uncommitted in case some else wanted to have a look. Now I feel a lot better committing

          REVISION DETAIL
          https://reviews.facebook.net/D729

          Show
          Phabricator added a comment - Karthik has commented on the revision " [jira] HBASE-4661 get list of store files.". I can commit - left it up accepted and uncommitted in case some else wanted to have a look. Now I feel a lot better committing REVISION DETAIL https://reviews.facebook.net/D729
          Hide
          Phabricator added a comment -

          madhuvaidya has committed the revision "[jira] HBASE-4661 get list of store files.".

          REVISION DETAIL
          https://reviews.facebook.net/D729

          COMMIT
          https://reviews.facebook.net/rHBASE1213487

          Show
          Phabricator added a comment - madhuvaidya has committed the revision " [jira] HBASE-4661 get list of store files.". REVISION DETAIL https://reviews.facebook.net/D729 COMMIT https://reviews.facebook.net/rHBASE1213487
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #30 (See https://builds.apache.org/job/HBase-TRUNK-security/30/)
          [jira] HBASE-4661 get list of store files.

          Summary:
          1. Ported getStoreFileList API to trunk.
          2. Also ported flushRegion API to trunk (with option of flushing if
          lastFlushTime < TS).

          Test Plan: Tested on 89.

          Reviewers: Karthik, Kannan, nspiegelberg, JIRA

          Reviewed By: Karthik

          CC: HBase Diffs Facebook Group, Karthik, Kannan, stack, madhuvaidya

          Differential Revision: 729

          karthik :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #30 (See https://builds.apache.org/job/HBase-TRUNK-security/30/ ) [jira] HBASE-4661 get list of store files. Summary: 1. Ported getStoreFileList API to trunk. 2. Also ported flushRegion API to trunk (with option of flushing if lastFlushTime < TS). Test Plan: Tested on 89. Reviewers: Karthik, Kannan, nspiegelberg, JIRA Reviewed By: Karthik CC: HBase Diffs Facebook Group, Karthik, Kannan, stack, madhuvaidya Differential Revision: 729 karthik : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2541 (See https://builds.apache.org/job/HBase-TRUNK/2541/)
          [jira] HBASE-4661 get list of store files.

          Summary:
          1. Ported getStoreFileList API to trunk.
          2. Also ported flushRegion API to trunk (with option of flushing if
          lastFlushTime < TS).

          Test Plan: Tested on 89.

          Reviewers: Karthik, Kannan, nspiegelberg, JIRA

          Reviewed By: Karthik

          CC: HBase Diffs Facebook Group, Karthik, Kannan, stack, madhuvaidya

          Differential Revision: 729

          karthik :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2541 (See https://builds.apache.org/job/HBase-TRUNK/2541/ ) [jira] HBASE-4661 get list of store files. Summary: 1. Ported getStoreFileList API to trunk. 2. Also ported flushRegion API to trunk (with option of flushing if lastFlushTime < TS). Test Plan: Tested on 89. Reviewers: Karthik, Kannan, nspiegelberg, JIRA Reviewed By: Karthik CC: HBase Diffs Facebook Group, Karthik, Kannan, stack, madhuvaidya Differential Revision: 729 karthik : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java

            People

            • Assignee:
              Madhuwanti Vaidya
              Reporter:
              Karthik Ranganathan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development