HBase
  1. HBase
  2. HBASE-1470

hbase and HADOOP-4379, dhruba's flush/sync

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None

      Description

      This covers work with HADOOP-4379

      1. recovery3.patch
        5 kB
        stack
      2. 1470-v2.patch
        6 kB
        stack
      3. 1470.patch
        7 kB
        stack

        Activity

        stack created issue -
        Hide
        stack added a comment -

        Patch that will do the open-for-append if the hadoop append configuration is set to true in a place where hbase client can see it.

        Show
        stack added a comment - Patch that will do the open-for-append if the hadoop append configuration is set to true in a place where hbase client can see it.
        stack made changes -
        Field Original Value New Value
        Attachment recovery3.patch [ 12409572 ]
        Hide
        Jonathan Gray added a comment -

        Need to figure something for 0.20.0 release. Since we rely on a patched hadoop server, either must have a separate hbase, a patch for hbase and hadoop, or if we can (via reflection or such) detect if the hadoop server supports appends we could commit this and only maintain one thing.

        Show
        Jonathan Gray added a comment - Need to figure something for 0.20.0 release. Since we rely on a patched hadoop server, either must have a separate hbase, a patch for hbase and hadoop, or if we can (via reflection or such) detect if the hadoop server supports appends we could commit this and only maintain one thing.
        Jonathan Gray made changes -
        Fix Version/s 0.20.0 [ 12313474 ]
        Hide
        stack added a comment -

        Patch that uses reflection to see if syncfs is available and if it is, uses this syncing.

        Show
        stack added a comment - Patch that uses reflection to see if syncfs is available and if it is, uses this syncing.
        stack made changes -
        Attachment 1470.patch [ 12413155 ]
        Hide
        stack added a comment -

        If dfs.support.append is set to true AND you have put an hadoop with hadoop-4379 into hbase CLASSPATH, this patch will make it so sync use the hadoop-4379 new syncFs call.

        This is a bit tough to test. You have to do messing with different hadop jars. I played around a bunch with it and it seems to do the right thing. I'll just commit since it satisfies a probably exotic need (Last time I was running 4379, namenode shutdown because of a bad generation stamp. I don't think this a 4379 issue. Its probably something that can happen if you ahve dfs.support.append=true, a bug).

        Show
        stack added a comment - If dfs.support.append is set to true AND you have put an hadoop with hadoop-4379 into hbase CLASSPATH, this patch will make it so sync use the hadoop-4379 new syncFs call. This is a bit tough to test. You have to do messing with different hadop jars. I played around a bunch with it and it seems to do the right thing. I'll just commit since it satisfies a probably exotic need (Last time I was running 4379, namenode shutdown because of a bad generation stamp. I don't think this a 4379 issue. Its probably something that can happen if you ahve dfs.support.append=true, a bug).
        stack made changes -
        Attachment 1470-v2.patch [ 12413170 ]
        stack made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        dhruba borthakur added a comment -

        > Last time I was running 4379, namenode shutdown because of a bad generation stamp

        I saw the same bug in our cluster. Have posted a fix as part of the new patch file attached to HDFS-200. Attachment fsyncConcurrentReaders13_20.txt

        Show
        dhruba borthakur added a comment - > Last time I was running 4379, namenode shutdown because of a bad generation stamp I saw the same bug in our cluster. Have posted a fix as part of the new patch file attached to HDFS-200 . Attachment fsyncConcurrentReaders13_20.txt
        Hide
        stack added a comment -

        @dhurba Thanks. Was going to start in testing 4379 again today. Good stuff.

        Show
        stack added a comment - @dhurba Thanks. Was going to start in testing 4379 again today. Good stuff.
        stack made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        39d 5h 57m 1 stack 11/Jul/09 00:18
        Resolved Resolved Closed Closed
        64d 23h 6m 1 stack 13/Sep/09 23:24

          People

          • Assignee:
            Unassigned
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development