HBase
  1. HBase
  2. HBASE-1470

hbase and HADOOP-4379, dhruba's flush/sync

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None

      Description

      This covers work with HADOOP-4379

      1. recovery3.patch
        5 kB
        stack
      2. 1470.patch
        7 kB
        stack
      3. 1470-v2.patch
        6 kB
        stack

        Activity

        Hide
        stack added a comment -

        Patch that will do the open-for-append if the hadoop append configuration is set to true in a place where hbase client can see it.

        Show
        stack added a comment - Patch that will do the open-for-append if the hadoop append configuration is set to true in a place where hbase client can see it.
        Hide
        Jonathan Gray added a comment -

        Need to figure something for 0.20.0 release. Since we rely on a patched hadoop server, either must have a separate hbase, a patch for hbase and hadoop, or if we can (via reflection or such) detect if the hadoop server supports appends we could commit this and only maintain one thing.

        Show
        Jonathan Gray added a comment - Need to figure something for 0.20.0 release. Since we rely on a patched hadoop server, either must have a separate hbase, a patch for hbase and hadoop, or if we can (via reflection or such) detect if the hadoop server supports appends we could commit this and only maintain one thing.
        Hide
        stack added a comment -

        Patch that uses reflection to see if syncfs is available and if it is, uses this syncing.

        Show
        stack added a comment - Patch that uses reflection to see if syncfs is available and if it is, uses this syncing.
        Hide
        stack added a comment -

        If dfs.support.append is set to true AND you have put an hadoop with hadoop-4379 into hbase CLASSPATH, this patch will make it so sync use the hadoop-4379 new syncFs call.

        This is a bit tough to test. You have to do messing with different hadop jars. I played around a bunch with it and it seems to do the right thing. I'll just commit since it satisfies a probably exotic need (Last time I was running 4379, namenode shutdown because of a bad generation stamp. I don't think this a 4379 issue. Its probably something that can happen if you ahve dfs.support.append=true, a bug).

        Show
        stack added a comment - If dfs.support.append is set to true AND you have put an hadoop with hadoop-4379 into hbase CLASSPATH, this patch will make it so sync use the hadoop-4379 new syncFs call. This is a bit tough to test. You have to do messing with different hadop jars. I played around a bunch with it and it seems to do the right thing. I'll just commit since it satisfies a probably exotic need (Last time I was running 4379, namenode shutdown because of a bad generation stamp. I don't think this a 4379 issue. Its probably something that can happen if you ahve dfs.support.append=true, a bug).
        Hide
        dhruba borthakur added a comment -

        > Last time I was running 4379, namenode shutdown because of a bad generation stamp

        I saw the same bug in our cluster. Have posted a fix as part of the new patch file attached to HDFS-200. Attachment fsyncConcurrentReaders13_20.txt

        Show
        dhruba borthakur added a comment - > Last time I was running 4379, namenode shutdown because of a bad generation stamp I saw the same bug in our cluster. Have posted a fix as part of the new patch file attached to HDFS-200 . Attachment fsyncConcurrentReaders13_20.txt
        Hide
        stack added a comment -

        @dhurba Thanks. Was going to start in testing 4379 again today. Good stuff.

        Show
        stack added a comment - @dhurba Thanks. Was going to start in testing 4379 again today. Good stuff.

          People

          • Assignee:
            Unassigned
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development