HBase
  1. HBase
  2. HBASE-2340

Add end-to-end test of sync/flush

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add a test to do the following:

      + Start a HBase/HDFS cluster (local node is fine).
       + Use top-level (HTable) level APIs to put items. 
      + Try about single column puts, as well as puts which span multiple columns/multiple column families, etc.
      + Then kill one region server.
      + Wait for recovery to happen.
      + And then check the rows exist.
      

      Assigning myself.

      1. 2340-v2.patch
        6 kB
        stack
      2. 2340.patch
        44 kB
        stack

        Activity

        Hide
        stack added a comment -

        Marking these as fixed against 0.21.0 rather than against 0.20.5.

        Show
        stack added a comment - Marking these as fixed against 0.21.0 rather than against 0.20.5.
        Hide
        stack added a comment -

        Committed to branch without flush (Seems to work w/o it). Thanks for review Kannan.

        Show
        stack added a comment - Committed to branch without flush (Seems to work w/o it). Thanks for review Kannan.
        Hide
        Kannan Muthukkaruppan added a comment -

        Typo in my previous post: <<<I'll add specific cases for Puts containing multiple edits, Deletes, etc. (related to HBASE-2383 & HBASE-2338).>>>

        The relevant JIRAs should be HBASE-2283 & HBASE-2338.

        Show
        Kannan Muthukkaruppan added a comment - Typo in my previous post: <<<I'll add specific cases for Puts containing multiple edits, Deletes, etc. (related to HBASE-2383 & HBASE-2338 ).>>> The relevant JIRAs should be HBASE-2283 & HBASE-2338 .
        Hide
        Jean-Daniel Cryans added a comment -

        That test is imperfect, I don't remember exactly why I put a flush there but I remember I had some issues without it.

        Show
        Jean-Daniel Cryans added a comment - That test is imperfect, I don't remember exactly why I put a flush there but I remember I had some issues without it.
        Hide
        Kannan Muthukkaruppan added a comment -

        Curious why the "flush" in the middle of the test-- coz flushing the row and loading the same rows again could mask problems with recovery. So perhaps we should remove the flush?

            for(int i = 0; i < 4; i++) {
              TEST_UTIL.loadTable(table, FAMILY);
              if(i == 2) {
                TEST_UTIL.flush();
              }
            }
        

        Otherwise, the test looks good for testing log reconstruction. Perhaps you can commit this, and then I'll add specific cases for Puts containing multiple edits, Deletes, etc. (related to HBASE-2383 & HBASE-2338).

        We should also enhance the test or add tests to test datanode failures-- but please commit this for now.

        Show
        Kannan Muthukkaruppan added a comment - Curious why the "flush" in the middle of the test-- coz flushing the row and loading the same rows again could mask problems with recovery. So perhaps we should remove the flush? for ( int i = 0; i < 4; i++) { TEST_UTIL.loadTable(table, FAMILY); if (i == 2) { TEST_UTIL.flush(); } } Otherwise, the test looks good for testing log reconstruction. Perhaps you can commit this, and then I'll add specific cases for Puts containing multiple edits, Deletes, etc. (related to HBASE-2383 & HBASE-2338 ). We should also enhance the test or add tests to test datanode failures-- but please commit this for now.
        Hide
        Jean-Daniel Cryans added a comment -

        Working on replication, I wrote a test that does something like TestFullLogReconstruction except that instead of waiting for the full insert to finish, I start a thread an kill a RS after some seconds. It passes half of the time because of what people described in HBASE-2231 e.g. a region server is able to append and even roll logs! I think TestFullLogReconstruction should also do something like that.

        Show
        Jean-Daniel Cryans added a comment - Working on replication, I wrote a test that does something like TestFullLogReconstruction except that instead of waiting for the full insert to finish, I start a thread an kill a RS after some seconds. It passes half of the time because of what people described in HBASE-2231 e.g. a region server is able to append and even roll logs! I think TestFullLogReconstruction should also do something like that.
        Hide
        Kannan Muthukkaruppan added a comment -

        Thanks Stack! I will take a look at this.

        Show
        Kannan Muthukkaruppan added a comment - Thanks Stack! I will take a look at this.
        Hide
        stack added a comment -

        Assigning Kannan so he'll take a look at posted patch (Thanks K).

        Show
        stack added a comment - Assigning Kannan so he'll take a look at posted patch (Thanks K).
        Hide
        stack added a comment -

        @Kannan Please take a look. What else do you need it to do? Would you like to be able to kill datanodes too? Let me know.

        Show
        stack added a comment - @Kannan Please take a look. What else do you need it to do? Would you like to be able to kill datanodes too? Let me know.
        Hide
        stack added a comment -

        I attached the wrong file previously.

        Show
        stack added a comment - I attached the wrong file previously.
        Hide
        stack added a comment -

        TestFullLogReconstruction works now after enabling dfs.append.support=true (and other configs. from Nicolas hbase-2345) and making it so exiting regionserver doesn't shutdown the filesystem (because then the filesystem is closed for daemons in the JVM). The test runs for 200 seconds, mostly because we're loading up a good bit of data. Could make it less I suppose.

        I see this in test log:

        2010-03-22 23:10:33,232 INFO  [HMaster] regionserver.HLog(1096): Splitting 1 hlog(s) in hdfs://localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078
        2010-03-22 23:10:33,232 DEBUG [HMaster] regionserver.HLog(1183): Splitting hlog 1 of 1: hdfs://localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191, length=0
        2010-03-22 23:10:33,233 WARN  [IPC Server handler 8 on 61468] namenode.FSNamesystem(1144): DIR* NameSystem.startFile: failed to create file /user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191 for DFSClient_343812212 on client 127.0.0.1 because current leaseholder is trying to recreate file.
        2010-03-22 23:10:33,240 INFO  [HMaster] regionserver.HLog(1427): Failed open for append, waiting on lease recovery: hdfs://localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191
        org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191 for DFSClient_343812212 on client 127.0.0.1 because current leaseholder is trying to recreate file.
        ...
        

        ... and after RS fully goes down we're able to open the log and process its content.

        Show
        stack added a comment - TestFullLogReconstruction works now after enabling dfs.append.support=true (and other configs. from Nicolas hbase-2345) and making it so exiting regionserver doesn't shutdown the filesystem (because then the filesystem is closed for daemons in the JVM). The test runs for 200 seconds, mostly because we're loading up a good bit of data. Could make it less I suppose. I see this in test log: 2010-03-22 23:10:33,232 INFO [HMaster] regionserver.HLog(1096): Splitting 1 hlog(s) in hdfs: //localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078 2010-03-22 23:10:33,232 DEBUG [HMaster] regionserver.HLog(1183): Splitting hlog 1 of 1: hdfs: //localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191, length=0 2010-03-22 23:10:33,233 WARN [IPC Server handler 8 on 61468] namenode.FSNamesystem(1144): DIR* NameSystem.startFile: failed to create file /user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191 for DFSClient_343812212 on client 127.0.0.1 because current leaseholder is trying to recreate file. 2010-03-22 23:10:33,240 INFO [HMaster] regionserver.HLog(1427): Failed open for append, waiting on lease recovery: hdfs: //localhost:61468/user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191 org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/stack/.logs/192.168.1.157,61497,1269324534078/hlog.dat.1269324534191 for DFSClient_343812212 on client 127.0.0.1 because current leaseholder is trying to recreate file. ... ... and after RS fully goes down we're able to open the log and process its content.
        Hide
        Jean-Daniel Cryans added a comment -

        I don't see that test enabling dfs.append.enabled and IIRC it's required in 0.20 for a working fsSync?

        Show
        Jean-Daniel Cryans added a comment - I don't see that test enabling dfs.append.enabled and IIRC it's required in 0.20 for a working fsSync?
        Hide
        stack added a comment -

        Here is patch that adds TestFullLogReconstruction.java. It currently fails. Looking to why.

        Testcase: testReconstruction took 275.704 sec
                Caused an ERROR
        Trying to contact region server 192.168.1.157:55472 for region tabletest,,1269121970753, row '', but failed after 10 attempts.
        Exceptions:
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        
        org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.1.157:55472 for region tabletest,,1269121970753, row '', but failed after 10 attempts.
        Exceptions:
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        java.net.ConnectException: Connection refused
        
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1073)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1935)
                at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1855)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:376)
                at org.apache.hadoop.hbase.TestFullLogReconstruction.testReconstruction(TestFullLogReconstruction.java:115)
        
        Show
        stack added a comment - Here is patch that adds TestFullLogReconstruction.java. It currently fails. Looking to why. Testcase: testReconstruction took 275.704 sec Caused an ERROR Trying to contact region server 192.168.1.157:55472 for region tabletest,,1269121970753, row '', but failed after 10 attempts. Exceptions: java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 192.168.1.157:55472 for region tabletest,,1269121970753, row '', but failed after 10 attempts. Exceptions: java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1073) at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1935) at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1855) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:376) at org.apache.hadoop.hbase.TestFullLogReconstruction.testReconstruction(TestFullLogReconstruction.java:115)
        Hide
        stack added a comment -

        I'm backporting J-D's test for a start. Required me to backport HBaseTestingUtility from trunk and adding a junit jar that supports junit4. Working on making TestFullLogReconstruction work on branch now.

        Once this is working, we can discuss what else we'd like to add.

        Show
        stack added a comment - I'm backporting J-D's test for a start. Required me to backport HBaseTestingUtility from trunk and adding a junit jar that supports junit4. Working on making TestFullLogReconstruction work on branch now. Once this is working, we can discuss what else we'd like to add.
        Hide
        stack added a comment -

        J-D points out that TRUNK has a basic test of failed regionserver verifying edits are in place: TestFullLogReconstruction. Let this issue then be about backporting that test and adding in support for the suggestions made above.

        Show
        stack added a comment - J-D points out that TRUNK has a basic test of failed regionserver verifying edits are in place: TestFullLogReconstruction. Let this issue then be about backporting that test and adding in support for the suggestions made above.
        Hide
        Todd Lipcon added a comment -

        Yea, what I'd love to see eventually is a "rambo" script that will do any number of nasty things periodically to nodes in the cluster. Including but not limited to:

        • kill -9 a {RS,Master,DN,ZKNode}
          - kill -INT a {RS,Master,DN,ZKNode}
        • kill -STOP a {RS,Master,DN,ZKNode}

          , followed by -CONT a few minutes later

        • ifconfig eth0 down ; sleep 10; ifconfig eth0 up
        • insert bad routes to {ZK,NN,HM}

          for short periods of time (to simulate situation where you can't talk to DFS but you can still talk to ZK, etc)

        this script could be done rather generally and would be useful for testing Hadoop by itself as well.

        Show
        Todd Lipcon added a comment - Yea, what I'd love to see eventually is a "rambo" script that will do any number of nasty things periodically to nodes in the cluster. Including but not limited to: kill -9 a {RS,Master,DN,ZKNode} - kill -INT a {RS,Master,DN,ZKNode} kill -STOP a {RS,Master,DN,ZKNode} , followed by -CONT a few minutes later ifconfig eth0 down ; sleep 10; ifconfig eth0 up insert bad routes to {ZK,NN,HM} for short periods of time (to simulate situation where you can't talk to DFS but you can still talk to ZK, etc) this script could be done rather generally and would be useful for testing Hadoop by itself as well.
        Hide
        Jonathan Gray added a comment -

        Excellent jira. Another interesting thing to test would be what happens if there's a master failover at the same time (or at some random time in between failure and finishing recovery). Seems there would definitely be some holes found there.

        At the least we might test the situation where the failed server is hosting both the regionserver and active master and there is a backup master waiting which is more likely than two distinct nodes crashing that close to each other.

        Show
        Jonathan Gray added a comment - Excellent jira. Another interesting thing to test would be what happens if there's a master failover at the same time (or at some random time in between failure and finishing recovery). Seems there would definitely be some holes found there. At the least we might test the situation where the failed server is hosting both the regionserver and active master and there is a backup master waiting which is more likely than two distinct nodes crashing that close to each other.
        Hide
        Todd Lipcon added a comment -

        A reasonably simple test would also be the following:

        Write a simple client that does something like:

        Random r = new Random(myClientId);
        int i = 0;
        while (true) {
          long toWrite = r.nextLong();
          writeKey(toWrite);
          writeToLocalDiskAndFsync(i);
        }
        

        then run N of these clients concurrently while injecting various bits of churn into the cluster (kill -9, kill -STOP, pull network jacks, etc).

        Success criterion 1 is that all of the clients proceed without throwing exceptions. Success criterion 2 is to start the clients again in verification mode and run the same number of iterations as recorded on the local disk, checking that all of the data is there.

        Show
        Todd Lipcon added a comment - A reasonably simple test would also be the following: Write a simple client that does something like: Random r = new Random(myClientId); int i = 0; while ( true ) { long toWrite = r.nextLong(); writeKey(toWrite); writeToLocalDiskAndFsync(i); } then run N of these clients concurrently while injecting various bits of churn into the cluster (kill -9, kill -STOP, pull network jacks, etc). Success criterion 1 is that all of the clients proceed without throwing exceptions. Success criterion 2 is to start the clients again in verification mode and run the same number of iterations as recorded on the local disk, checking that all of the data is there.

          People

          • Assignee:
            stack
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development