Accumulo
  1. Accumulo
  2. ACCUMULO-2694

Offline tables block balancing for online tables

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0, 1.6.0
    • Fix Version/s: 1.5.2, 1.6.1, 1.7.0
    • Component/s: master
    • Labels:
    • Environment:

      1.6.0-RC2 Started CI with a 10-tablet pre-split table.

      Description

      Both DefaultLoadBalancer and ChaoticLoadBalancer won't balance if there are outstanding migrations.

      In this instance, we have offline tables from previous CI runs. one of these tables had outstanding migrations

      2014-04-17 09:47:34,716 [balancer.TabletBalancer] DEBUG: Scanning tablet server a2438.halxg.cloudera.com:10011[544d5edf1fec529] for table 8
      2014-04-17 09:47:36,217 [balancer.TabletBalancer] DEBUG: Scanning tablet server a2416.halxg.cloudera.com:10011[244d5edf0b0c4ff] for table 8
      2014-04-17 09:47:36,222 [balancer.DefaultLoadBalancer] DEBUG: balance ended with 4 migrations
      2014-04-17 09:47:36,222 [balancer.DefaultLoadBalancer] DEBUG: balance ended with 0 migrations
      2014-04-17 09:47:36,222 [master.Master] DEBUG: migration 8;4aa09c;4a809d: a2438.halxg.cloudera.com:10011[544d5edf1fec529] -> a2422.halxg.cloudera.com:10011[3451dd2d9fa6761]
      2014-04-17 09:47:36,222 [master.Master] DEBUG: migration 8;7e603;7e4029: a2438.halxg.cloudera.com:10011[544d5edf1fec529] -> a2422.halxg.cloudera.com:10011[3451dd2d9fa6761]
      2014-04-17 09:47:36,222 [master.Master] DEBUG: migration 8;21a044;21803d: a2438.halxg.cloudera.com:10011[544d5edf1fec529] -> a2422.halxg.cloudera.com:10011[3451dd2d9fa6761]
      2014-04-17 09:47:36,223 [master.Master] DEBUG: migration 8;59c02e;59a02b: a2416.halxg.cloudera.com:10011[244d5edf0b0c4ff] -> a2414.halxg.cloudera.com:10011[444d5f6b43ac4aa]
      

      Later messages show these tablets being unloaded successfully. However, since the table is offline they never get loaded on the new tablet server. This means they never leave the queue, so balancing stops.

      As an added complication, this last set of migrations was added after the table was already offline. I think this is because there had been unhosted tablets which caused a bunch of contention around when balancing would finally happen.

      A few needed changes:

      1. If the balancer isn't going to balance it needs a log message saying so. Ideally, this message should also include information about the outstanding migrations that are blocking it.
      2. the Migration cleanup thread should look for migrations involving offline tables and clear them (I'd prefer this to trying to have the balancer figure out if a table is offline or online)
      3. When we offline a table, we should probably clear migrations related to that table. This isn't strictly necessary if the cleanup thread will get them eventually, but it would speed things up.

      Workarounds:

      1. migration state is only stored in Master memory, failing over to a different master will force recalculation which will not include offline tables.
      2. if for some reason you can't handle a failure of the current master, bringing the involved table back online (which might mean all offline tables) will allow migrations to resume. the table must remain online until there are no longer migrations involving it.
      3. I think that if you clone the offline table and then delete the original, that will clear the outstanding migrations related to it. I did not test this, because the above two options are much better.

      The latter option will cause considerably more churn, especially if the offline table isn't actually providing utility.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Josh Elser added a comment -

          Just pushed a fix for 1.6.1 and 1.7.0 which appear to have solved the issue for me.

          Show
          Josh Elser added a comment - Just pushed a fix for 1.6.1 and 1.7.0 which appear to have solved the issue for me.
          Hide
          Josh Elser added a comment -

          Maybe I used a non-blocking table creation call?

          Also, I should have remembered this: no, you used the right call. Table operations (specifically create) are always blocking and do not have an option to return fast.

          Show
          Josh Elser added a comment - Maybe I used a non-blocking table creation call? Also, I should have remembered this: no, you used the right call. Table operations (specifically create ) are always blocking and do not have an option to return fast.
          Hide
          Josh Elser added a comment -

          Looking at TabletServer#getStats(Map), the tableMap that is returned by each TabletServer (which is what's ultimately returned in MMI, is constructed from the set of online tablets. Thus, my hunch is that if the tablet didn't get assigned yet, the test would fail. I think we could easily solve this by just waiting for a scanner to read the newly created table's data (despite it being empty). I'll write this up quick and see if it solves the issue for me.

          Show
          Josh Elser added a comment - Looking at TabletServer#getStats(Map) , the tableMap that is returned by each TabletServer (which is what's ultimately returned in MMI, is constructed from the set of online tablets. Thus, my hunch is that if the tablet didn't get assigned yet, the test would fail. I think we could easily solve this by just waiting for a scanner to read the newly created table's data (despite it being empty). I'll write this up quick and see if it solves the issue for me.
          Hide
          Sean Busbey added a comment -

          At least for now, the 1.6 job has its logs from the failure. Unfortunately, they appear to be useless because they're at INFO.

          Show
          Sean Busbey added a comment - At least for now, the 1.6 job has its logs from the failure. Unfortunately, they appear to be useless because they're at INFO.
          Hide
          Josh Elser added a comment -

          Maybe it's a timing thing? Maybe I used a non-blocking table creation call?

          That may be the case. Last night it ended up passing for me. Let me see if I can re-run it a few times and get a failure with all of the logs (didn't pull them off the system when it failed).

          Show
          Josh Elser added a comment - Maybe it's a timing thing? Maybe I used a non-blocking table creation call? That may be the case. Last night it ended up passing for me. Let me see if I can re-run it a few times and get a failure with all of the logs (didn't pull them off the system when it failed).
          Hide
          Sean Busbey added a comment -

          Passes locally, looks like it unit test passed on apache build (while giving the same WARN), but failed on the apache build for 1.6.

          Maybe it's a timing thing? Maybe I used a non-blocking table creation call?

          Show
          Sean Busbey added a comment - Passes locally, looks like it unit test passed on apache build (while giving the same WARN) , but failed on the apache build for 1.6 . Maybe it's a timing thing? Maybe I used a non-blocking table creation call?
          Hide
          Sean Busbey added a comment -

          that's a test that got added by the changes on the 1.6 branch, and that failure would happen if the MAC hadn't added the test table. The stdout just looks like a WARN with a stacktrace. Can I get at any more of the test failure info?

          Show
          Sean Busbey added a comment - that's a test that got added by the changes on the 1.6 branch, and that failure would happen if the MAC hadn't added the test table. The stdout just looks like a WARN with a stacktrace. Can I get at any more of the test failure info?
          Hide
          Josh Elser added a comment -

          Re-opening to triage a test failure.

          Show
          Josh Elser added a comment - Re-opening to triage a test failure.
          Hide
          Josh Elser added a comment -

          I just saw a failure on my jenkins last night for the master branch last night that I've never seen fail before:

          Error Message
          
          our test table should exist in [!0, +r]
          
          Stacktrace
          
          java.lang.AssertionError: our test table should exist in [!0, +r]
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImplTest.saneMonitorInfo(MiniAccumuloClusterImplTest.java:102)
          
          Standard Output
          
          2014-08-03 08:53:34,483 [impl.MiniAccumuloClusterImplTest] INFO : ensure monitor info includes some base information.
          2014-08-03 08:53:34,503 [conf.SiteConfiguration] WARN : accumulo-site.xml not found on classpath
          java.lang.Throwable
          	at org.apache.accumulo.core.conf.SiteConfiguration.getXmlConfig(SiteConfiguration.java:77)
          	at org.apache.accumulo.core.conf.SiteConfiguration.getHadoopConfiguration(SiteConfiguration.java:144)
          	at org.apache.accumulo.core.conf.SiteConfiguration.getProperties(SiteConfiguration.java:121)
          	at org.apache.accumulo.core.conf.AccumuloConfiguration.iterator(AccumuloConfiguration.java:112)
          	at org.apache.accumulo.core.conf.ConfigSanityCheck.validate(ConfigSanityCheck.java:42)
          	at org.apache.accumulo.core.conf.SiteConfiguration.getInstance(SiteConfiguration.java:62)
          	at org.apache.accumulo.core.conf.SiteConfiguration.getInstance(SiteConfiguration.java:68)
          	at org.apache.accumulo.server.security.SystemCredentials$SystemToken.get(SystemCredentials.java:123)
          	at org.apache.accumulo.server.security.SystemCredentials$SystemToken.access$000(SystemCredentials.java:98)
          	at org.apache.accumulo.server.security.SystemCredentials.<init>(SystemCredentials.java:56)
          	at org.apache.accumulo.server.security.SystemCredentials.get(SystemCredentials.java:83)
          	at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImpl.getMasterMonitorInfo(MiniAccumuloClusterImpl.java:764)
          	at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImplTest.saneMonitorInfo(MiniAccumuloClusterImplTest.java:94)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          

          Can you verify that your changes didn't cause this failure?

          Show
          Josh Elser added a comment - I just saw a failure on my jenkins last night for the master branch last night that I've never seen fail before: Error Message our test table should exist in [!0, +r] Stacktrace java.lang.AssertionError: our test table should exist in [!0, +r] at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImplTest.saneMonitorInfo(MiniAccumuloClusterImplTest.java:102) Standard Output 2014-08-03 08:53:34,483 [impl.MiniAccumuloClusterImplTest] INFO : ensure monitor info includes some base information. 2014-08-03 08:53:34,503 [conf.SiteConfiguration] WARN : accumulo-site.xml not found on classpath java.lang.Throwable at org.apache.accumulo.core.conf.SiteConfiguration.getXmlConfig(SiteConfiguration.java:77) at org.apache.accumulo.core.conf.SiteConfiguration.getHadoopConfiguration(SiteConfiguration.java:144) at org.apache.accumulo.core.conf.SiteConfiguration.getProperties(SiteConfiguration.java:121) at org.apache.accumulo.core.conf.AccumuloConfiguration.iterator(AccumuloConfiguration.java:112) at org.apache.accumulo.core.conf.ConfigSanityCheck.validate(ConfigSanityCheck.java:42) at org.apache.accumulo.core.conf.SiteConfiguration.getInstance(SiteConfiguration.java:62) at org.apache.accumulo.core.conf.SiteConfiguration.getInstance(SiteConfiguration.java:68) at org.apache.accumulo.server.security.SystemCredentials$SystemToken.get(SystemCredentials.java:123) at org.apache.accumulo.server.security.SystemCredentials$SystemToken.access$000(SystemCredentials.java:98) at org.apache.accumulo.server.security.SystemCredentials.<init>(SystemCredentials.java:56) at org.apache.accumulo.server.security.SystemCredentials.get(SystemCredentials.java:83) at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImpl.getMasterMonitorInfo(MiniAccumuloClusterImpl.java:764) at org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImplTest.saneMonitorInfo(MiniAccumuloClusterImplTest.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Can you verify that your changes didn't cause this failure?
          Hide
          Josh Elser added a comment -

          Great, thanks, Sean.

          Show
          Josh Elser added a comment - Great, thanks, Sean.
          Hide
          Sean Busbey added a comment -

          yeah, I'll pick up again on trying to figure out why MAC is giving me incorrect table information on 1.6.

          Show
          Sean Busbey added a comment - yeah, I'll pick up again on trying to figure out why MAC is giving me incorrect table information on 1.6.
          Hide
          Josh Elser added a comment -

          This has been sitting idle for about a month now. Can we get this committed to unblock 1.5.2 and 1.6.1?

          Show
          Josh Elser added a comment - This has been sitting idle for about a month now. Can we get this committed to unblock 1.5.2 and 1.6.1?
          Hide
          Mike Drob added a comment -

          Also there are a ton of major compactions queued up on this table, which makes sense, since everything is on a single node.

          Show
          Mike Drob added a comment - Also there are a ton of major compactions queued up on this table, which makes sense, since everything is on a single node.
          Hide
          Mike Drob added a comment -

          Looking at the master logs, the most recent mention of "balance" is from a while ago:

          2014-04-17 13:31:16,047 [master.EventCoordinator] INFO : Table state in zookeeper changed for a to NEW
          2014-04-17 13:31:16,099 [balancer.TableLoadBalancer] INFO : Loaded class org.apache.accumulo.server.master.balancer.DefaultLoadBalancer for table a
          2014-04-17 13:31:16,332 [master.EventCoordinator] INFO : Table state in zookeeper changed for a to ONLINE
          
          Show
          Mike Drob added a comment - Looking at the master logs, the most recent mention of "balance" is from a while ago: 2014-04-17 13:31:16,047 [master.EventCoordinator] INFO : Table state in zookeeper changed for a to NEW 2014-04-17 13:31:16,099 [balancer.TableLoadBalancer] INFO : Loaded class org.apache.accumulo.server.master.balancer.DefaultLoadBalancer for table a 2014-04-17 13:31:16,332 [master.EventCoordinator] INFO : Table state in zookeeper changed for a to ONLINE
          Hide
          Mike Drob added a comment -

          This table is up to ~250 tablets and they are still all hosted on a single tablet server.

          Show
          Mike Drob added a comment - This table is up to ~250 tablets and they are still all hosted on a single tablet server.

            People

            • Assignee:
              Sean Busbey
              Reporter:
              Mike Drob
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 20m
                2h 20m

                  Development