[ACCUMULO-3609] BalanceInPresenceOfOfflineTableIT seems to be hanging

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.3, 1.6.3, 1.7.0
• Component/s: test
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 20/Feb/15 21:23
• Updated: 24/Feb/15 06:03
• Resolved: 24/Feb/15 06:03

Description

I noticed today that BalanceInPresenceOfOfflineTableIT seems to have started failing for me.

I see the following repeated ad infinitum in the tserver logs

2015-02-20 15:52:25,830 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
2015-02-20 15:52:25,833 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
2015-02-20 15:52:25,840 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
2015-02-20 15:52:25,841 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
2015-02-20 15:52:25,850 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,855 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,862 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,865 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,878 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
2015-02-20 15:52:25,881 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
2015-02-20 15:52:25,889 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
2015-02-20 15:52:25,891 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000100199 and row_00001002 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c2.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf]
2015-02-20 15:52:25,897 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,900 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000141299 and row_00001413 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000c7.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,904 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,906 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000179599 and row_00001796 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000eq.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000dk.rf]
2015-02-20 15:52:25,920 [util.FileUtil] WARN : Failed to find mid point using indexes, falling back to data files which is slower. No entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]
2015-02-20 15:52:25,923 [tablet.Tablet] ERROR: Failed to find midpoint Failed to find mid point, no entries between row_0000040499 and row_00000405 for [file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000aw.rf, file:/.../test/target/mini-tests/org.apache.accumulo.test.functional.BalanceInPresenceOfOfflineTableIT_test/accumulo/tables/2/default_tablet/F00000au.rf]

It does seem to eventually finish (occasionally), so I'm not really sure what's happening yet.

Issue Links

• is broken by: ACCUMULO-3601 master does not process migrations