[ACCUMULO-3096] Scans stuck and seeing error message about constraint violation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1, 1.6.0
• Fix Version/s: 1.5.2, 1.6.1, 1.7.0
• Assignee: Keith Turner
• Reporter: Keith Turner
• Created: 29/Aug/14 20:17
• Updated: 04/Sep/14 16:01
• Resolved: 04/Sep/14 16:01

Description

Just helped someone debug an issue. Their scans were getting stuck on a certain tserver (determined tserver by turning on debug in shell). On the tserver, there was a contant stream of messages about a metadata table contstraint violate because Bulk load transaction no longer running.

The following code in Tablet.importMapFiles()

          synchronized (timeLock) {
            if (bulkTime > persistedTime)
              persistedTime = bulkTime;

            MetadataTableUtil.updateTabletDataFile(tid, extent, paths, tabletTime.getMetadataValue(persistedTime), creds, tabletServer.getLock());
          }

Ended up calling the following code in MetadataTableUtil.

public static void update(Credentials credentials, ZooLock zooLock, Mutation m, KeyExtent extent) {
    Writer t = extent.isMeta() ? getRootTable(credentials) : getMetadataTable(credentials);
    if (zooLock != null)
      putLockID(zooLock, m);
    while (true) {
      try {
        t.update(m);
        return;
      } catch (AccumuloException e) {
        log.error(e, e);
      } catch (AccumuloSecurityException e) {
        log.error(e, e);
      } catch (ConstraintViolationException e) {
        log.error(e, e);
      } catch (TableNotFoundException e) {
        log.error(e, e);
      }
      UtilWaitThread.sleep(1000);
    }

  }

So when the constraint failed, it retried forever. It did this while holding timeLock, which in turn prevented compactions from completing, which eventually gummed up scans.

Issue Links

• links to: Review