Accumulo
  1. Accumulo
  2. ACCUMULO-1062

large numbers of threads make in-memory updates slow

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.3, 1.5.0
    • Component/s: tserver
    • Labels:
      None
    • Environment:

      testing on large clusters

      Description

      Using hundreds of ingest programs, each running several batch writers, ingest would eventually become slow. Testing (attached) proved that performance fell off as the number of threads increased.

      1. SlamTest.java
        2 kB
        Eric Newton

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Keith Turner added a comment -

          Eric and I were poking at this issue using the test program. We found the following code in InMemoryMap is causing the performance problem.

                synchronized (this) {
                  // Can not update mutationCount while writes that started before
                  // are in progress, this would cause partial mutations to be seen.
                  // Also, can not continue until mutation count is updated, because
                  // a read may not see a successful write. Therefore writes must
                  // wait for writes that started before to finish.
                  
                  while (kvCount.get() != kv - 1) {
                    try {
                      wait();
                    } catch (InterruptedException ex) {
                      // ignored
                    }
                  }
                  kvCount.set(kv + numKVs - 1);
                  notifyAll();
                }
          
          Show
          Keith Turner added a comment - Eric and I were poking at this issue using the test program. We found the following code in InMemoryMap is causing the performance problem. synchronized ( this ) { // Can not update mutationCount while writes that started before // are in progress, this would cause partial mutations to be seen. // Also, can not continue until mutation count is updated, because // a read may not see a successful write. Therefore writes must // wait for writes that started before to finish. while (kvCount.get() != kv - 1) { try { wait(); } catch (InterruptedException ex) { // ignored } } kvCount.set(kv + numKVs - 1); notifyAll(); }
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk-Hadoop-2.0 #90 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/90/)
          ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts (Revision 1445876)

          Result = UNSTABLE
          ecn :
          Files :

          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #90 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/90/ ) ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts (Revision 1445876) Result = UNSTABLE ecn : Files : /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.4.x #274 (See https://builds.apache.org/job/Accumulo-1.4.x/274/)
          ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts; merge to 1.4 branch (Revision 1445880)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java
          • /accumulo/branches/1.4/src/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.4.x #274 (See https://builds.apache.org/job/Accumulo-1.4.x/274/ ) ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts; merge to 1.4 branch (Revision 1445880) Result = SUCCESS ecn : Files : /accumulo/branches/1.4/src/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java /accumulo/branches/1.4/src/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #732 (See https://builds.apache.org/job/Accumulo-Trunk/732/)
          ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts (Revision 1445876)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #732 (See https://builds.apache.org/job/Accumulo-Trunk/732/ ) ACCUMULO-1062 serialize writes to ensure counts, rather than serializing the return using check/notify for counts (Revision 1445876) Result = SUCCESS ecn : Files : /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/InMemoryMap.java /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk-Hadoop-2.0 #93 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/93/)
          ACCUMULO-1062 2000 threads is a little too much for a unit test (Revision 1446290)

          Result = UNSTABLE
          ecn :
          Files :

          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #93 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/93/ ) ACCUMULO-1062 2000 threads is a little too much for a unit test (Revision 1446290) Result = UNSTABLE ecn : Files : /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.4.x #275 (See https://builds.apache.org/job/Accumulo-1.4.x/275/)
          ACCUMULO-1062 2000 threads is a little too much for a unit test; merge to 1.4 branch (Revision 1446291)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/branches/1.4
          • /accumulo/branches/1.4/src
          • /accumulo/branches/1.4/src/core
          • /accumulo/branches/1.4/src/server
          • /accumulo/branches/1.4/src/server/src
          • /accumulo/branches/1.4/src/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.4.x #275 (See https://builds.apache.org/job/Accumulo-1.4.x/275/ ) ACCUMULO-1062 2000 threads is a little too much for a unit test; merge to 1.4 branch (Revision 1446291) Result = SUCCESS ecn : Files : /accumulo/branches/1.4 /accumulo/branches/1.4/src /accumulo/branches/1.4/src/core /accumulo/branches/1.4/src/server /accumulo/branches/1.4/src/server/src /accumulo/branches/1.4/src/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #735 (See https://builds.apache.org/job/Accumulo-Trunk/735/)
          ACCUMULO-1062 2000 threads is a little too much for a unit test (Revision 1446290)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #735 (See https://builds.apache.org/job/Accumulo-Trunk/735/ ) ACCUMULO-1062 2000 threads is a little too much for a unit test (Revision 1446290) Result = SUCCESS ecn : Files : /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #790 (See https://builds.apache.org/job/Accumulo-Trunk/790/)
          ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459069)

          Result = FAILURE
          ecn :
          Files :

          • /accumulo/trunk
          • /accumulo/trunk/assemble
          • /accumulo/trunk/core
          • /accumulo/trunk/examples
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java
          • /accumulo/trunk/server
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          • /accumulo/trunk/src
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #790 (See https://builds.apache.org/job/Accumulo-Trunk/790/ ) ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459069) Result = FAILURE ecn : Files : /accumulo/trunk /accumulo/trunk/assemble /accumulo/trunk/core /accumulo/trunk/examples /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java /accumulo/trunk/server /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java /accumulo/trunk/src
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.5 #45 (See https://builds.apache.org/job/Accumulo-1.5/45/)
          ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459068)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.5 #45 (See https://builds.apache.org/job/Accumulo-1.5/45/ ) ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459068) Result = SUCCESS ecn : Files : /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.5-Hadoop-2.0 #43 (See https://builds.apache.org/job/Accumulo-1.5-Hadoop-2.0/43/)
          ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459068)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.5-Hadoop-2.0 #43 (See https://builds.apache.org/job/Accumulo-1.5-Hadoop-2.0/43/ ) ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459068) Result = SUCCESS ecn : Files : /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk-Hadoop-2.0 #149 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/149/)
          ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459069)

          Result = ABORTED
          ecn :
          Files :

          • /accumulo/trunk
          • /accumulo/trunk/assemble
          • /accumulo/trunk/core
          • /accumulo/trunk/examples
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java
          • /accumulo/trunk/server
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          • /accumulo/trunk/src
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #149 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/149/ ) ACCUMULO-1062 I suspect large number of threads is making the apache build servers fail our builds (Revision 1459069) Result = ABORTED ecn : Files : /accumulo/trunk /accumulo/trunk/assemble /accumulo/trunk/core /accumulo/trunk/examples /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java /accumulo/trunk/server /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java /accumulo/trunk/src
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk-Hadoop-2.0 #162 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/162/)
          ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461553)

          Result = UNSTABLE
          ecn :
          Files :

          • /accumulo/trunk
          • /accumulo/trunk/assemble
          • /accumulo/trunk/core
          • /accumulo/trunk/examples
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java
          • /accumulo/trunk/server
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          • /accumulo/trunk/src
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk-Hadoop-2.0 #162 (See https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/162/ ) ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461553) Result = UNSTABLE ecn : Files : /accumulo/trunk /accumulo/trunk/assemble /accumulo/trunk/core /accumulo/trunk/examples /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java /accumulo/trunk/server /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java /accumulo/trunk/src
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.5-Hadoop-2.0 #54 (See https://builds.apache.org/job/Accumulo-1.5-Hadoop-2.0/54/)
          ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461552)

          Result = UNSTABLE
          ecn :
          Files :

          • /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.5-Hadoop-2.0 #54 (See https://builds.apache.org/job/Accumulo-1.5-Hadoop-2.0/54/ ) ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461552) Result = UNSTABLE ecn : Files : /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-1.5 #57 (See https://builds.apache.org/job/Accumulo-1.5/57/)
          ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461552)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Show
          Hudson added a comment - Integrated in Accumulo-1.5 #57 (See https://builds.apache.org/job/Accumulo-1.5/57/ ) ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461552) Result = SUCCESS ecn : Files : /accumulo/branches/1.5/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #803 (See https://builds.apache.org/job/Accumulo-Trunk/803/)
          ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461553)

          Result = UNSTABLE
          ecn :
          Files :

          • /accumulo/trunk
          • /accumulo/trunk/assemble
          • /accumulo/trunk/core
          • /accumulo/trunk/examples
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java
          • /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java
          • /accumulo/trunk/server
          • /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java
          • /accumulo/trunk/src
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #803 (See https://builds.apache.org/job/Accumulo-Trunk/803/ ) ACCUMULO-1062 disabling testParallelWriteSpeed (Revision 1461553) Result = UNSTABLE ecn : Files : /accumulo/trunk /accumulo/trunk/assemble /accumulo/trunk/core /accumulo/trunk/examples /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/ZooStore.java /accumulo/trunk/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooSession.java /accumulo/trunk/server /accumulo/trunk/server/src/test/java/org/apache/accumulo/server/tabletserver/InMemoryMapTest.java /accumulo/trunk/src
          Hide
          ASF subversion and git services added a comment -

          Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.4.5-SNAPSHOT from Eric C. Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ]

          ACCUMULO-1062 disabling testParallelWriteSpeed

          git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.4.5-SNAPSHOT from Eric C. Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ] ACCUMULO-1062 disabling testParallelWriteSpeed git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68
          Hide
          ASF subversion and git services added a comment -

          Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.5.1-SNAPSHOT from Eric C. Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ]

          ACCUMULO-1062 disabling testParallelWriteSpeed

          git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.5.1-SNAPSHOT from Eric C. Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ] ACCUMULO-1062 disabling testParallelWriteSpeed git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68
          Hide
          ASF subversion and git services added a comment -

          Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.6.0-SNAPSHOT from Eric C. Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ]

          ACCUMULO-1062 disabling testParallelWriteSpeed

          git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68

          Show
          ASF subversion and git services added a comment - Commit ae0be1f60db5b3b2493f5816f4481c1b092ae6bf in branch refs/heads/1.6.0-SNAPSHOT from Eric C. Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=ae0be1f ] ACCUMULO-1062 disabling testParallelWriteSpeed git-svn-id: https://svn.apache.org/repos/asf/accumulo/branches/1.5@1461552 13f79535-47bb-0310-9956-ffa450edef68

            People

            • Assignee:
              Eric Newton
              Reporter:
              Eric Newton
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development