Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3784

Chunk commit data using lower of byte-based and row-count limits

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.11.0
    • Labels:
      None

      Description

      We have a byte-based limit that determines how much data we send over at a time when a commit occurs (PHOENIX-541), but we should also have a row-count limit. We could check both the byte-based limit and the row-count limit and ensure the batch size meets both constraints. This would help prevent too many rows from being submitted to the server at one time and decrease the likelihood of conflicting rows amongst batches.

      1. PHOENIX-3784.patch
        15 kB
        Thomas D'Silva
      2. PHOENIX-3784-final.patch
        16 kB
        Thomas D'Silva

        Issue Links

          Activity

          Hide
          jamestaylor James Taylor added a comment -

          This seems like an important one to get into 4.11.0 and a natural extension of the nice work Geoffrey Jacoby did for PHOENIX-514. FYI, Lars Hofhansl Andrew Purtell.

          Show
          jamestaylor James Taylor added a comment - This seems like an important one to get into 4.11.0 and a natural extension of the nice work Geoffrey Jacoby did for PHOENIX-514 . FYI, Lars Hofhansl Andrew Purtell .
          Hide
          tdsilva Thomas D'Silva added a comment -

          James Taylor
          Can you please review?

          Show
          tdsilva Thomas D'Silva added a comment - James Taylor Can you please review?
          Hide
          jamestaylor James Taylor added a comment -

          +1, but please lower DEFAULT_MUTATE_BATCH_SIZE_BYTES to 2MB. Thanks, Thomas D'Silva!

          Show
          jamestaylor James Taylor added a comment - +1, but please lower DEFAULT_MUTATE_BATCH_SIZE_BYTES to 2MB. Thanks, Thomas D'Silva !
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12869972/PHOENIX-3784.patch
          against master branch at commit 5f9cf15e272fc9d92a3165753ac2157396851bd6.
          ATTACHMENT ID: 12869972

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 47 warning messages.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + connection = (PhoenixConnection) DriverManager.getConnection(getUrl(), connectionProperties);
          + if (readyToCommit(rowCount, mutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) {
          + if (readyToCommit(rowCount, indexMutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) {
          + private boolean readyToCommit(int rowCount, long mutationSize, int maxBatchSize, long maxBatchSizeBytes) {
          + int maxBatchSize = config.getInt(MUTATE_BATCH_SIZE_ATTRIB, QueryServicesOptions.DEFAULT_MUTATE_BATCH_SIZE);
          + if (readyToCommit(rowCount, mutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) {
          + List<List<Mutation>> mutationBatchList = getMutationBatchList(batchSize, batchSizeBytes, mutationList);
          + public static List<List<Mutation>> getMutationBatchList(long batchSize, long batchSizeBytes, List<Mutation> allMutationList) {
          + if (currentList.size() == batchSize || currentBatchSizeBytes + mutationSizeBytes > batchSizeBytes) {

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//testReport/
          Javadoc warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12869972/PHOENIX-3784.patch against master branch at commit 5f9cf15e272fc9d92a3165753ac2157396851bd6. ATTACHMENT ID: 12869972 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 47 warning messages. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + connection = (PhoenixConnection) DriverManager.getConnection(getUrl(), connectionProperties); + if (readyToCommit(rowCount, mutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) { + if (readyToCommit(rowCount, indexMutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) { + private boolean readyToCommit(int rowCount, long mutationSize, int maxBatchSize, long maxBatchSizeBytes) { + int maxBatchSize = config.getInt(MUTATE_BATCH_SIZE_ATTRIB, QueryServicesOptions.DEFAULT_MUTATE_BATCH_SIZE); + if (readyToCommit(rowCount, mutations.heapSize(), maxBatchSize, maxBatchSizeBytes)) { + List<List<Mutation>> mutationBatchList = getMutationBatchList(batchSize, batchSizeBytes, mutationList); + public static List<List<Mutation>> getMutationBatchList(long batchSize, long batchSizeBytes, List<Mutation> allMutationList) { + if (currentList.size() == batchSize || currentBatchSizeBytes + mutationSizeBytes > batchSizeBytes) { -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//testReport/ Javadoc warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/908//console This message is automatically generated.
          Hide
          tdsilva Thomas D'Silva added a comment -

          Sure, I have attached the final patch.

          Show
          tdsilva Thomas D'Silva added a comment - Sure, I have attached the final patch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Phoenix-master #1627 (See https://builds.apache.org/job/Phoenix-master/1627/)
          PHOENIX-3784 Chunk commit data using lower of byte-based and row-count (thomas: rev e2ec76559b1d36fee7b696d7ee2989059e2bbe8c)

          • (edit) phoenix-core/src/main/java/org/apache/phoenix/util/JDBCUtil.java
          • (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
          • (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java
          • (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
          • (edit) phoenix-core/src/test/java/org/apache/phoenix/jdbc/PhoenixDriverTest.java
          • (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryMoreIT.java
          • (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Phoenix-master #1627 (See https://builds.apache.org/job/Phoenix-master/1627/ ) PHOENIX-3784 Chunk commit data using lower of byte-based and row-count (thomas: rev e2ec76559b1d36fee7b696d7ee2989059e2bbe8c) (edit) phoenix-core/src/main/java/org/apache/phoenix/util/JDBCUtil.java (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java (edit) phoenix-core/src/test/java/org/apache/phoenix/jdbc/PhoenixDriverTest.java (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryMoreIT.java (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java

            People

            • Assignee:
              tdsilva Thomas D'Silva
              Reporter:
              jamestaylor James Taylor
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development