[GEODE-2088] A get in transaction may get incorrect TransactionDataNotColocatedException when bucket is not hosted locally

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: transactions
• Assignee: Eric Shu
• Reporter: Eric Shu
• Created: 08/Nov/16 01:19
• Updated: 09/Feb/17 22:53
• Resolved: 15/Nov/16 21:31

Description

Currently a get of a PR in transaction on a data node with TXState will first check if the bucket is hosted locally by following code.

  public Object getLocally(int bucketId, final Object key, final Object aCallbackArgument,
      boolean disableCopyOnRead, boolean preferCD, ClientProxyMembershipID requestingClient,
      EntryEventImpl clientEvent, boolean returnTombstones, boolean opScopeIsLocal)
      throws PrimaryBucketException, ForceReattemptException, PRLocallyDestroyedException {
    final BucketRegion bucketRegion = getInitializedBucketForId(key, Integer.valueOf(bucketId));

The BucketNotFoundException is thrown if the node does not host the bucket:

  public BucketRegion getInitializedBucketForId(Object key, Integer bucketId)
      throws ForceReattemptException {
    final BucketRegion bucketRegion = this.localBucket2RegionMap.get(bucketId);
    if (null == bucketRegion) {
      this.partitionedRegion.checkReadiness();
      if (logger.isDebugEnabled()) {
        logger.debug("Got null bucket region for bucketId={}{}{} for PartitionedRegion = {}",
            this.partitionedRegion.getPRId(), PartitionedRegion.BUCKET_ID_SEPARATOR, bucketId,
            this.partitionedRegion);
      }
      ForceReattemptException fre = new BucketNotFoundException(
          LocalizedStrings.PartitionedRegionDataStore_BUCKET_ID_0_NOT_FOUND_ON_VM_1
              .toLocalizedString(
                  new Object[] {this.partitionedRegion.bucketStringForLogs(bucketId.intValue()),
                      this.partitionedRegion.getMyId()}));
      if (key != null) {
        fre.setHash(key.hashCode());
      }
      throw fre;
    }

Currently, transaction would fail with the TransactionDataNotColocatedException if bucket is not on local data store

          if (prce instanceof BucketNotFoundException) {
            TransactionException ex = new TransactionDataNotColocatedException(
                LocalizedStrings.PartitionedRegion_KEY_0_NOT_COLOCATED_WITH_TRANSACTION
                    .toLocalizedString(key));
            ex.initCause(prce);
            throw ex;
          }

This TransactionDataNotColocatedException is fine if the node never hosts bucket, or only previous operations within the transaction touches replicate regions earlier. However, if a bucket is moved to another node due to rebalance, and previous entry operations within the transaction only touches colocated regions, the transaction should fail with TransactionDataRebalancedException.

We do not have a good way currently to throw the correct exception. One idea is to iterate through the existing TXRegions in the TXState to see whether we should throw TransactionDataRebalancedException.