Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4027

Mark index as disabled during partial rebuild after configurable amount of time

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.12.0
    • Labels:

      Description

      Instead of marking an index as permanently disabled in the partial index rebuilder when a failure occurs, we should let it try again up to a configurable amount of time. The reason is that the fail-fast approach with the lower RPC timeout will continue to cause a failure until the index region can be written to. This will allow us to ride out region moves without a long RPC time out and thus without holding handler threads for long periods of time. We can base the failure on the INDEX_DISABLE_TIMESTAMP value of an index as we walk through the scan results here in MetaDataRegionObserver. :

                      do {
                          results.clear();
                          hasMore = scanner.next(results);
                          if (results.isEmpty()) break;
      
                          Result r = Result.create(results);
                          byte[] disabledTimeStamp = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
                              PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES);
                          byte[] indexState = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
                                  PhoenixDatabaseMetaData.INDEX_STATE_BYTES);
      
                          if (disabledTimeStamp == null || disabledTimeStamp.length == 0) {
                              continue;
                          }
      
                          // TODO: if disabledTimeStamp - System.currentTimeMillis() > configurableAmount 
                          // then disable the index.
      

      I'd propose we allow 30 minutes to get an index back online.

        Attachments

        1. PHOENIX-4027.patch
          13 kB
          Samarth Jain
        2. PHOENIX-4027_addendum.patch
          16 kB
          Samarth Jain
        3. PHOENIX-4027_addendum_2.patch
          2 kB
          Rajeshbabu Chintaguntla

          Activity

            People

            • Assignee:
              samarthjain Samarth Jain
              Reporter:
              jamestaylor James Taylor
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: