Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4027

Mark index as disabled during partial rebuild after configurable amount of time

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.12.0
    • None

    Description

      Instead of marking an index as permanently disabled in the partial index rebuilder when a failure occurs, we should let it try again up to a configurable amount of time. The reason is that the fail-fast approach with the lower RPC timeout will continue to cause a failure until the index region can be written to. This will allow us to ride out region moves without a long RPC time out and thus without holding handler threads for long periods of time. We can base the failure on the INDEX_DISABLE_TIMESTAMP value of an index as we walk through the scan results here in MetaDataRegionObserver. :

                      do {
                          results.clear();
                          hasMore = scanner.next(results);
                          if (results.isEmpty()) break;
      
                          Result r = Result.create(results);
                          byte[] disabledTimeStamp = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
                              PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES);
                          byte[] indexState = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
                                  PhoenixDatabaseMetaData.INDEX_STATE_BYTES);
      
                          if (disabledTimeStamp == null || disabledTimeStamp.length == 0) {
                              continue;
                          }
      
                          // TODO: if disabledTimeStamp - System.currentTimeMillis() > configurableAmount 
                          // then disable the index.
      

      I'd propose we allow 30 minutes to get an index back online.

      Attachments

        1. PHOENIX-4027_addendum_2.patch
          2 kB
          Rajeshbabu Chintaguntla
        2. PHOENIX-4027_addendum.patch
          16 kB
          Samarth Jain
        3. PHOENIX-4027.patch
          13 kB
          Samarth Jain

        Activity

          People

            samarthjain Samarth Jain
            jamestaylor James R. Taylor
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: