[CONNECTORS-1731] Suggestion for adding handling process for ManifoldCFException in Generic Repository Connector

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: ManifoldCF 2.24
• Assignee: Karl Wright
• Reporter: Nguyen Huu Nhat
• Created: 21/Sep/22 02:26
• Updated: 08/Dec/22 06:45
• Resolved: 21/Sep/22 09:39

Description

Hi there,

I would like to suggest the following retry-related improvements:

1. Connector name

Generic Repository Connector

2. Reasons for improvement

In the process of ExecuteSeedingThread, DocumentVersionThread and ExecuteProcessThread, when HttpClient.execute is executed, there are chances that connection gets interrupted and connection error occurs (HTTP status code <> 200).
When HTTP status code <> 200, exception of type ManifoldCFException will be thrown. However, there is no process to handle this, which leads to job being aborted.

As errors relating to connection (HTTP status code <> 200) can be resolved automatically, retry should be added for cases like this.

3. Improvements

Improvement includes the followings:

• Adding method to handle retry for exception of type ManifoldCFException
• Calling method to handle ManifoldCFException exception generated when executing in threads: ExecuteSeedingThread, DocumentVersionThread, ExecuteProcessThread

4. Suggested source code (based on release 2.22.1)

• Adding method to handle retry for exception of type ManifoldCFException
  https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L1026

    /**
     * Function for handling ManifoldCFException exception caused by connection error.
     * In case of connection error, ServiceInterruption exception is thrown to perform retry.
     * 
     * @param e ManifoldCFException
     * @throws ServiceInterruption
     */
    protected static void handleManifoldCFException(ManifoldCFException e)
      throws ServiceInterruption {
      long currentTime = System.currentTimeMillis();
      throw new ServiceInterruption("Connection error: " + e.getMessage(), e, currentTime + 300000L,
        currentTime + 3 * 60 * 60000L, -1, false);
    }
  

• Calling method to handle ManifoldCFException exception generated when executing in threads: ExecuteSeedingThread, DocumentVersionThread, ExecuteProcessThread
  • ExecuteSeedingThread
    https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L256

        } catch (InterruptedException e) {
          t.interrupt();
          throw new ManifoldCFException("Interrupted: " + e.getMessage(), e,
            ManifoldCFException.INTERRUPTED);
    +   } catch (ManifoldCFException e) {
    +     handleManifoldCFException(e);
        }
        return new Long(seedTime).toString();
    

• • DocumentVersionThread
    https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L304

          try {
            versions = versioningThread.finishUp();
          } catch (IOException ex) {
            handleIOException((IOException)ex);
    +     } catch (ManifoldCFException ex) {
    +       handleManifoldCFException(ex);
          } catch (InterruptedException ex) {
            throw new ManifoldCFException(ex.getMessage(), ex, ManifoldCFException.INTERRUPTED);
          }
    

• • ExecuteProcessThread
    https://github.com/apache/manifoldcf/blob/release-2.22.1/connectors/generic/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/generic/GenericConnector.java#L445

                } catch (InterruptedIOException e) {
                  t.interrupt();
                  throw new ManifoldCFException("Interrupted: " + e.getMessage(), e, ManifoldCFException.INTERRUPTED);
                } catch (IOException e) {
                  handleIOException(e);
    +           } catch (ManifoldCFException e) {
    +             handleManifoldCFException(e);
                }