[SOLR-9954] SnapShooter createSnapshot can swallow an exception raised by the underlying backup repo

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 6.2.1, 6.3
• Fix Version/s: 6.4, 7.0
• Component/s: Hadoop Integration
• Security Level: Public (Default Security Level. Issues are Public)
• Assignee: Timothy Potter
• Reporter: Timothy Potter
• Created: 10/Jan/17 23:19
• Updated: 13/Jan/17 20:23
• Resolved: 11/Jan/17 01:52

Description

While configuring the HdfsBackupRepository to use Google compute storage, I misconfigured the permissions on my bucket. Unfortunately, the exception that would have pointed me in the right direction gets squelched by the finally block in createSnapshot:

    } finally {
      if (!success) {
        backupRepo.deleteDirectory(snapshotDirPath);
      }
    }

If there's a permissions issue, then the deleteDelectory is going to fail and raise another exception from the finally block, which swallows the original exception. For example:

ERROR - 2017-01-10 18:38:52.650; [c:gettingstarted s:shard1 r:core_node1 x:gettingstarted_shard1_replica1] org.apache.solr.handler.SnapShooter; Exception while creating snapshot
java.io.IOException: GoogleHadoopFileSystem has been closed or not initialized.
    at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.checkOpen(GoogleHadoopFileSystemBase.java:1927)
    at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.delete(GoogleHadoopFileSystemBase.java:1255)
    at org.apache.solr.core.backup.repository.HdfsBackupRepository.deleteDirectory(HdfsBackupRepository.java:160)
    at org.apache.solr.handler.SnapShooter.createSnapshot(SnapShooter.java:234)
    at org.apache.solr.handler.SnapShooter.lambda$createSnapAsync$1(SnapShooter.java:186)
    at org.apache.solr.handler.SnapShooter$$Lambda$89/43739789.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

That's merely the symptom and not the actual cause of the failure.

Issue Links

• links to: GitHub Pull Request #137