Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9958

The FileSystem used by HdfsBackupRepository gets closed before the backup completes.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Workaround
    • Affects Version/s: 6.2.1
    • Fix Version/s: None
    • Component/s: Hadoop Integration, hdfs
    • Labels:
      None

      Description

      My shards get backed up correctly, but then it fails when backing up the state from ZK. From the logs, it looks like the underlying FS gets closed before the config stuff is written:

      DEBUG - 2017-01-11 22:39:12.889; [   ] com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase; GHFS.close:=> 
      INFO  - 2017-01-11 22:39:12.889; [   ] org.apache.solr.handler.SnapShooter; Done creating backup snapshot: shard1 at gs://master-sector-142100.appspot.com/backups2/tim5
      INFO  - 2017-01-11 22:39:12.889; [   ] org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores params={core=gettingstarted_shard1_replica1&qt=/admin/cores&name=shard1&action=BACKUPCORE&location=gs://master-sector-142100.appspot.com/backups2/tim5&wt=javabin&version=2} status=0 QTime=24954
      INFO  - 2017-01-11 22:39:12.890; [   ] org.apache.solr.cloud.BackupCmd; Starting to backup ZK data for backupName=tim5
      INFO  - 2017-01-11 22:39:12.890; [   ] org.apache.solr.common.cloud.ZkStateReader; Load collection config from: [/collections/gettingstarted]
      INFO  - 2017-01-11 22:39:12.891; [   ] org.apache.solr.common.cloud.ZkStateReader; path=[/collections/gettingstarted] [configName]=[gettingstarted] specified config exists in ZooKeeper
      ERROR - 2017-01-11 22:39:12.892; [   ] org.apache.solr.common.SolrException; Collection: gettingstarted operation: backup failed:java.io.IOException: GoogleHadoopFileSystem has been closed or not initialized.
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.checkOpen(GoogleHadoopFileSystemBase.java:1927)
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.mkdirs(GoogleHadoopFileSystemBase.java:1367)
          at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
          at org.apache.solr.core.backup.repository.HdfsBackupRepository.createDirectory(HdfsBackupRepository.java:153)
          at org.apache.solr.core.backup.BackupManager.downloadConfigDir(BackupManager.java:186)
          at org.apache.solr.cloud.BackupCmd.call(BackupCmd.java:111)
          at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:222)
          at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
          at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              thelabdude Timothy Potter

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment