Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9958

The FileSystem used by HdfsBackupRepository gets closed before the backup completes.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Workaround
    • Affects Version/s: 6.2.1
    • Fix Version/s: None
    • Component/s: Hadoop Integration, hdfs
    • Labels:
      None

      Description

      My shards get backed up correctly, but then it fails when backing up the state from ZK. From the logs, it looks like the underlying FS gets closed before the config stuff is written:

      DEBUG - 2017-01-11 22:39:12.889; [   ] com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase; GHFS.close:=> 
      INFO  - 2017-01-11 22:39:12.889; [   ] org.apache.solr.handler.SnapShooter; Done creating backup snapshot: shard1 at gs://master-sector-142100.appspot.com/backups2/tim5
      INFO  - 2017-01-11 22:39:12.889; [   ] org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores params={core=gettingstarted_shard1_replica1&qt=/admin/cores&name=shard1&action=BACKUPCORE&location=gs://master-sector-142100.appspot.com/backups2/tim5&wt=javabin&version=2} status=0 QTime=24954
      INFO  - 2017-01-11 22:39:12.890; [   ] org.apache.solr.cloud.BackupCmd; Starting to backup ZK data for backupName=tim5
      INFO  - 2017-01-11 22:39:12.890; [   ] org.apache.solr.common.cloud.ZkStateReader; Load collection config from: [/collections/gettingstarted]
      INFO  - 2017-01-11 22:39:12.891; [   ] org.apache.solr.common.cloud.ZkStateReader; path=[/collections/gettingstarted] [configName]=[gettingstarted] specified config exists in ZooKeeper
      ERROR - 2017-01-11 22:39:12.892; [   ] org.apache.solr.common.SolrException; Collection: gettingstarted operation: backup failed:java.io.IOException: GoogleHadoopFileSystem has been closed or not initialized.
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.checkOpen(GoogleHadoopFileSystemBase.java:1927)
          at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.mkdirs(GoogleHadoopFileSystemBase.java:1367)
          at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
          at org.apache.solr.core.backup.repository.HdfsBackupRepository.createDirectory(HdfsBackupRepository.java:153)
          at org.apache.solr.core.backup.BackupManager.downloadConfigDir(BackupManager.java:186)
          at org.apache.solr.cloud.BackupCmd.call(BackupCmd.java:111)
          at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:222)
          at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
          at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. SOLR-9958.patch
          0.7 kB
          Timothy Potter

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                thelabdude Timothy Potter
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: