Uploaded image for project: 'Airavata'
  1. Airavata
  2. AIRAVATA-2624

Stampede2 cluster SSH connectivity issue

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Job submission fails at env creation due to JSch initialization error.
      Error messages
      2018-01-09 09:46:10,786 [pool-7-thread-15] ERROR o.a.a.gfac.core.GFacException process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - JSch initialization error
      com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:512)
      at com.jcraft.jsch.Session.connect(Session.java:183)
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:138)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:315)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.makeDirectory(HPCRemoteCluster.java:242)
      at org.apache.airavata.gfac.impl.task.EnvironmentSetupTask.execute(EnvironmentSetupTask.java:51)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:553)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      2018-01-09 09:46:10,786 [pool-7-thread-15] ERROR o.a.a.g.i.t.EnvironmentSetupTask process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - Error while environment setup
      org.apache.airavata.gfac.core.GFacException: JSch initialization error
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:138)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:315)
      at org.apache.airavata.gfac.impl.HPCRemoteCluster.makeDirectory(HPCRemoteCluster.java:242)
      at org.apache.airavata.gfac.impl.task.EnvironmentSetupTask.execute(EnvironmentSetupTask.java:51)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:553)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.jcraft.jsch.JSchException: Auth fail
      at com.jcraft.jsch.Session.connect(Session.java:512)
      at com.jcraft.jsch.Session.connect(Session.java:183)
      at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
      ... 14 common frames omitted
      2018-01-09 09:46:10,786 [pool-7-thread-15] INFO o.a.a.g.c.context.TaskContext process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, taskId: TASK_03ea62e9-55c0-4d9f-bfc9-af26c9c45aa0, type: ENV_SETUP : Task status changed EXECUTING -> FAILED
      2018-01-09 09:46:10,793 [pool-7-thread-15] ERROR o.a.a.gfac.impl.GFacEngineImpl process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, taskId: TASK_03ea62e9-55c0-4d9f-bfc9-af26c9c45aa0 type: ENV_SETUP,:- Input staging failed, reason: Error while environment setup
      2018-01-09 09:46:10,803 [pool-7-thread-15] ERROR o.a.a.gfac.impl.GFacWorker process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - GFac Worker throws an exception
      org.apache.airavata.gfac.core.GFacException: Error while environment setup
      at org.apache.airavata.gfac.impl.GFacEngineImpl.configureWorkspace(GFacEngineImpl.java:572)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:324)
      at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
      at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
      at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
      at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      2018-01-09 09:46:10,803 [pool-7-thread-15] INFO o.a.a.g.c.c.ProcessContext process_id=PROCESS_650014f6-fcb6-4680-90ea-898bee373f37, token_id=3d65bf6d-2c9f-4166-a51b-e76e0022bd3b, experiment_id=Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, gateway_id=seagrid - expId: Clone_of_st2molcastest_e2942a34-c9c7-4f04-8ccb-af6fe27e0990, processId: PROCESS_650014f6-fcb6-4680-90ea-898bee373f37 :- Process status changed CONFIGURING_WORKSPACE -> FAILE

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            marcuschristie Marcus Christie
            eroma_a Eroma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment