Uploaded image for project: 'Brooklyn'
  1. Brooklyn
  2. BROOKLYN-601

Provisioning very long when using RPM package

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      AWS Linux (RPM-based) but would presumably affect any environment which does not have SSH keys ready-to-use.

      Description

      System is an AWS Linux box (RPM-based) using the apache-brooklyn-1.0.0-M1-rc1_1.noarch.rpm package.

      Deployments using this system take a very long time to provision instances (tested on AWS).

      On examining the debug log, I can see it is search for the brooklyn system user's SSH key files:

      Since this newly-created system user doesn't have any SSH key files, it ends up trying to log in to the instance with no credentials:

      2018-09-13T14:12:38,734 - DEBUG 133 o.a.b.l.j.JcloudsLocation [nager-DV7OEufF-4] Credentials extracted for {id=eu-west-1/i-05bd13b5ba7ff1f69, providerId=i-05bd13b5ba7ff1f69, name=brooklyn-pezzf5-applicationaijwo-aijw-server-lant-dvr8, location=
      {{

      {scope=ZONE, id=eu-west-1b, description=eu-west-1b, parent=eu-west-1, iso3166Codes=[IE]}

      }}, group=brooklyn-pezzf5-applicationaijwo-aijw-server-lant, imageId=eu-west-1/ami-0eb66a0c3eb9f5183, os={{

      {family=ubuntu, arch=hvm, version=16.04, description=aws-marketplace/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180814-d83d0782-cb94-46d7-8993-f4ce15d1a484-ami-04169656fea786776.4, is64Bit=true}

      }}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-41-218, privateAddresses=[172.31.41.218], publicAddresses=[52.214.254.195], hardware={id=t2.small, providerId=t2.small, processors=[{{

      {cores=1.0, speed=0.2}

      }}], ram=2048, volumes=[{{

      {id=vol-01112c6e3486ac4d0, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}

      }}], hypervisor=xen, supportsImage=Predicates.and(requiresRootDeviceType(ebs),requiresVirtualizationType(hvm),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=brooklyn, userMetadata={Name=brooklyn-pezzf5-applicationaijwo-aijw-server-lant-dvr8, brooklyn-user=brooklyn, brooklyn-app-id=aijwo81iay, brooklyn-app-name=Application (aijwo81iay), brooklyn-entity-id=lant5abn8y, brooklyn-entity-name=Server, brooklyn-server-creation-date=2018-09-13-1411: brooklyn/brooklyn with OsCredential[no-public-key;no-private-key,no-password]/[user=brooklyn, passwordPresent=true, privateKeyPresent=false, shouldAuthenticateSudo=false]}}
      {{ 2018-09-13T14:12:38,756 - DEBUG 133 o.a.b.l.j.JcloudsLocation [nager-DV7OEufF-4] VM aws-ec2:eu-west-1@EmptySoftwareProcessImpl}}{{

      {id=lant5abn8y}

      }}: reported online, now waiting 5m for it to be contactable on brooklyn@52.214.254.195:22; trying 1 credential: user=brooklyn, password=******, key=<absent>
      {{ 2018-09-13T14:12:38,759 - DEBUG 128 o.a.b.SSH [nager-DV7OEufF-4] check-connectivity, initiating ssh on machine SshMachineLocation[AWS Dublin:brooklyn@52.214.254.195/52.214.254.195:22(id=jd41hctb1o)]: #!/bin/bash -e}}
      {{ ; true}}
      {{ 2018-09-13T14:12:38,769 - DEBUG 128 o.a.b.l.s.SshMachineLocation [nager-DV7OEufF-4] Create-unmanaged for SshMachineLocation[AWS Dublin:brooklyn@52.214.254.195/52.214.254.195:22(id=jd41hctb1o)]; no explicit cleanup task; ssh-pool cache will only be closed when machine is closed}}
      {{ 2018-09-13T14:12:38,770 - DEBUG 128 o.a.b.l.s.SshMachineLocation [nager-DV7OEufF-4] org.apache.brooklyn.location.ssh.SshMachineLocation$4@7c8aa530 building ssh pool for 52.214.254.195:22 with properties:}}{{

      {connectTimeout=30000, sshTries=1, sessionTimeout=30000, sshTriesTimeout=30000}

      }}2018-09-13T14:12:38,863 - DEBUG 128 o.a.b.u.c.i.s.s.SshjTool [nager-DV7OEufF-4] << (brooklyn@52.214.254.195:22) error acquiring{{

      {hostAndPort=52.214.254.195:22, user=brooklyn, ssh=1626499876, password=xxxxxx, privateKeyFile=null, privateKey=null, connectTimeout=30000, sessionTimeout=30000}

      }}(attempt 1/1, in time 84ms/30s) (rethrowing, out of retries): Exhausted available authentication methods
      {{ 2018-09-13T14:12:38,864 - DEBUG 128 o.a.b.u.c.i.s.s.SshjTool [nager-DV7OEufF-4] brooklyn@52.214.254.195:22 failed to connect (rethrowing)}}
      {{ org.apache.brooklyn.util.core.internal.ssh.SshException: (brooklyn@52.214.254.195:22) (brooklyn@52.214.254.195:22) error acquiring}}{{

      {hostAndPort=52.214.254.195:22, user=brooklyn, ssh=1626499876, password=xxxxxx, privateKeyFile=null, privateKey=null, connectTimeout=30000, sessionTimeout=30000}

      }}(attempt 1/1, in time 84ms/30s); out of retries: Exhausted available authentication methods}}

      It then repeats the connection attempt for 10 minutes.

      Strangely, once this process has failed, it's not an error: Brooklyn continues, and somehow it's found some useful credentials and it logs in:

      2018-09-13T14:22:40,295 - DEBUG 128 o.a.b.c.l.LocationConfigUtils [nager-DV7OEufF-4] Inferring OS credentials
      2018-09-13T14:22:40,296 - DEBUG 128 o.a.b.c.l.LocationConfigUtils [nager-DV7OEufF-4] Public key data extracted
      2018-09-13T14:22:40,297 - DEBUG 128 o.a.b.c.l.LocationConfigUtils [nager-DV7OEufF-4] OS credential inference: OsCredential[ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDD1jMycpme/Acx57BwGmVExPbNFOISJD66IoI1zCVwr0sErgmQuukZpvw1/dP3ZGeRo3UImieiSZ1Re8lA7Mw5anYgOwRMzbt/lLnc/aphMq3qwV2jJHmz3oohDOADa13DWKRSn9zoOdQxnmzt1Q5JHPpxGEiiA5IdHJzWOfCfkQ==;private-key-present,no-password]

      This obviously gives a very poor impression. Adding an SSH key for the Brooklyn user means that provisioning happens much, much faster.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rdowner Richard Downer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: