[CLOUDSTACK-9965] Custom SSL Certificate deployment on SSVM is broken

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.10.0.0
• Component/s: Secondary Storage
• Security Level: Public (Anyone can view this level - this is the default.)

Environment

Hypervisor: KVM.
CloudStack version: 4.10.0-SNAPSHOT from 7-June-2016.
SSVM template source: http://cloudstack.apt-get.eu/systemvm/4.10/RC4/systemvm64template-master-4.10.0-kvm.qcow2.bz2

• Flags: Important
• Assignee: Unassigned
• Reporter: John McEleney
• Created: 18/Jun/17 11:13
• Updated: 18/Jun/17 13:31

Description

I've seen this bug with both the RC4 SSVM template and the prior release.

When a custom SSL certificate is added to CloudStack, the Secondary Storage VM (SSVM) deployment fails. In spite of the custom certificate being copied to the VM, Apache remains configured to use the default Snakeoil certificate.

I've looked through some of the scripting on the SSVM to try to work out why it's failing, and I've been able to tweak the running system to get the certificate working.

Relevant logs entries in /varlog/cloud.log:

2017-06-18 08:58:42,170 DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing: /usr/local/cloud/systemvm/config_ssl.sh -i 94.229.139.152 -h s-22-VM -k /tmp/prvkey2407180375973626296.tmp -p /tmp/pubcert1028956520690640221.tmp -t /tmp/certchain5546051081886085440.tmp -u /tmp/rootcert3072981443992074804.tmp 
sed: can't read /etc/apache2/ports.conf: No such file or directory
sed: can't read /etc/apache2/ports.conf: No such file or directory
sed: can't read /etc/apache2/ports.conf: No such file or directory
adding rewrite rules to file: /etc/apache2/sites-available/default-ssl
adding cors rules to file: /etc/apache2/sites-available/default-ssl
Stopping web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 94.229.139.152 for ServerName
 ... waiting .
Starting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using 94.229.139.152 for ServerName
.

At this point Apache is running and continuing to serve the snakeoil certificate.

The config_ssl.sh script is carrying out actions on the following files:
/etc/httpd/conf/httpd.conf
/etc/apache2/sites-available/default
/etc/apache2/sites-available/default-ssl
/etc/apache2/ports.conf

Of these files /etc/apache2/sites-available/default-ssl is clearly the most relevant. However, I see no evidence here of that file being linked to /etc/apache2/sites-enabled, so without that step, this work is wasted.

site-enabled contains one file: /etc/apache2/sites-enabled/vhost-[IPADDRESS].conf

That is the file that makes reference to the default snakeoil certiifcate:

        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

The custom certificate has been successfully copied to /etc/httpd/ssl/certs folder, but is not referenced.

To try to fix this, I do the following:

• Delete vhost-[IPADDRESS].conf
• Symlink default-ssl into sites-enabled
• Restart apache

  # rm -f /etc/apache2/sites-enabled/vhost-*.conf                                                                                                                                                         
  # ln -s /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/
  # service apache2 restart
  Restarting web server: apache2apache2: Could not reliably determine the server's fully qualified domain name, using [IPADDRESS] for ServerName
   ... waiting apache2: Could not reliably determine the server's fully qualified domain name, using [IPADDRESS] for ServerName
  no listening sockets available, shutting down
  Unable to open logs
  Action 'start' failed.
  The Apache error log may have more information.
   failed!
  

That's two remaining problems to solve. Apache needs a ServerName, and listening sockets need to be defined.

The listening socket is the interesting one and relates to the errors in the log. Looking in config_ssl.sh, we can see where it tried and fail to manipulate the Listen IP/socket:

  sed -i -e "s/Listen .*:80/Listen $ip:80/g" /etc/apache2/ports.conf
  sed -i -e "s/Listen .*:443/Listen $ip:443/g" /etc/apache2/ports.conf

As the log entries say, ports.conf does not exist. How did that happen? I found the answer to that in /etc/init.d/cloud-early:

clean_ipalias_config() {
  # Old
  rm -f /etc/apache2/conf.d/ports.*.meta-data.conf
  rm -f /etc/apache2/sites-available/ipAlias*
  rm -f /etc/apache2/sites-enabled/ipAlias*
  rm -f /etc/apache2/conf.d/vhost*.conf
  rm -f /etc/apache2/ports.conf
  rm -f /etc/apache2/vhostexample.conf
  rm -f /etc/apache2/sites-available/default
  rm -f /etc/apache2/sites-available/default-ssl
  rm -f /etc/apache2/sites-enabled/default
  rm -f /etc/apache2/sites-enabled/default-ssl

  # New
  rm -f /etc/apache2/sites-enabled/vhost-*.conf
  rm -f /etc/apache2/sites-enabled/000-default

  rm -rf /etc/failure_config
}

The actions taken in /etc/init.d/cloud-early-config directly cause the Listen socket actions in config_ssl.sh to fail.
My next step is to bring back the ServerName and Listen directives:

# cat > /etc/apache2/conf.d/ssvm <<'EOF'
> ServerName server.mydomain.com
> Listen 0.0.0.0:80
> 
> <IfModule ssl_module>
> Listen 0.0.0.0:443
> </IfModule>
> 
> <IfModule mod_gnutls.c>
> Listen 0.0.0.0:443
> </IfModule>
> 
> # vim: syntax=apache ts=4 sw=4 sts=4 sr noet
> EOF
# 

Next I tweak config_ssl.sh to have it operate on this file rather than ports.conf. I also have it create the sites/enabled symlinks and remove the vhost file:

--- config_ssl.sh       2017-06-06 21:51:12.000000000 +0000
+++ config_ssl.sh.new   2017-06-18 11:05:38.197646907 +0000
@@ -47,9 +47,9 @@
   cp -f /etc/apache2/sites-available/default-ssl.orig /etc/apache2/sites-available/default-ssl
   sed -i -e "s/<VirtualHost.*>/<VirtualHost $ip:80>/" /etc/apache2/sites-available/default
   sed -i -e "s/<VirtualHost.*>/<VirtualHost $ip:443>/" /etc/apache2/sites-available/default-ssl
-  sed -i -e "s/Listen .*:80/Listen $ip:80/g" /etc/apache2/ports.conf
-  sed -i -e "s/Listen .*:443/Listen $ip:443/g" /etc/apache2/ports.conf
-  sed -i -e "s/NameVirtualHost .*:80/NameVirtualHost $ip:80/g" /etc/apache2/ports.conf
+  sed -i -e "s/Listen .*:80/Listen $ip:80/g" /etc/apache2/conf.d/ssvm
+  sed -i -e "s/Listen .*:443/Listen $ip:443/g" /etc/apache2/conf.d/ssvm
+  sed -i -e "s/NameVirtualHost .*:80/NameVirtualHost $ip:80/g" /etc/apache2/conf.d/ssvm
   sed -i  's/ssl-cert-snakeoil.key/cert_apache.key/' /etc/apache2/sites-available/default-ssl
   sed -i  's/ssl-cert-snakeoil.pem/cert_apache.crt/' /etc/apache2/sites-available/default-ssl
   sed -i  's/SSLProtocol.*$/SSLProtocol all -SSLv2 -SSLv3/' /etc/apache2/sites-available/default-ssl
@@ -80,7 +80,8 @@
         sed -i -e "s/<\/VirtualHost>/Header always set Access-Control-Allow-Headers \"x-requested-with, Content-Type, origin, authorization, accept, client-security-token, x-signature, x-metadata, x-expires\" \n&/" $SSL_FILE
     fi
   fi
-
+  ln -s /etc/apache2/sites-available/default /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/
+  rm -f /etc/apache2/sites-enabled/vhost-*
 }
 
 copy_certs() {

I also found that my changes to config_ssl.sh would be overwritten at reboot, so I made it immutable with "chattr +i".

What I end up with is a system working as expected but relying on a really dirty hack. I'm afraid I don't know enough about the conflicting needs of those that wrote these conflicting scripts to see a clean fix.