Uploaded image for project: 'Guacamole'
  1. Guacamole
  2. GUACAMOLE-194

Double free() in guac_common_ssh_destroy_user()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.10-incubating
    • Fix Version/s: 0.9.13-incubating
    • Component/s: RDP, VNC
    • Labels:
      None
    • Environment:
      guacd docker container on ubuntu and debian

      Description

      We are heavy users of the RDP feature with SSH/SFTP for file downloads and uploads. We are seeing frequent guacd crashes, even the first session log off can cause a crash.

      After a few sessions (Sometimes hundreds, often thousands) the container becomes unstable, causes very high system load and no new connections via guacamole are possible. We typically also see the virtual docker Ethernet adapter enter the disabled state.

      void guac_common_ssh_destroy_user(guac_common_ssh_user* user) {
      
          /* Free private key, if present */
          if (user->private_key != NULL)
              guac_common_ssh_key_free(user->private_key);
      
          /* Free all other data */
          free(user->password);
          free(user->username);
          free(user);
      
      }
      
      1. guacd-stack trace.txt
        48 kB
        Robin
      2. GATEWAY1-logs.zip
        1.42 MB
        Robin
      3. GATEWAY2-logs.zip
        25 kB
        Robin

        Activity

        Hide
        mike.jumper Michael Jumper added a comment -

        Robin - I believe this should be fixed on master. Please let us know if you still encounter the double free() with the latest.

        Show
        mike.jumper Michael Jumper added a comment - Robin - I believe this should be fixed on master. Please let us know if you still encounter the double free() with the latest.
        Hide
        mike.jumper Michael Jumper added a comment -

        Looks like both VNC and RDP will double-free the SFTP user if client initialization fails: first within the init routine when it fails, and then again when the client itself is being cleaned up.

        Show
        mike.jumper Michael Jumper added a comment - Looks like both VNC and RDP will double-free the SFTP user if client initialization fails: first within the init routine when it fails, and then again when the client itself is being cleaned up.
        Hide
        mrbabbage Robin added a comment -

        We are using RDP for all sessions, with the SFTP option enabled.
        On some of our gateway servers, these segfaults ARE survivable for other connections.

        However, in this instance, it seemed that every thread started crashing with the same error.
        This is a Debian gateway VM: 24GB RAM, 2 cores, 40GB HDD - we'll call it GATEWAY1
        The logs show the server being quiet from 00:00 then threads crashing repeatedly from around 01:19 until 03:10 when the docker network port entered the disabled state.
        Please see attached in GATEWAY1-logs.zip:

        • combined guacd & guacamole logs (from SIEM - apologies for reverse order)
        • /var/log/messages file showing summary of segfaults & the error where docker networking gets disabled

        Another gateway (on Ubuntu - let's say GATEWAY2) has been up for 10 days and seen 454 instances of one of these segfaults:

        • free(): invalid pointer: 0x00007f80d8004000
        • double free or corruption (fasttop): 0x00007f80d800ae10
        • double free or corruption (out): 0x00007f80d80008d0
          But in this instance, the container has survived and continues to serve connections.

        However, we do see instances of repeated user disconnections on this gateway which don't correlate with the timings of the segfaults, so I'm unsure whether these are truly related.
        I have attached guacamole and guacd logs of an instance where a single user (mdsl_user1) attempted to connect but was repeatedly disconnected.
        It looks to us like "Error waiting for file descriptor." then "User is not responding." then ERRINFO_UNKNOWN 0x0000000C: Unknown error.
        Subsequently, it seems like the user is getting disconnected by himself: "Another user connected to the server, forcing the disconnection of the current connection."
        We have attached the relevant logs as GATEWAY2-logs.zip
        We are not experienced in unpicking these logs so are not sure whether there is more we could be investigating, or whether we should raise a separate item for this.

        Show
        mrbabbage Robin added a comment - We are using RDP for all sessions, with the SFTP option enabled. On some of our gateway servers, these segfaults ARE survivable for other connections. However, in this instance, it seemed that every thread started crashing with the same error. This is a Debian gateway VM: 24GB RAM, 2 cores, 40GB HDD - we'll call it GATEWAY1 The logs show the server being quiet from 00:00 then threads crashing repeatedly from around 01:19 until 03:10 when the docker network port entered the disabled state. Please see attached in GATEWAY1-logs.zip: combined guacd & guacamole logs (from SIEM - apologies for reverse order) /var/log/messages file showing summary of segfaults & the error where docker networking gets disabled Another gateway (on Ubuntu - let's say GATEWAY2) has been up for 10 days and seen 454 instances of one of these segfaults: free(): invalid pointer: 0x00007f80d8004000 double free or corruption (fasttop): 0x00007f80d800ae10 double free or corruption (out): 0x00007f80d80008d0 But in this instance, the container has survived and continues to serve connections. However, we do see instances of repeated user disconnections on this gateway which don't correlate with the timings of the segfaults, so I'm unsure whether these are truly related. I have attached guacamole and guacd logs of an instance where a single user (mdsl_user1) attempted to connect but was repeatedly disconnected. It looks to us like "Error waiting for file descriptor." then "User is not responding." then ERRINFO_UNKNOWN 0x0000000C: Unknown error. Subsequently, it seems like the user is getting disconnected by himself: "Another user connected to the server, forcing the disconnection of the current connection." We have attached the relevant logs as GATEWAY2-logs.zip We are not experienced in unpicking these logs so are not sure whether there is more we could be investigating, or whether we should raise a separate item for this.
        Hide
        mike.jumper Michael Jumper added a comment -

        We are heavy users of the RDP feature with SSH/SFTP for file downloads and uploads. We are seeing frequent guacd crashes, even the first session log off can cause a crash.

        In this particular instance, the segfault due to the invalid free() is occurring when the connection is being cleaned up and has already been disconnected for other reasons. It needs to be fixed, for sure, but it should have zero affect on connection stability as each connection is isolated to its own process.

        Do you have the logs from guacd when this occurs?

        After a few sessions (Sometimes hundreds, often thousands) the container becomes unstable, causes very high system load and no new connections via guacamole are possible.

        Are all these sessions RDP? What are the specifications of your server?

        We typically also see the virtual docker Ethernet adapter enter the disabled state.

        This is troubling. I've never heard of Docker's internal networking simply dying.

        Show
        mike.jumper Michael Jumper added a comment - We are heavy users of the RDP feature with SSH/SFTP for file downloads and uploads. We are seeing frequent guacd crashes, even the first session log off can cause a crash. In this particular instance, the segfault due to the invalid free() is occurring when the connection is being cleaned up and has already been disconnected for other reasons. It needs to be fixed, for sure, but it should have zero affect on connection stability as each connection is isolated to its own process. Do you have the logs from guacd when this occurs? After a few sessions (Sometimes hundreds, often thousands) the container becomes unstable, causes very high system load and no new connections via guacamole are possible. Are all these sessions RDP? What are the specifications of your server? We typically also see the virtual docker Ethernet adapter enter the disabled state. This is troubling. I've never heard of Docker's internal networking simply dying.

          People

          • Assignee:
            mike.jumper Michael Jumper
            Reporter:
            mrbabbage Robin
          • Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development