Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6
    • Component/s: PHP - Library
    • Labels:
      None
    • Environment:

      Fedora 8, 64bit, php 5.2.4

    • Patch Info:
      Patch Available

      Description

      I'm working with the Hive thrift service and I get the following exception when I execute a SELECT statement:

      PHP Fatal error: Uncaught exception 'TException' with message 'TSocket: timed out reading 4 bytes from localhost:10000' in /root/leap/dev/servers/hive/thriftroot/
      transport/TSocket.php:228
      Stack trace:
      #0 /root/leap/dev/servers/hive/thriftroot/protocol/TBinaryProtocol.php(292): TSocket->readAll(4)
      #1 /root/leap/dev/servers/hive/thriftroot/protocol/TBinaryProtocol.php(184): TBinaryProtocol->readI32(NULL)
      #2 /root/leap/dev/servers/hive/thriftroot/packages/hive_service/ThriftHive.php(59): TBinaryProtocol->readMessageBegin(NULL, 0, 0)
      #3 /root/leap/dev/servers/hive/thriftroot/packages/hive_service/ThriftHive.php(28): ThriftHiveClient->recv_execute()
      #4 /root/leap/dev/servers/hive/testscript.php(30): ThriftHiveClient->execute('SELECT num FROM...')
      #5

      {main}

      thrown in /root/leap/dev/servers/hive/thriftroot/transport/TSocket.php on line 228

      The script I'm using to cause this is:

      ======================================
      #!/usr/bin/php
      <?php

      $GLOBALS['THRIFT_ROOT'] = 'thriftroot/';

      require_once $GLOBALS['THRIFT_ROOT'] . 'packages/hive_service/ThriftHive.php';
      require_once $GLOBALS['THRIFT_ROOT'] . 'transport/TSocket.php';
      require_once $GLOBALS['THRIFT_ROOT'] . 'protocol/TBinaryProtocol.php';

      $transport = new TSocket('localhost', 10000);
      $protocol = new TBinaryProtocol($transport);
      $client = new ThriftHiveClient($protocol);
      $transport->open();

      try

      { $client->execute('DROP TABLE testOverThrift'); }

      catch (Exception $e)

      { error_log("Got exception while trying to drop table: " . $e->getMessage()); }

      $client->execute('CREATE TABLE testOverThrift (num int)');

      $client->execute('LOAD DATA LOCAL
      INPATH "/root/testdata.txt"
      INTO TABLE testOverThrift');

      $client->execute('SELECT num FROM testOverThrift WHERE num < 5');

      $result = $client->fetchAll();

      var_dump($result);
      ==================================

      I have a patch that fixes the problem that I'll attach. It looks like fetchAll() doesn't know the difference between a timeout and a blocking port.

      1. TSocket.php.diff
        0.8 kB
        Gary Richardson
      2. TSocket.php.patch
        1.0 kB
        Takuya Watabe
      3. thrift-347-0.5.0.txt
        2 kB
        Tyler Hobbs

        Activity

        Hide
        Bryan Duxbury added a comment -

        Can any php folks chime in on whether this patch should be committed?

        Show
        Bryan Duxbury added a comment - Can any php folks chime in on whether this patch should be committed?
        Hide
        Mark Slee added a comment -

        Hmm, seems okay. But how are these sockets getting into blocking mode? Presumably this requires a call to stream_set_blocking(), which we never do in TSocket.php.

        I'm not sure if the rest of the code recovers properly in this case. When you debug log on your system, do you actually get the socket call coming back with the 'blocked' key set?

        My concern is that with this change, fetchAll could potentially infinite loop, continuously receiving empty string back. That probably won't happen, but it does seem like it could use up more than the timeout, due to teh while(true). I'm not sure how the PHP streams lib handles timeouts in blocking mode. As long as you can verify it still handles timeouts appropriately, then consider this diff approved.

        But yeah, my main question is how you're getting client sockets in blocking mode here?

        Show
        Mark Slee added a comment - Hmm, seems okay. But how are these sockets getting into blocking mode? Presumably this requires a call to stream_set_blocking(), which we never do in TSocket.php. I'm not sure if the rest of the code recovers properly in this case. When you debug log on your system, do you actually get the socket call coming back with the 'blocked' key set? My concern is that with this change, fetchAll could potentially infinite loop, continuously receiving empty string back. That probably won't happen, but it does seem like it could use up more than the timeout, due to teh while(true). I'm not sure how the PHP streams lib handles timeouts in blocking mode. As long as you can verify it still handles timeouts appropriately, then consider this diff approved. But yeah, my main question is how you're getting client sockets in blocking mode here?
        Hide
        Gary Richardson added a comment -

        I've moved on from the code, so it's a bit hazy now. From what I remember, yes, the socket call came with with 'blocked'.

        Show
        Gary Richardson added a comment - I've moved on from the code, so it's a bit hazy now. From what I remember, yes, the socket call came with with 'blocked'.
        Hide
        Wade Arnold added a comment -

        I ran into this issue intermittently with calls against Cassandra. I am going to try and hunt down the issue but it seems to happen after no requests have been made for a period and then the first subsequent request creates the error. If you follow this code and just run it randomly over and over it will fail eventually fail.

        http://wiki.apache.org/cassandra/ClientExamples

        Show
        Wade Arnold added a comment - I ran into this issue intermittently with calls against Cassandra. I am going to try and hunt down the issue but it seems to happen after no requests have been made for a period and then the first subsequent request creates the error. If you follow this code and just run it randomly over and over it will fail eventually fail. http://wiki.apache.org/cassandra/ClientExamples
        Hide
        Chris Goffinet added a comment -

        Mark,

        By default when calling this in TSocket:

        $this->handle_ = @fsockopen($this->host_,
        $this->port_,
        $errno,
        $errstr,
        $this->sendTimeout_/1000.0);

        Here is an excerpt from php.net:

        http://us3.php.net/fsockopen

        The socket will by default be opened in blocking mode. You can switch it to non-blocking mode by using stream_set_blocking().

        Show
        Chris Goffinet added a comment - Mark, By default when calling this in TSocket: $this->handle_ = @fsockopen($this->host_, $this->port_, $errno, $errstr, $this->sendTimeout_/1000.0); Here is an excerpt from php.net: http://us3.php.net/fsockopen The socket will by default be opened in blocking mode. You can switch it to non-blocking mode by using stream_set_blocking().
        Hide
        Chris Goffinet added a comment -

        Wade Arnold,

        Can you provide a test case for this so I can reproduce? I haven't run into this issue.

        Show
        Chris Goffinet added a comment - Wade Arnold, Can you provide a test case for this so I can reproduce? I haven't run into this issue.
        Hide
        Chris Goffinet added a comment -

        I tested out this patch, as one of our developers came across this issue. We were using FramedTransport in PHP, and ran into the 4 bytes error. Applying this patch worked for us.

        Show
        Chris Goffinet added a comment - I tested out this patch, as one of our developers came across this issue. We were using FramedTransport in PHP, and ran into the 4 bytes error. Applying this patch worked for us.
        Hide
        David Reiss added a comment -

        So, I'm looking at http://us3.php.net/manual/en/function.stream-get-meta-data.php. The "blocked" field is really poorly named. It indicates whether the stream in in block*ing* mode, which should always be true. Are you seeing this coming up as false at any time? Also, could someone clarify in the description what the expected behavior is?

        Show
        David Reiss added a comment - So, I'm looking at http://us3.php.net/manual/en/function.stream-get-meta-data.php . The "blocked" field is really poorly named. It indicates whether the stream in in block*ing* mode, which should always be true. Are you seeing this coming up as false at any time? Also, could someone clarify in the description what the expected behavior is?
        Hide
        Mike Roberts added a comment -

        I've tested this patch. I was using regular (not framed) transport against Cassandra. This patch fixed the 4 bytes error I would get after large data loads.

        Show
        Mike Roberts added a comment - I've tested this patch. I was using regular (not framed) transport against Cassandra. This patch fixed the 4 bytes error I would get after large data loads.
        Hide
        Nathan Marz added a comment -

        I ran into a very similar issue, but applying this patch did not help. After applying the patch, the client hangs forever and never times out.

        The client only times out when there is a large object returned - when the returned object is relatively small the socket works fine.

        Show
        Nathan Marz added a comment - I ran into a very similar issue, but applying this patch did not help. After applying the patch, the client hangs forever and never times out. The client only times out when there is a large object returned - when the returned object is relatively small the socket works fine.
        Hide
        Kevin Bombino added a comment - - edited

        +1 for the patch. Solved TSocket timeout issues on CentOS 5.3, PHP 5.2, talking to a Ruby Thrift server running on localhost. I have no idea how it solves the problem, or what was causing the problem, but we've been using this patch in production for a few months with no further errors.

        EDIT: upon further review, while this patch did prevent errors from being shown, it did not address the root issue. Sorry for adding confusion to the matter.

        Show
        Kevin Bombino added a comment - - edited +1 for the patch. Solved TSocket timeout issues on CentOS 5.3, PHP 5.2, talking to a Ruby Thrift server running on localhost. I have no idea how it solves the problem, or what was causing the problem, but we've been using this patch in production for a few months with no further errors. EDIT: upon further review, while this patch did prevent errors from being shown, it did not address the root issue. Sorry for adding confusion to the matter.
        Hide
        Eric Lam added a comment -

        One of the reason is "function overloading" feature of mbstring that changes the behavior of strlen.

        Disable the mbstring.func_overload in php.ini solved this issue for me.

        It may not be the only cause though.

        Show
        Eric Lam added a comment - One of the reason is "function overloading" feature of mbstring that changes the behavior of strlen. Disable the mbstring.func_overload in php.ini solved this issue for me. It may not be the only cause though.
        Hide
        Mike Lerch added a comment -

        Same issue here as Nathan Marz. Everything was working great until I tried to retrieve a larger bit of data (about 5kb). Before applying the patch, the request would fail with the 'TSocket: timed out reading 4 bytes from' message. After the patch, the script hangs indefinitely using 100% CPU.

        Tried TBuffered, TFramed, and standard TSocket connections, all with the same result.

        Does not happen with other client libraries, only PHP, which tells me it isn't the server itself.

        Any advice?

        Show
        Mike Lerch added a comment - Same issue here as Nathan Marz. Everything was working great until I tried to retrieve a larger bit of data (about 5kb). Before applying the patch, the request would fail with the 'TSocket: timed out reading 4 bytes from' message. After the patch, the script hangs indefinitely using 100% CPU. Tried TBuffered, TFramed, and standard TSocket connections, all with the same result. Does not happen with other client libraries, only PHP, which tells me it isn't the server itself. Any advice?
        Hide
        Mike Lerch added a comment -

        Wanted to follow-up on what we found. Somehow a null value was stored in one of the Cassandra values. Removing that fixed our issue.

        Show
        Mike Lerch added a comment - Wanted to follow-up on what we found. Somehow a null value was stored in one of the Cassandra values. Removing that fixed our issue.
        Hide
        Ken Sandney added a comment -

        I am still having the same issue with phpcassa http://github.com/hoan/phpcassa , any one could share your solution here?

        Show
        Ken Sandney added a comment - I am still having the same issue with phpcassa http://github.com/hoan/phpcassa , any one could share your solution here?
        Hide
        Tupshin Harper added a comment -

        Kevin: Have you tried combining this patch with the two patches attached to https://issues.apache.org/jira/browse/THRIFT-638
        ?

        Show
        Tupshin Harper added a comment - Kevin: Have you tried combining this patch with the two patches attached to https://issues.apache.org/jira/browse/THRIFT-638 ?
        Hide
        Loreto Parisi added a comment -

        There is a similar problem in socket write, described here:
        https://issues.apache.org/jira/browse/THRIFT-826

        Show
        Loreto Parisi added a comment - There is a similar problem in socket write, described here: https://issues.apache.org/jira/browse/THRIFT-826
        Hide
        Takuya Watabe added a comment -

        I found similar problem on Hive (thirft v0.50), and patch on this page was not enough for my case.
        This is the patch for my case.

        By using this patch, my php code below is working.

        // Set up the transport/protocol/client
        $transport = new TSocket('localhost', 10000);
        $protocol = new TBinaryProtocol($transport);
        $client = new ThriftHiveClient($protocol);
        $transport->open();
        $client->execute('SELECT * from sample order by id desc');
        var_dump($client->fetchAll());
        $transport->close();

        Show
        Takuya Watabe added a comment - I found similar problem on Hive (thirft v0.50), and patch on this page was not enough for my case. This is the patch for my case. By using this patch, my php code below is working. // Set up the transport/protocol/client $transport = new TSocket('localhost', 10000); $protocol = new TBinaryProtocol($transport); $client = new ThriftHiveClient($protocol); $transport->open(); $client->execute('SELECT * from sample order by id desc'); var_dump($client->fetchAll()); $transport->close();
        Hide
        Tyler Hobbs added a comment - - edited

        The patch I've attached seems to resolve this issue with TFramedTransport and TBinaryProtocolAccelerated using thrift_protocol.so. It's similar to Gary's original patch, and is against the version in Thrift 0.5.0. I do not know how well this works with other transports and protocols.

        phpcassa (https://github.com/thobbs/phpcassa) ships with this patch applied.

        Show
        Tyler Hobbs added a comment - - edited The patch I've attached seems to resolve this issue with TFramedTransport and TBinaryProtocolAccelerated using thrift_protocol.so. It's similar to Gary's original patch, and is against the version in Thrift 0.5.0. I do not know how well this works with other transports and protocols. phpcassa ( https://github.com/thobbs/phpcassa ) ships with this patch applied.
        Hide
        T Jake Luciani added a comment -

        Thanks for the patch Tyler!

        We'll wait for any feedback than add this for inclusion in 0.6 if left uncontested.

        Show
        T Jake Luciani added a comment - Thanks for the patch Tyler! We'll wait for any feedback than add this for inclusion in 0.6 if left uncontested.
        Hide
        Bryan Duxbury added a comment -

        I just committed Tyler's patch. Thanks for knocking this one out!

        Show
        Bryan Duxbury added a comment - I just committed Tyler's patch. Thanks for knocking this one out!
        Hide
        Ruben de Vries added a comment -

        afaik this issue is back in hive-0.8.1, probally since regenerating the PHP libs with thrift reintroduces the bug

        Show
        Ruben de Vries added a comment - afaik this issue is back in hive-0.8.1, probally since regenerating the PHP libs with thrift reintroduces the bug
        Hide
        Ruben de Vries added a comment -

        nvm, ignore that, wrong issue tracker

        Show
        Ruben de Vries added a comment - nvm, ignore that, wrong issue tracker

          People

          • Assignee:
            Tyler Hobbs
            Reporter:
            Gary Richardson
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development