Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6
    • Component/s: PHP - Library
    • Labels:
      None
    • Environment:

      Fedora 8, 64bit, php 5.2.4

    • Patch Info:
      Patch Available

      Description

      I'm working with the Hive thrift service and I get the following exception when I execute a SELECT statement:

      PHP Fatal error: Uncaught exception 'TException' with message 'TSocket: timed out reading 4 bytes from localhost:10000' in /root/leap/dev/servers/hive/thriftroot/
      transport/TSocket.php:228
      Stack trace:
      #0 /root/leap/dev/servers/hive/thriftroot/protocol/TBinaryProtocol.php(292): TSocket->readAll(4)
      #1 /root/leap/dev/servers/hive/thriftroot/protocol/TBinaryProtocol.php(184): TBinaryProtocol->readI32(NULL)
      #2 /root/leap/dev/servers/hive/thriftroot/packages/hive_service/ThriftHive.php(59): TBinaryProtocol->readMessageBegin(NULL, 0, 0)
      #3 /root/leap/dev/servers/hive/thriftroot/packages/hive_service/ThriftHive.php(28): ThriftHiveClient->recv_execute()
      #4 /root/leap/dev/servers/hive/testscript.php(30): ThriftHiveClient->execute('SELECT num FROM...')
      #5

      {main}

      thrown in /root/leap/dev/servers/hive/thriftroot/transport/TSocket.php on line 228

      The script I'm using to cause this is:

      ======================================
      #!/usr/bin/php
      <?php

      $GLOBALS['THRIFT_ROOT'] = 'thriftroot/';

      require_once $GLOBALS['THRIFT_ROOT'] . 'packages/hive_service/ThriftHive.php';
      require_once $GLOBALS['THRIFT_ROOT'] . 'transport/TSocket.php';
      require_once $GLOBALS['THRIFT_ROOT'] . 'protocol/TBinaryProtocol.php';

      $transport = new TSocket('localhost', 10000);
      $protocol = new TBinaryProtocol($transport);
      $client = new ThriftHiveClient($protocol);
      $transport->open();

      try

      { $client->execute('DROP TABLE testOverThrift'); }

      catch (Exception $e)

      { error_log("Got exception while trying to drop table: " . $e->getMessage()); }

      $client->execute('CREATE TABLE testOverThrift (num int)');

      $client->execute('LOAD DATA LOCAL
      INPATH "/root/testdata.txt"
      INTO TABLE testOverThrift');

      $client->execute('SELECT num FROM testOverThrift WHERE num < 5');

      $result = $client->fetchAll();

      var_dump($result);
      ==================================

      I have a patch that fixes the problem that I'll attach. It looks like fetchAll() doesn't know the difference between a timeout and a blocking port.

      1. thrift-347-0.5.0.txt
        2 kB
        Tyler Hobbs
      2. TSocket.php.diff
        0.8 kB
        Gary Richardson
      3. TSocket.php.patch
        1.0 kB
        Takuya Watabe

        Issue Links

          Activity

          Hide
          Ruben de Vries added a comment -

          nvm, ignore that, wrong issue tracker

          Show
          Ruben de Vries added a comment - nvm, ignore that, wrong issue tracker
          Hide
          Ruben de Vries added a comment -

          afaik this issue is back in hive-0.8.1, probally since regenerating the PHP libs with thrift reintroduces the bug

          Show
          Ruben de Vries added a comment - afaik this issue is back in hive-0.8.1, probally since regenerating the PHP libs with thrift reintroduces the bug
          Hide
          Bryan Duxbury added a comment -

          I just committed Tyler's patch. Thanks for knocking this one out!

          Show
          Bryan Duxbury added a comment - I just committed Tyler's patch. Thanks for knocking this one out!
          Hide
          T Jake Luciani added a comment -

          Thanks for the patch Tyler!

          We'll wait for any feedback than add this for inclusion in 0.6 if left uncontested.

          Show
          T Jake Luciani added a comment - Thanks for the patch Tyler! We'll wait for any feedback than add this for inclusion in 0.6 if left uncontested.
          Hide
          Tyler Hobbs added a comment - - edited

          The patch I've attached seems to resolve this issue with TFramedTransport and TBinaryProtocolAccelerated using thrift_protocol.so. It's similar to Gary's original patch, and is against the version in Thrift 0.5.0. I do not know how well this works with other transports and protocols.

          phpcassa (https://github.com/thobbs/phpcassa) ships with this patch applied.

          Show
          Tyler Hobbs added a comment - - edited The patch I've attached seems to resolve this issue with TFramedTransport and TBinaryProtocolAccelerated using thrift_protocol.so. It's similar to Gary's original patch, and is against the version in Thrift 0.5.0. I do not know how well this works with other transports and protocols. phpcassa ( https://github.com/thobbs/phpcassa ) ships with this patch applied.
          Hide
          Takuya Watabe added a comment -

          I found similar problem on Hive (thirft v0.50), and patch on this page was not enough for my case.
          This is the patch for my case.

          By using this patch, my php code below is working.

          // Set up the transport/protocol/client
          $transport = new TSocket('localhost', 10000);
          $protocol = new TBinaryProtocol($transport);
          $client = new ThriftHiveClient($protocol);
          $transport->open();
          $client->execute('SELECT * from sample order by id desc');
          var_dump($client->fetchAll());
          $transport->close();

          Show
          Takuya Watabe added a comment - I found similar problem on Hive (thirft v0.50), and patch on this page was not enough for my case. This is the patch for my case. By using this patch, my php code below is working. // Set up the transport/protocol/client $transport = new TSocket('localhost', 10000); $protocol = new TBinaryProtocol($transport); $client = new ThriftHiveClient($protocol); $transport->open(); $client->execute('SELECT * from sample order by id desc'); var_dump($client->fetchAll()); $transport->close();
          Hide
          Loreto Parisi added a comment -

          There is a similar problem in socket write, described here:
          https://issues.apache.org/jira/browse/THRIFT-826

          Show
          Loreto Parisi added a comment - There is a similar problem in socket write, described here: https://issues.apache.org/jira/browse/THRIFT-826
          Hide
          Tupshin Harper added a comment -

          Kevin: Have you tried combining this patch with the two patches attached to https://issues.apache.org/jira/browse/THRIFT-638
          ?

          Show
          Tupshin Harper added a comment - Kevin: Have you tried combining this patch with the two patches attached to https://issues.apache.org/jira/browse/THRIFT-638 ?
          Hide
          Ken Sandney added a comment -

          I am still having the same issue with phpcassa http://github.com/hoan/phpcassa , any one could share your solution here?

          Show
          Ken Sandney added a comment - I am still having the same issue with phpcassa http://github.com/hoan/phpcassa , any one could share your solution here?
          Hide
          Mike Lerch added a comment -

          Wanted to follow-up on what we found. Somehow a null value was stored in one of the Cassandra values. Removing that fixed our issue.

          Show
          Mike Lerch added a comment - Wanted to follow-up on what we found. Somehow a null value was stored in one of the Cassandra values. Removing that fixed our issue.
          Hide
          Mike Lerch added a comment -

          Same issue here as Nathan Marz. Everything was working great until I tried to retrieve a larger bit of data (about 5kb). Before applying the patch, the request would fail with the 'TSocket: timed out reading 4 bytes from' message. After the patch, the script hangs indefinitely using 100% CPU.

          Tried TBuffered, TFramed, and standard TSocket connections, all with the same result.

          Does not happen with other client libraries, only PHP, which tells me it isn't the server itself.

          Any advice?

          Show
          Mike Lerch added a comment - Same issue here as Nathan Marz. Everything was working great until I tried to retrieve a larger bit of data (about 5kb). Before applying the patch, the request would fail with the 'TSocket: timed out reading 4 bytes from' message. After the patch, the script hangs indefinitely using 100% CPU. Tried TBuffered, TFramed, and standard TSocket connections, all with the same result. Does not happen with other client libraries, only PHP, which tells me it isn't the server itself. Any advice?
          Hide
          Eric Lam added a comment -

          One of the reason is "function overloading" feature of mbstring that changes the behavior of strlen.

          Disable the mbstring.func_overload in php.ini solved this issue for me.

          It may not be the only cause though.

          Show
          Eric Lam added a comment - One of the reason is "function overloading" feature of mbstring that changes the behavior of strlen. Disable the mbstring.func_overload in php.ini solved this issue for me. It may not be the only cause though.
          Hide
          Kevin Bombino added a comment - - edited

          +1 for the patch. Solved TSocket timeout issues on CentOS 5.3, PHP 5.2, talking to a Ruby Thrift server running on localhost. I have no idea how it solves the problem, or what was causing the problem, but we've been using this patch in production for a few months with no further errors.

          EDIT: upon further review, while this patch did prevent errors from being shown, it did not address the root issue. Sorry for adding confusion to the matter.

          Show
          Kevin Bombino added a comment - - edited +1 for the patch. Solved TSocket timeout issues on CentOS 5.3, PHP 5.2, talking to a Ruby Thrift server running on localhost. I have no idea how it solves the problem, or what was causing the problem, but we've been using this patch in production for a few months with no further errors. EDIT: upon further review, while this patch did prevent errors from being shown, it did not address the root issue. Sorry for adding confusion to the matter.
          Hide
          Nathan Marz added a comment -

          I ran into a very similar issue, but applying this patch did not help. After applying the patch, the client hangs forever and never times out.

          The client only times out when there is a large object returned - when the returned object is relatively small the socket works fine.

          Show
          Nathan Marz added a comment - I ran into a very similar issue, but applying this patch did not help. After applying the patch, the client hangs forever and never times out. The client only times out when there is a large object returned - when the returned object is relatively small the socket works fine.
          Hide
          Mike Roberts added a comment -

          I've tested this patch. I was using regular (not framed) transport against Cassandra. This patch fixed the 4 bytes error I would get after large data loads.

          Show
          Mike Roberts added a comment - I've tested this patch. I was using regular (not framed) transport against Cassandra. This patch fixed the 4 bytes error I would get after large data loads.
          Hide
          David Reiss added a comment -

          So, I'm looking at http://us3.php.net/manual/en/function.stream-get-meta-data.php. The "blocked" field is really poorly named. It indicates whether the stream in in block*ing* mode, which should always be true. Are you seeing this coming up as false at any time? Also, could someone clarify in the description what the expected behavior is?

          Show
          David Reiss added a comment - So, I'm looking at http://us3.php.net/manual/en/function.stream-get-meta-data.php . The "blocked" field is really poorly named. It indicates whether the stream in in block*ing* mode, which should always be true. Are you seeing this coming up as false at any time? Also, could someone clarify in the description what the expected behavior is?
          Hide
          Chris Goffinet added a comment -

          I tested out this patch, as one of our developers came across this issue. We were using FramedTransport in PHP, and ran into the 4 bytes error. Applying this patch worked for us.

          Show
          Chris Goffinet added a comment - I tested out this patch, as one of our developers came across this issue. We were using FramedTransport in PHP, and ran into the 4 bytes error. Applying this patch worked for us.
          Hide
          Chris Goffinet added a comment -

          Wade Arnold,

          Can you provide a test case for this so I can reproduce? I haven't run into this issue.

          Show
          Chris Goffinet added a comment - Wade Arnold, Can you provide a test case for this so I can reproduce? I haven't run into this issue.
          Hide
          Chris Goffinet added a comment -

          Mark,

          By default when calling this in TSocket:

          $this->handle_ = @fsockopen($this->host_,
          $this->port_,
          $errno,
          $errstr,
          $this->sendTimeout_/1000.0);

          Here is an excerpt from php.net:

          http://us3.php.net/fsockopen

          The socket will by default be opened in blocking mode. You can switch it to non-blocking mode by using stream_set_blocking().

          Show
          Chris Goffinet added a comment - Mark, By default when calling this in TSocket: $this->handle_ = @fsockopen($this->host_, $this->port_, $errno, $errstr, $this->sendTimeout_/1000.0); Here is an excerpt from php.net: http://us3.php.net/fsockopen The socket will by default be opened in blocking mode. You can switch it to non-blocking mode by using stream_set_blocking().
          Hide
          Wade Arnold added a comment -

          I ran into this issue intermittently with calls against Cassandra. I am going to try and hunt down the issue but it seems to happen after no requests have been made for a period and then the first subsequent request creates the error. If you follow this code and just run it randomly over and over it will fail eventually fail.

          http://wiki.apache.org/cassandra/ClientExamples

          Show
          Wade Arnold added a comment - I ran into this issue intermittently with calls against Cassandra. I am going to try and hunt down the issue but it seems to happen after no requests have been made for a period and then the first subsequent request creates the error. If you follow this code and just run it randomly over and over it will fail eventually fail. http://wiki.apache.org/cassandra/ClientExamples
          Hide
          Gary Richardson added a comment -

          I've moved on from the code, so it's a bit hazy now. From what I remember, yes, the socket call came with with 'blocked'.

          Show
          Gary Richardson added a comment - I've moved on from the code, so it's a bit hazy now. From what I remember, yes, the socket call came with with 'blocked'.
          Hide
          Mark Slee added a comment -

          Hmm, seems okay. But how are these sockets getting into blocking mode? Presumably this requires a call to stream_set_blocking(), which we never do in TSocket.php.

          I'm not sure if the rest of the code recovers properly in this case. When you debug log on your system, do you actually get the socket call coming back with the 'blocked' key set?

          My concern is that with this change, fetchAll could potentially infinite loop, continuously receiving empty string back. That probably won't happen, but it does seem like it could use up more than the timeout, due to teh while(true). I'm not sure how the PHP streams lib handles timeouts in blocking mode. As long as you can verify it still handles timeouts appropriately, then consider this diff approved.

          But yeah, my main question is how you're getting client sockets in blocking mode here?

          Show
          Mark Slee added a comment - Hmm, seems okay. But how are these sockets getting into blocking mode? Presumably this requires a call to stream_set_blocking(), which we never do in TSocket.php. I'm not sure if the rest of the code recovers properly in this case. When you debug log on your system, do you actually get the socket call coming back with the 'blocked' key set? My concern is that with this change, fetchAll could potentially infinite loop, continuously receiving empty string back. That probably won't happen, but it does seem like it could use up more than the timeout, due to teh while(true). I'm not sure how the PHP streams lib handles timeouts in blocking mode. As long as you can verify it still handles timeouts appropriately, then consider this diff approved. But yeah, my main question is how you're getting client sockets in blocking mode here?
          Hide
          Bryan Duxbury added a comment -

          Can any php folks chime in on whether this patch should be committed?

          Show
          Bryan Duxbury added a comment - Can any php folks chime in on whether this patch should be committed?

            People

            • Assignee:
              Tyler Hobbs
              Reporter:
              Gary Richardson
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development