[THRIFT-2933] v0.9.2: doubles encoded in node with compact protocol cannot be decoded by python

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2
• Fix Version/s: 0.9.3
• Component/s: Node.js - Library
• Assignee: Randy Abernethy
• Reporter: Dan Heller
• Created: 07/Jan/15 23:18
• Updated: 21/Jul/15 02:21
• Resolved: 02/Feb/15 12:48

Description

I've noticed that with Thrift v0.9.2, using the Compact protocol, I can't encode doubles in node and decode them in python. It looks like in python, we're decoding little-endian for compact and big-endian for binary, and in node, writing big-endian in all cases (therefore, Binary works fine). I've attached a test program to show this behavior. I have not found the thrift binary spec, so I'm not sure which side is wrong.

[~/thrift-double]$ thrift --version
Thrift version 0.9.2
[~/thrift-double]$ grep -i thrift package.json 
    "thrift": "0.9.2"
    "compile": "thrift --gen js:node --gen py test.thrift"
[~/thrift-double]$ pip show thrift
---
Name: thrift
Version: 0.9.2
Location: /Library/Python/2.7/site-packages
Requires: 
[~/thrift-double]$ npm run compile

> double-decode-bug@0.0.1 compile /Users/dh/thrift-double
> thrift --gen js:node --gen py test.thrift

[~/thrift-double]$ node test.js binary | python test.py binary
Trying to send { intField: 100, doubleField: 99.88 }
Reading with binary protocol.
Got test object: TestType(intField=100, doubleField=99.88)
[~/thrift-double]$ node test.js compact  | python test.py compact
Trying to send { intField: 100, doubleField: 99.88 }
Reading with compact protocol.
Got test object: TestType(intField=100, doubleField=-2.24248483894553e-38)
[~/thrift-double]$ 

I notice the following snippets.

• TBinaryProtocol.py: network format (big-endian)

    def readDouble(self):
      buff = self.trans.readAll(8)
      val, = unpack('!d', buff)
      return val
  

• TCompactProtocol.py: little-endian

    def readDouble(self):
      buff = self.trans.readAll(8)
      val, = unpack('<d', buff)
      return val
  

• binary.js: my best IEEE floating point days are behind me but this looks big-endian. In any case it's used for both compact and binary.

  exports.writeDouble = function(buff, v) {
    var m, e, c;
  
    buff[0] = (v < 0 ? 0x80 : 0x00);
  
    v = Math.abs(v);
    if (v !== v) {
      // NaN, use QNaN IEEE format
      m = 2251799813685248;
      e = 2047;
    } else if (v === Infinity) {
      m = 0;
      e = 2047;
    } else {
      e = Math.floor(Math.log(v) / Math.LN2);
      c = Math.pow(2, -e);
      if (v * c < 1) {
        e--;
        c *= 2;
      }
  
      if (e + 1023 >= 2047)
      {
        // Overflow
        m = 0;
        e = 2047;
      }
      else if (e + 1023 >= 1)
      {
        // Normalized - term order matters, as Math.pow(2, 52-e) and v*Math.pow(2, 52) can overflow
        m = (v*c-1) * POW_52;
        e += 1023;
      }
      else
      {
        // Denormalized - also catches the '0' case, somewhat by chance
        m = (v * POW_1022) * POW_52;
        e = 0;
      }
    }
  
    buff[1] = (e << 4) & 0xf0;
    buff[0] |= (e >> 4) & 0x7f;
  
    buff[7] = m & 0xff;
    m = Math.floor(m / POW_8);
    buff[6] = m & 0xff;
    m = Math.floor(m / POW_8);
    buff[5] = m & 0xff;
    m = Math.floor(m / POW_8);
    buff[4] = m & 0xff;
    m >>= 8;
    buff[3] = m & 0xff;
    m >>= 8;
    buff[2] = m & 0xff;
    m >>= 8;
    buff[1] |= m & 0x0f;
  
    return buff;
  };