Uploaded image for project: 'Thrift'
  1. Thrift
  2. THRIFT-2933

v0.9.2: doubles encoded in node with compact protocol cannot be decoded by python

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.2
    • Fix Version/s: 0.9.3
    • Component/s: Node.js - Library
    • Labels:
      None

      Description

      I've noticed that with Thrift v0.9.2, using the Compact protocol, I can't encode doubles in node and decode them in python. It looks like in python, we're decoding little-endian for compact and big-endian for binary, and in node, writing big-endian in all cases (therefore, Binary works fine). I've attached a test program to show this behavior. I have not found the thrift binary spec, so I'm not sure which side is wrong.

      [~/thrift-double]$ thrift --version
      Thrift version 0.9.2
      [~/thrift-double]$ grep -i thrift package.json 
          "thrift": "0.9.2"
          "compile": "thrift --gen js:node --gen py test.thrift"
      [~/thrift-double]$ pip show thrift
      ---
      Name: thrift
      Version: 0.9.2
      Location: /Library/Python/2.7/site-packages
      Requires: 
      [~/thrift-double]$ npm run compile
      
      > double-decode-bug@0.0.1 compile /Users/dh/thrift-double
      > thrift --gen js:node --gen py test.thrift
      
      [~/thrift-double]$ node test.js binary | python test.py binary
      Trying to send { intField: 100, doubleField: 99.88 }
      Reading with binary protocol.
      Got test object: TestType(intField=100, doubleField=99.88)
      [~/thrift-double]$ node test.js compact  | python test.py compact
      Trying to send { intField: 100, doubleField: 99.88 }
      Reading with compact protocol.
      Got test object: TestType(intField=100, doubleField=-2.24248483894553e-38)
      [~/thrift-double]$ 
      

      I notice the following snippets.

      • TBinaryProtocol.py: network format (big-endian)
          def readDouble(self):
            buff = self.trans.readAll(8)
            val, = unpack('!d', buff)
            return val
        
      • TCompactProtocol.py: little-endian
          def readDouble(self):
            buff = self.trans.readAll(8)
            val, = unpack('<d', buff)
            return val
        
      • binary.js: my best IEEE floating point days are behind me but this looks big-endian. In any case it's used for both compact and binary.
        exports.writeDouble = function(buff, v) {
          var m, e, c;
        
          buff[0] = (v < 0 ? 0x80 : 0x00);
        
          v = Math.abs(v);
          if (v !== v) {
            // NaN, use QNaN IEEE format
            m = 2251799813685248;
            e = 2047;
          } else if (v === Infinity) {
            m = 0;
            e = 2047;
          } else {
            e = Math.floor(Math.log(v) / Math.LN2);
            c = Math.pow(2, -e);
            if (v * c < 1) {
              e--;
              c *= 2;
            }
        
            if (e + 1023 >= 2047)
            {
              // Overflow
              m = 0;
              e = 2047;
            }
            else if (e + 1023 >= 1)
            {
              // Normalized - term order matters, as Math.pow(2, 52-e) and v*Math.pow(2, 52) can overflow
              m = (v*c-1) * POW_52;
              e += 1023;
            }
            else
            {
              // Denormalized - also catches the '0' case, somewhat by chance
              m = (v * POW_1022) * POW_52;
              e = 0;
            }
          }
        
          buff[1] = (e << 4) & 0xf0;
          buff[0] |= (e >> 4) & 0x7f;
        
          buff[7] = m & 0xff;
          m = Math.floor(m / POW_8);
          buff[6] = m & 0xff;
          m = Math.floor(m / POW_8);
          buff[5] = m & 0xff;
          m = Math.floor(m / POW_8);
          buff[4] = m & 0xff;
          m >>= 8;
          buff[3] = m & 0xff;
          m >>= 8;
          buff[2] = m & 0xff;
          m >>= 8;
          buff[1] |= m & 0x0f;
        
          return buff;
        };
        
        

        Attachments

        1. double-endianness-2.patch
          3 kB
          Dan Heller
        2. little-endian.patch
          5 kB
          Dan Heller
        3. thrift-double.tgz
          1 kB
          Dan Heller

          Activity

            People

            • Assignee:
              codesf Randy Abernethy
              Reporter:
              dh Dan Heller
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: