[AVRO-892] Python snappy error: "integer out of range for 'I' format code"

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.4
• Fix Version/s: 1.6.0
• Component/s: python

Environment

Linux michaelc 2.6.38-11-generic #48-Ubuntu SMP Fri Jul 29 19:02:55 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 11.04
Python 2.7.1+ (ubuntu stock version)

avro-1.5.4-py2.7.egg
snappy-1.0.4 (c library)
python-snappy-0.3.2

• Tags: python, snappy
• Assignee: Michael Cooper
• Reporter: Michael Cooper
• Created: 21/Sep/11 04:35
• Updated: 19/Sep/18 10:06
• Resolved: 22/Sep/11 20:50

Description

The Python library for avro fails to write some blocks when used with snappy compression.

The error is:

Traceback (most recent call last):
  File "tools/json_to_avro.py", line 74, in <module>
    writer.append(line)
  File "/home/michaelc/.python/2.7/avro-1.5.4-py2.7.egg/avro/datafile.py", line 185, in append
    self._write_block()
  File "/home/michaelc/.python/2.7/avro-1.5.4-py2.7.egg/avro/datafile.py", line 169, in _write_block
    self.encoder.write_crc32(uncompressed_data)
  File "/home/michaelc/.python/2.7/avro-1.5.4-py2.7.egg/avro/io.py", line 364, in write_crc32
    self.write(STRUCT_CRC32.pack(crc32(bytes)));
struct.error: integer out of range for 'I' format code

From my investigation, str(crc32(bytes)) is showing negative integers, so the issue seems to be fixed by masking the output.

This fix appears to work from my limited testing:

--- io.old.py	2011-09-21 14:32:38.992544680 +1000
+++ io.py	2011-09-21 14:33:11.492544686 +1000
@@ -360,7 +360,7 @@
     """
     A 4-byte, big-endian CRC32 checksum
     """
-    self.write(STRUCT_CRC32.pack(crc32(bytes)));
+    self.write(STRUCT_CRC32.pack(crc32(bytes) & 0xffffffff));
 
 #
 # DatumReader/Writer

Issue Links

• is cloned by: AVRO-2227 CLONE - Python snappy error: "integer out of range for 'I' format code"