Avro
  1. Avro
  2. AVRO-240

For Python 2.6 and above, use "json" module from stdlib instead of simplejson

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: python
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As described at http://bob.pythonmac.org/archives/2008/10/02/python-26-released-now-with-json/:

      1. Use simplejson or Python 2.6 json, prefer simplejson.
        try:
        import simplejson as json
        except ImportError:
        import json
      1. AVRO-240.patch
        2 kB
        Jeff Hammerbacher

        Activity

        Hide
        Jeff Hammerbacher added a comment -

        Attached patch also cleans up the import style at the top of procol.py to match PEP 8 style guidelines.

        Show
        Jeff Hammerbacher added a comment - Attached patch also cleans up the import style at the top of procol.py to match PEP 8 style guidelines.
        Hide
        Patrick Hunt added a comment -

        fyi, noticed this recently, it's a python json lib performance evaluation on python 2.6. You might want to
        add some detail in the avro py docs talking about how to get better/best performance, esp as it
        relates to the json library used (json stdlib performance is an order of magnitude behind the others tested)

        "I honestly didn't expect the stdlib json to be this far behind.

        Among the other C based libraries there isn't a clear winner. cjson is the best decoder but the slowest encoder, simplejson compiled with C speedups is the fastest encoder but the slowest decoder while jsonlib2 is somewhere in the middle for both cases."

        http://www.mikealrogers.com/archives/695

        Show
        Patrick Hunt added a comment - fyi, noticed this recently, it's a python json lib performance evaluation on python 2.6. You might want to add some detail in the avro py docs talking about how to get better/best performance, esp as it relates to the json library used (json stdlib performance is an order of magnitude behind the others tested) "I honestly didn't expect the stdlib json to be this far behind. Among the other C based libraries there isn't a clear winner. cjson is the best decoder but the slowest encoder, simplejson compiled with C speedups is the fastest encoder but the slowest decoder while jsonlib2 is somewhere in the middle for both cases." http://www.mikealrogers.com/archives/695
        Hide
        Jeff Hammerbacher added a comment -

        While we can have an accelerated implementation (cf. Thrift's binary protocol) and do significant benchmarking at AVRO-217, I think the best practice right now is to use the standard library.

        Show
        Jeff Hammerbacher added a comment - While we can have an accelerated implementation (cf. Thrift's binary protocol) and do significant benchmarking at AVRO-217 , I think the best practice right now is to use the standard library.
        Hide
        Doug Cutting added a comment -

        +1 This looks reasonable to me. It lets things still work when simplejson is not available, right? Unless there are objections, I will commit this.

        Show
        Doug Cutting added a comment - +1 This looks reasonable to me. It lets things still work when simplejson is not available, right? Unless there are objections, I will commit this.
        Hide
        Jeff Hammerbacher added a comment -

        Doug: this will not work when simplejson is not available for Python 2.5 and below, but there is no json parsing library in the standard library before Python 2.6. It will work when simplejson is not available for Python 2.6 and above.

        Show
        Jeff Hammerbacher added a comment - Doug: this will not work when simplejson is not available for Python 2.5 and below, but there is no json parsing library in the standard library before Python 2.6. It will work when simplejson is not available for Python 2.6 and above.
        Hide
        Jeff Hammerbacher added a comment -

        Still ready to commit, though.

        Show
        Jeff Hammerbacher added a comment - Still ready to commit, though.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Jeff.

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Jeff.

          People

          • Assignee:
            Jeff Hammerbacher
            Reporter:
            Jeff Hammerbacher
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development