Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-240

For Python 2.6 and above, use "json" module from stdlib instead of simplejson

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: python
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As described at http://bob.pythonmac.org/archives/2008/10/02/python-26-released-now-with-json/:

      1. Use simplejson or Python 2.6 json, prefer simplejson.
        try:
        import simplejson as json
        except ImportError:
        import json
      1. AVRO-240.patch
        2 kB
        Jeff Hammerbacher

        Activity

        Hide
        cutting Doug Cutting added a comment -

        I just committed this. Thanks, Jeff.

        Show
        cutting Doug Cutting added a comment - I just committed this. Thanks, Jeff.
        Hide
        hammer Jeff Hammerbacher added a comment -

        Still ready to commit, though.

        Show
        hammer Jeff Hammerbacher added a comment - Still ready to commit, though.
        Hide
        hammer Jeff Hammerbacher added a comment -

        Doug: this will not work when simplejson is not available for Python 2.5 and below, but there is no json parsing library in the standard library before Python 2.6. It will work when simplejson is not available for Python 2.6 and above.

        Show
        hammer Jeff Hammerbacher added a comment - Doug: this will not work when simplejson is not available for Python 2.5 and below, but there is no json parsing library in the standard library before Python 2.6. It will work when simplejson is not available for Python 2.6 and above.
        Hide
        cutting Doug Cutting added a comment -

        +1 This looks reasonable to me. It lets things still work when simplejson is not available, right? Unless there are objections, I will commit this.

        Show
        cutting Doug Cutting added a comment - +1 This looks reasonable to me. It lets things still work when simplejson is not available, right? Unless there are objections, I will commit this.
        Hide
        hammer Jeff Hammerbacher added a comment -

        While we can have an accelerated implementation (cf. Thrift's binary protocol) and do significant benchmarking at AVRO-217, I think the best practice right now is to use the standard library.

        Show
        hammer Jeff Hammerbacher added a comment - While we can have an accelerated implementation (cf. Thrift's binary protocol) and do significant benchmarking at AVRO-217 , I think the best practice right now is to use the standard library.
        Hide
        phunt Patrick Hunt added a comment -

        fyi, noticed this recently, it's a python json lib performance evaluation on python 2.6. You might want to
        add some detail in the avro py docs talking about how to get better/best performance, esp as it
        relates to the json library used (json stdlib performance is an order of magnitude behind the others tested)

        "I honestly didn't expect the stdlib json to be this far behind.

        Among the other C based libraries there isn't a clear winner. cjson is the best decoder but the slowest encoder, simplejson compiled with C speedups is the fastest encoder but the slowest decoder while jsonlib2 is somewhere in the middle for both cases."

        http://www.mikealrogers.com/archives/695

        Show
        phunt Patrick Hunt added a comment - fyi, noticed this recently, it's a python json lib performance evaluation on python 2.6. You might want to add some detail in the avro py docs talking about how to get better/best performance, esp as it relates to the json library used (json stdlib performance is an order of magnitude behind the others tested) "I honestly didn't expect the stdlib json to be this far behind. Among the other C based libraries there isn't a clear winner. cjson is the best decoder but the slowest encoder, simplejson compiled with C speedups is the fastest encoder but the slowest decoder while jsonlib2 is somewhere in the middle for both cases." http://www.mikealrogers.com/archives/695
        Hide
        hammer Jeff Hammerbacher added a comment -

        Attached patch also cleans up the import style at the top of procol.py to match PEP 8 style guidelines.

        Show
        hammer Jeff Hammerbacher added a comment - Attached patch also cleans up the import style at the top of procol.py to match PEP 8 style guidelines.

          People

          • Assignee:
            hammer Jeff Hammerbacher
            Reporter:
            hammer Jeff Hammerbacher
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development