Avro
  1. Avro
  2. AVRO-1296

Python: schemas retrieved from protocol types ignore namespace

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.4
    • Fix Version/s: 1.7.5
    • Component/s: python
    • Labels:
      None

      Description

      If I parse a protocol p using avro.protocol.parse, which defines "namespace": "ns" and then retrieve a child schema s from the protocol's proto.types (or proto.types_dict), then s does not have its namespace set (to ns), even if p has a namespace.

      This is particularly problematic if I'm using s to write out an avro file intended to be read by a specific-type reader, because the file header will claim to be objects of type s (not ns.s, as expected).

      I've attached two patches: one that makes sure that the namespace property of protocol types is set to the default namespace of the protocol when not otherwise set.

      The second patch ensures that the namespace is not rendered into JSON when a default protocol specifies the right value already.

      1. AVRO-1296b.patch
        3 kB
        Jeremy Kahn
      2. AVRO-1296a.patch
        2 kB
        Jeremy Kahn

        Activity

        Hide
        Jeremy Kahn added a comment -

        Philip Zeyliger merged the patch, and a followup patch that restored test functionality for Python 2.6.

        Show
        Jeremy Kahn added a comment - Philip Zeyliger merged the patch, and a followup patch that restored test functionality for Python 2.6.
        Hide
        Hudson added a comment -

        Integrated in AvroJava #363 (See https://builds.apache.org/job/AvroJava/363/)
        AVRO-1296. Fixing build breakage for python2.6. (Revision 1470716)
        AVRO-1296. Python: Fix schemas retrieved from protocol types to not ignore namespaces. Contributed by Jeremy Kahn. (Revision 1470635)

        Result = SUCCESS
        philz :
        Files :

        • /avro/trunk/lang/py/test/test_protocol.py

        philz :
        Files :

        • /avro/trunk/CHANGES.txt
        • /avro/trunk/lang/py/src/avro/protocol.py
        • /avro/trunk/lang/py/src/avro/schema.py
        • /avro/trunk/lang/py/test/test_protocol.py
        Show
        Hudson added a comment - Integrated in AvroJava #363 (See https://builds.apache.org/job/AvroJava/363/ ) AVRO-1296 . Fixing build breakage for python2.6. (Revision 1470716) AVRO-1296 . Python: Fix schemas retrieved from protocol types to not ignore namespaces. Contributed by Jeremy Kahn. (Revision 1470635) Result = SUCCESS philz : Files : /avro/trunk/lang/py/test/test_protocol.py philz : Files : /avro/trunk/CHANGES.txt /avro/trunk/lang/py/src/avro/protocol.py /avro/trunk/lang/py/src/avro/schema.py /avro/trunk/lang/py/test/test_protocol.py
        Hide
        Jeremy Kahn added a comment -

        Looks like the Ubuntu 9.10 buildbot complains about this test patch. Updated test code is included here:

        https://github.com/jkahn/avro/commit/9724cd0e17f338db6a12ebc1fce5132cdf934bc7

        @@ -379,7 +379,7 @@ def test_inner_namespace_not_rendered(self):
             self.assertEqual('com.acme.Greeting', proto.types[0].fullname)
             self.assertEqual('Greeting', proto.types[0].name)
             # but there shouldn't be 'namespace' rendered to json on the inner type
        -    self.assertNotIn('namespace', proto.to_json()['types'][0])
        +    self.assertFalse('namespace' in proto.to_json()['types'][0])
         
           def test_valid_cast_to_string_after_parse(self):
             """
        
        Show
        Jeremy Kahn added a comment - Looks like the Ubuntu 9.10 buildbot complains about this test patch. Updated test code is included here: https://github.com/jkahn/avro/commit/9724cd0e17f338db6a12ebc1fce5132cdf934bc7 @@ -379,7 +379,7 @@ def test_inner_namespace_not_rendered(self): self.assertEqual('com.acme.Greeting', proto.types[0].fullname) self.assertEqual('Greeting', proto.types[0].name) # but there shouldn't be 'namespace' rendered to json on the inner type - self.assertNotIn('namespace', proto.to_json()['types'][0]) + self.assertFalse('namespace' in proto.to_json()['types'][0]) def test_valid_cast_to_string_after_parse(self): """
        Hide
        Philip Zeyliger added a comment -

        Jeremy,

        Thanks for your contribution! I've committed it to trunk. Sorry 'bout the delay.

        [0]cairn::avro-svn(480326)$svn commit
        Sending        CHANGES.txt
        Sending        lang/py/src/avro/protocol.py
        Sending        lang/py/src/avro/schema.py
        Sending        lang/py/test/test_protocol.py
        Transmitting file data ....
        Committed revision 1470635.
        
        Show
        Philip Zeyliger added a comment - Jeremy, Thanks for your contribution! I've committed it to trunk. Sorry 'bout the delay. [0]cairn::avro-svn(480326)$svn commit Sending CHANGES.txt Sending lang/py/src/avro/protocol.py Sending lang/py/src/avro/schema.py Sending lang/py/test/test_protocol.py Transmitting file data .... Committed revision 1470635.
        Hide
        Jeremy Kahn added a comment -

        Philip, have you received any objections? Could you commit this to trunk?

        Show
        Jeremy Kahn added a comment - Philip, have you received any objections? Could you commit this to trunk?
        Hide
        Philip Zeyliger added a comment -

        Patches look good to me. Unless there are objections, I'll commit tomorrow or so.

        Show
        Philip Zeyliger added a comment - Patches look good to me. Unless there are objections, I'll commit tomorrow or so.
        Hide
        Jeremy Kahn added a comment -

        Commenting to nudge this issue.

        Can somebody review these Python patches? It's a small change but it fixes a fairly serious obstacle to using Avro files as a Java/Python interlingua for on-disk storage.

        Show
        Jeremy Kahn added a comment - Commenting to nudge this issue. Can somebody review these Python patches? It's a small change but it fixes a fairly serious obstacle to using Avro files as a Java/Python interlingua for on-disk storage.
        Hide
        Jeremy Kahn added a comment -

        Python tests pass ant clean build test after each of these patches are included. Each patch includes new tests that fail before and succeed after.

        Show
        Jeremy Kahn added a comment - Python tests pass ant clean build test after each of these patches are included. Each patch includes new tests that fail before and succeed after.
        Hide
        Jeremy Kahn added a comment -

        AVRO-1296a and AVRO-1296b are the two patches mentioned in the OP.

        Show
        Jeremy Kahn added a comment - AVRO-1296 a and AVRO-1296 b are the two patches mentioned in the OP.

          People

          • Assignee:
            Jeremy Kahn
            Reporter:
            Jeremy Kahn
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development