add RPC per-call metadata and a plugin API to access it

• Key: AVRO-66
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: George Porter
• Reporter: George Porter
• Component/s: c, c++, java, python, spec
• Fix Version/s: 1.0.0

Description

The RPC specification should support per-call metadata maps. Requestor and Responder should have methods that access this per-call metadata.

Comments

[George Porter]

This is a simple Java implementation that supports an empty map, but does not include a public API.

[Doug Cutting]

A simpler and more complete way to implement this might be to:

private static final Schema META = Schema.createMap(Schema.create(Schema.Type.BYTES));
private static final GenericDatumReader META_READER = new GenericDatumReader(META);
private static final GenericDatumWriter META_WRITER = new GenericDatumWriter(META);
...
Map<String,ByteBuffer> meta = (Map<String,ByteBuffer>)META_READER.read(null, in);
...
META_WRITER.write(meta, out);

Also, it would be good to have a Python impl committed alongside this, so that interop tests still pass. This could probably do something similar to the above. Do you think you'll be able to do this today?

[George Porter]

This patch incorporates Doug's suggestion to use a GenericDatumReader/Writer to consume and produce a map (vs low-level calls to Encoder and Decoder).

This patch always sends empty maps from the Requestor to the Responder. The Responder simply echos back to the Requestor the same Map it was given. Finally, the Requestor discards the metadata it receives. This will change once we implement the plugin handler mechanism shortly.

[Doug Cutting]

Here's a new version with some minor improvements:

• responder should not echo meta by default, but return empty;
• moved 'static final' fields ahead of non-static fields;
• removed unneeded casts

We still need equivalent Python. I'm not a Python programmer, but will attempt to fake it this afternoon if no one else does first.

[George Porter]

Fixed a small problem with the previous patch by replacing Strings with Utf8s

[George Porter]

Looks like our patches crossed in the mail. I like this patch, but perhaps we should switch "String" to "Utf8"?

[Doug Cutting]

Yes, Utf8 is correct. Will you repair or should I?

[George Porter]

I will
-George

[George Porter]

Updated Doug's patch to change Strings to Utf8s

[George Porter]

This patch includes the Java changes as before, but also changes to Python.

All tests pass, and the interop tests pass as well.

I'm not a Python programmer, and so I'm not sure if there is a better way to manifest a Map schema in the program than what I did, which was to read it out of an on-disk file called Meta.avsc

[George Porter]

Forgot to include Meta.avsc in the previous patch.

[Doug Cutting]

Here's a version that includes Python support.

I didn't see your patch when I submitted mine. You missed the error case in the Python responder. I inlined the schema, rather than reading it from a file. It seems simple enough that it's not worth reading from a file. What do you think?

[George Porter]

It looks like our patches crossed in the mail again...

I'm happy with using the Python code that you just committed vs mine, since mine included a separate Meta.avsc file that isn't necessary.

[Doug Cutting]

I just committed this. Thanks, George.

[Sharad Agarwal]

Thanks George and Doug for quickly getting in python support.