Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Add an Elasticsearch writer to Gobblin.
Let me know if anyone is already working on this, or has any thoughts on this writer. Will probably work on it over the new few weeks.
Github Url : https://github.com/linkedin/gobblin/issues/1912
Github Reporter : stakiar
Github Created At : 2017-06-02T07:13:53Z
Github Updated At : 2017-07-13T15:58:19Z
Comments
shirshanka wrote on 2017-06-02T17:41:49Z : Model it on couchbase/Kafka/eventhub writer so you get streaming plus async for free.
Shirshanka
> On Jun 2, 2017, at 12:13 AM, Sahil Takiar <notifications@github.com> wrote:
>
> Add an Elasticsearch writer to Gobblin.
>
> Let me know if anyone is already working on this, or has any thoughts on this writer. Will probably work on it over the new few weeks.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-305862211
zxc wrote on 2017-06-06T17:39:18Z : Is it a writer based on http protocol?
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-306562022
stakiar wrote on 2017-06-06T22:15:25Z : @shirshanka will do. Right now I have the writer implementing `BatchAsyncDataWriter` and using `BufferedAsyncDataWriter`. This way it gets batching for free.
@zxcware the plan to is provide two writers, one that uses http and the other that uses the Elasticsearch's Java Client `TransportClient`. I haven't done work for the http writer yet, but I will probably base it on `RestJsonWriter`.
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-306633069
zxc wrote on 2017-06-07T01:01:33Z : @sahilTakiar Don't do it based on `RestJsonWriter`, which will be deprecated. Checkout `AsyncHttpWriter`
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-306655993
zxc wrote on 2017-06-07T01:03:45Z : @sahilTakiar Just heads up, I am doing some refactoring over the whole sync piece in order to support http read. You may see some back incompatible changes
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-306656295
stakiar wrote on 2017-06-07T01:09:33Z : @zxcware thanks for the heads up, will do.
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-306657099
zxc wrote on 2017-07-13T01:17:37Z : @sahilTakiar How is the ElasticsearchWriter going?
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-314941960
stakiar wrote on 2017-07-13T15:58:19Z : I have a working version and some basic integration tests. Just working on cleaning up the code and writing some documentation. So probably a few more weeks.
Github Url : https://github.com/linkedin/gobblin/issues/1912#issuecomment-315122324