[SAMZA-212] Explore using PostgreSQL changelog as input stream

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Labels: project
• Assignee: Martin Kleppmann
• Reporter: Martin Kleppmann
• Created: 30/Mar/14 21:38
• Updated: 23/Apr/15 19:54
• Resolved: 23/Apr/15 19:54

Description

It would be good to be able to capture changes to a PostgreSQL database, and use it as an input to Samza (like SAMZA-200 but for Postgres instead of MySQL).

On a high level, it seems like the options are:

• Creating a trigger, and using LISTEN/[NOTIFY|http://www.postgresql.org/docs/9.3/static/sql-notify.html] to send changes to an external process. (Example)
• Logical WAL decoding, which has been committed and should be in the upcoming Postgres 9.4 release.

@selenamarie points out: "problem with it is that LISTEN/NOTIFY are stateless. if you want reliable transport, will need logical replication".

This Twitter conversation has some more detail.

Issue Links

• relates to: SAMZA-200 Explore using MySQL changelog as input stream