Details
-
Improvement
-
Status: In Progress
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
bulk loader(TransferToHFile.scala) job assumes that bulk loading into new hbase table only with insertBulk operations in bulk data. it can`t process incremental bulk load onto existing table.
in many cases when there is no realtime updates but there is only batch process to update in bulk manner, processing these bulk update through hbase rpc on region server can be problematic in many way, most importantly too frequent memstore flush yield extra latency on read request while applying bulk updates.
loader project utilize hbase`s bulk load feature and hbase`s bulk load(https://issues.apache.org/jira/browse/HBASE-1923) already support incremental load into existing table. the problem is that loader`s TransferToHFile job only assumed insert, not delete. so I suggest TransferToHFile to support both insert and delete operations so incremental bulk load into existing graph could be possible.
one thing I am not sure is how we going to deal with degree value. if it is first bulk load on new hbase table, then it is simple, just group by from or to and count number of edges. after counting, we can use put instead of increment because it is safe to assume previous value is 0.
when we load incrementally, we need to get previous degree and increment by current batch`s degree value. this require read operation and can be much slow comparing to just put.
