[HUDI-2630] Add undocumented features

• Type: Task
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Epic Link: Improve Hudi website UI and documentation
• Assignee: Kyle Weller
• Reporter: Kyle Weller
• Created: 26/Oct/21 16:43
• Updated: 07/Jan/22 02:58
• Resolved: 24/Nov/21 05:47

Description

Ideas:

• [ ] Metadata table/table metadata
• [ ] File sizing
• [ ] Compaction, configs, strategies (Siva)
• [ ] Keys
  • [ ] virtual keys
  • [ ] meta fields (_hoodie_record_key and such)
• [ ] Document every single write operations :
  • [ ] incremental: upsert, insert, delete
  • [ ] batch ones : insert_overwrite, ...
• [ ] DeltaStreamer
  • [ ] deployment
  • [ ] sources
  • [ ] de-duplication
  • [ ] checkpoint provider
• [ ] Marker mechanism
• [ ] Timeline server and metaserver
• [ ] Indexing needs a page
• [ ] Table services - 1 page each
  • [ ] bootstrap
• [ ] Document all the different utilities (try them out locally)
  • [ ] multi/deltastreamer
  • [ ] snapshotter
  • [ ] exporter
  • [ ] SchemaProvider
  • [ ] Precommit validators SparkPreCommitValidator
  • [ ] It’d be very helpful if there was a table of sources + schema providers, along with examples of the input format for the source in the Hudi documentation
• [ ] Hive sync
• [ ] Document prometheus reporter
• [ ] Document all public APIs (keygenerators, payloads, ... )
• [ ] Commit Notifications
• [ ] Address stuff from https://issues.apache.org/jira/browse/HUDI-1958