[DRILL-7756] Storage IPFS: GroupScan: detect malformed datasets when planning

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Storage - Other
• Assignee: Bowen Ding
• Reporter: Bowen Ding
• Created: 26/Jun/20 14:13
• Updated: 26/Jun/20 14:13

Description

The initial design of a Drill-queriable dataset on IPFS is a MerkleTree, where the leaf nodes contain the actual data, and the intermediate nodes record the hashes of the leaf nodes. Not all datasets in the wild are well-formed MerkleTrees, because IPFS allows arbitrary DAGs.

The current implementation in IPFSGroupScan does not check if a dataset is well-formed, and if it contains recursive links, the planner will have no way to detect it and will loop forever.