[IMPALA-11476] Support for Ozone with Erasure Coding

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Labels: ozone
• Epic Link: Impala Ozone Phase 2
• Epic Color: ghx-label-11
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 03/Aug/22 21:05
• Updated: 13/Mar/23 22:17
• Resolved: 25/Jan/23 18:20

Description

Similar to HDFS Erasure Coding, Ozone has the ability to use erasure coding rather than replication to store data: https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/feature/erasurecoding.html.

We have a mode to enable Erasure Coding, we should make it work with Ozone.

Erasure Coding is not yet in a public release, although it's expected in CDP 7.1.8. Doing this will depend on having releases available to test with.

May need a few updates to mark things as erasure-coded so Impala schedules them correctly.

Issue Links

• causes: IMPALA-11873 run-mini-dfs failed by "OZONE_ERASURECODE_POLICY: unbound variable"
• causes: IMPALA-11903 Ozone tests failing due to mismatch in ERASURECODE_POLICY
• depends upon: HDDS-7208 Erasure coding and encryption are not flagged on FileStatus
• depends upon: HDDS-7603 Erasure coding and encryption are not flagged on FileStatus for ofs
• relates to: IMPALA-11698 Ozone local reads do not update local-bytes-read
• relates to: IMPALA-7003 Support erasure-coding in impala