Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
Impala 2.13.0
-
None
-
None
-
ghx-label-9
Description
Currently the practical limit for catalog topic update is in the neighborhood of 4GB - the fundamental limit of max thrift message size. This is an architectural limitation and not a resource limitation
Larger enterprise clusters with high file counts can easily surpass this with normal usage.
The high level ask here is for a more scalable implementation for metadata handling. The amount metadata that a cluster can handle should be proportional to the amount of hardware resource that an user is willing to allocate to it.