Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.7.0
-
Component/s: None
-
Labels:None
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Also, it might be simpler to add the new function 'getContentSummary' in all existing
input formats.
You can create a dummy class which all other hive input formats (other than symlinktextinputformat) extend.
In the abstract dummy class, the existing defn. can be there.
FileSystem fs = p.getFileSystem(ctx.getConf());
cs = fs.getContentSummary(p);
That waym you dont need any special checking in Utilities.java - it calls getContentSummary(),
which is implemented by all input formats that hive supports.
namit, a .q test file can not include what this jira does. From a .q file, it is very difficult to know SymlinkTextInputFormat get the input size correctly.
>>getContentSummary' in all existing input formats.
There is no guarantee that the inputformat is from Hive. It is very difficult to change all input format.
fixed a typo
+1
Committed. Thanks Yongqiang!
It might be simpler to add a .q file testcase.
Just load 2 files (say a1.q and a2.q in a hdfs directory).
Then load a new file, say foo, for the table 'T' - the contents of the file 'foo' are
a1.q
a2.q
Then, 'T' can be queried