Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
None
-
None
-
None
Description
The balancing of the record to the loaders in done internally in SQOOP today
While writing the Kite Connector Qian noticed that this is not done fairly.
While I am testing kite connector, I allocated 2 loaders. I thought data will be divided by 50% and 50% to both loaders. But actually the second loader does nothing, because its DataReader does not have any data to provide. Is it by design?
>> About loaders do not have data in a balanced way.
My scenario is 4 "jdbc_mysql" extractors to extract 100k row data (10MB). There are 2 Kite loaders to read data.
This must be a bug that needs to be fixed in SQOOP ( abec confirmed it.
Attachments
Issue Links
- is cloned by
-
SQOOP-1603 Sqoop2: Explicit support for Merge in the Sqoop Job lifecyle in the MR engine
- Open
- relates to
-
SQOOP-1938 DOC:update the sqoop MR engine implementation details
- Resolved