[TRAFODION-2373] odb load 500k rows while set parameter max 505k

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: db-utility-odb
• Assignee: Unassigned
• Reporter: zhangliang
• Created: 29/Nov/16 03:05
• Updated: 28/Jun/17 02:34

Description

I want to load 505k rows to trafodion with odb, by setting parameter like max=505000/rows=5000/parallel=6.
However, odb just load 500k rows indeed.
Logs here:

[root@gy08 odb_test]# ./odb64luo -u zale -p zale -d gy_13_1 -l src=data_500k:tgt=trafodion.odb_copy.load_2:max=505000:rows=5000:parallel=6:loadcmd=UL:fs=,:bad=bad2 -vv
odb [main(1512)] - Execution Table content (7 items):
etab[0]:
Type (.type): l
Thread ID (.id): 0
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[1]:
Type (.type): L
Thread ID (.id): 1
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[2]:
Type (.type): L
Thread ID (.id): 2
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[3]:
Type (.type): L
Thread ID (.id): 3
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[4]:
Type (.type): L
Thread ID (.id): 4
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[5]:
Type (.type): L
Thread ID (.id): 5
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
etab[6]:
Type (.type): L
Thread ID (.id): 6
Parent EID (.parent): 0
Child EID (.child): 0
TBE flag (.tbe): 1
Field Separator (.fs): 44 (decimal value)
Record Separator (.rs): 10 (decimal value)
String Qualifier (.sq): 0 (decimal value)
Escape Character (.ec): 92 (decimal value)
Embed file Character (.em): 0 (decimal value)
Pad Character (.pc): 0 (decimal value)
Run (.run): (null)
Max Rows (.mr): 505000
Max Errors (.mer): 0
EID flags (.flg/.flg2): 16001000400/200000 (octal values)
Rowset (.r): 5000 rows
Rowset Buffer Size (.rbs): 0 bytes
Row Buffer Size (.s): 0 bytes
IO Buffer Size (.iobuff): 0 bytes
RW Buffer Size (.buffsz): 262144 bytes
Pre SQL (.pre): (null)
MPre SQL (.mpre): (null)
Post SQL (.post): (null)
Database Type (.dbt): 0
Source (.src): data_500k
Target (.Ocso[0-2]): (null).(null).(null)
Map File (.map): (null)
Lines to skip (.k): 0
Parallel Streams (.ps): 6
Null String (.ns): (null)
Commit (.cmt): 0 (-1=end, 0=auto, >0 num rows)
Field Truncation (.fldtr): 0
Load Command (.loadcmd): UL
odb [2016-11-29 10:45:57]: starting ODBC connection(s)... (1) 1 2 3 4 5 6
Connected to Trafodion
[1] 5,000 records inserted [commit]
[2] 5,000 records inserted [commit]
[3] 5,000 records inserted [commit]
[4] 5,000 records inserted [commit]
[5] 5,000 records inserted [commit]
[6] 5,000 records inserted [commit]
[1] 10,000 records inserted [commit]
[2] 10,000 records inserted [commit]
[3] 10,000 records inserted [commit]
[4] 10,000 records inserted [commit]
[5] 10,000 records inserted [commit]
[6] 10,000 records inserted [commit]
[1] 15,000 records inserted [commit]
[2] 15,000 records inserted [commit]
[3] 15,000 records inserted [commit]
[4] 15,000 records inserted [commit]
[5] 15,000 records inserted [commit]
[6] 15,000 records inserted [commit]
[1] 20,000 records inserted [commit]
[2] 20,000 records inserted [commit]
[3] 20,000 records inserted [commit]
[4] 20,000 records inserted [commit]
[5] 20,000 records inserted [commit]
[6] 20,000 records inserted [commit]
[1] 25,000 records inserted [commit]
[2] 25,000 records inserted [commit]
[3] 25,000 records inserted [commit]
[4] 25,000 records inserted [commit]
[5] 25,000 records inserted [commit]
[6] 25,000 records inserted [commit]
[1] 30,000 records inserted [commit]
[2] 30,000 records inserted [commit]
[3] 30,000 records inserted [commit]
[4] 30,000 records inserted [commit]
[5] 30,000 records inserted [commit]
[6] 30,000 records inserted [commit]
[1] 35,000 records inserted [commit]
[2] 35,000 records inserted [commit]
[3] 35,000 records inserted [commit]
[4] 35,000 records inserted [commit]
[5] 35,000 records inserted [commit]
[6] 35,000 records inserted [commit]
[1] 40,000 records inserted [commit]
[2] 40,000 records inserted [commit]
[3] 40,000 records inserted [commit]
[4] 40,000 records inserted [commit]
[5] 40,000 records inserted [commit]
[6] 40,000 records inserted [commit]
[1] 45,000 records inserted [commit]
[2] 45,000 records inserted [commit]
[3] 45,000 records inserted [commit]
[4] 45,000 records inserted [commit]
[5] 45,000 records inserted [commit]
[6] 45,000 records inserted [commit]
[1] 50,000 records inserted [commit]
[2] 50,000 records inserted [commit]
[3] 50,000 records inserted [commit]
[4] 50,000 records inserted [commit]
[5] 50,000 records inserted [commit]
[6] 50,000 records inserted [commit]
[1] 55,000 records inserted [commit]
[2] 55,000 records inserted [commit]
[3] 55,000 records inserted [commit]
[4] 55,000 records inserted [commit]
[5] 55,000 records inserted [commit]
[6] 55,000 records inserted [commit]
[1] 60,000 records inserted [commit]
[2] 60,000 records inserted [commit]
[3] 60,000 records inserted [commit]
[4] 60,000 records inserted [commit]
[5] 60,000 records inserted [commit]
[6] 60,000 records inserted [commit]
[1] 65,000 records inserted [commit]
[2] 65,000 records inserted [commit]
[3] 65,000 records inserted [commit]
[4] 65,000 records inserted [commit]
[5] 65,000 records inserted [commit]
[6] 65,000 records inserted [commit]
[1] 70,000 records inserted [commit]
[2] 70,000 records inserted [commit]
[3] 70,000 records inserted [commit]
[4] 70,000 records inserted [commit]
[5] 70,000 records inserted [commit]
[6] 70,000 records inserted [commit]
[1] 75,000 records inserted [commit]
[2] 75,000 records inserted [commit]
[3] 75,000 records inserted [commit]
[4] 75,000 records inserted [commit]
[5] 75,000 records inserted [commit]
[6] 75,000 records inserted [commit]
[1] 80,000 records inserted [commit]
[2] 80,000 records inserted [commit]
[3] 80,000 records inserted [commit]
[4] 80,000 records inserted [commit]
[5] 80,000 records inserted [commit]
[1] 85,000 records inserted [commit]
[2] 85,000 records inserted [commit]
[3] 85,000 records inserted [commit]
[4] 85,000 records inserted [commit]
[6] 80,000 records inserted [commit]
[0] odb version 1.1.0 Load statistics:
[0] Target table: TRAFODION.ODB_COPY.LOAD_2
[0] Source: data_500k
[0] Pre-loading time: 28.536 s (00:00:28.536)
[0] Loading time: 25.695 s(00:00:25.695)
[0] Total records read: 500,000
[0] Total records inserted: 500,000
[0] Total number of columns: 2
[0] Total bytes read: 5,388,895
[0] Average input row size: 10.8 B
[0] ODBC row size: 14 B (data) + 16 B (len ind)
[0] Rowset size: 5,000
[0] Rowset buffer size: 146.48 KiB
[0] Load throughput (real data): 204.810 KiB/s
[0] Load throughput (ODBC): 266.042 KiB/s
[0] Reader Total/Wait Cycles: 100/94

This issue occurs when I set max=501k,502k and so on, odb just load 500k instead.

Issue Links

• is duplicated by: TRAFODION-2670 when I copy data from oracle table to trafodion table with max for limit rows， but the result rows in trafodion table is differ with max rows , and without any error.