Details
Description
Ingest of Flow records results in corrupted records that appear to be from the header. These are present in every hour of input within the Hive table. the 'input' field chosen for the query below is one of many that are NULL; The the treceived, sip, dip, and rip fields are also invalid; a check that sip/dip/rip are all valid ip addresses might make an easy filter.
Example query where this can be seen - my apologies for the inevitable line wrap.
Query: select * from flow where y=2017 and m=5 and d=14 and h=1 and input is null
treceived | unix_tstamp | tryear | trmonth | trday | trhour | trminute | trsec | tdur | sip | dip | sport | dport | proto | flag | fwd | stos | ipkt | ibyt | opkt | obyt | input | output | sas | das | dtos | dir | rip | y | m | d | h |
--------------------------------------------------------------------------------------------------------------------------------------------------------------+
tr | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | sa | da | NULL | NULL | pr | flg | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | ra | 2017 | 5 | 14 | 1 |
tr | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | sa | da | NULL | NULL | pr | flg | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | ra | 2017 | 5 | 14 | 1 |
Summary | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
flows | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
937977 | NULL | NULL | 330361067 | 54411321 | 4358 | 1560 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
Summary | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
flows | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
1010505 | NULL | NULL | 431994872 | 67325977 | 5566 | 1511 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2017 | 5 | 14 | 1 |
Fetched 8 row(s) in 1.10s