[SPARK-24199] Structured Streaming

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: shuke
• Created: 07/May/18 09:33
• Updated: 12/Dec/22 18:10
• Resolved: 08/May/18 01:10

Description

Hey,when i use the where operate to filter data while using Structured Streaming

I got some problem about it 

 

========

get_json_object(col("value"),"$.type").cast(DataTypes.StringType).alias("type"),
get_json_object(col("value"),"$.saleData.type").cast(DataTypes.StringType).alias("saleDataType",get_json_object(col("value"), "$.uid").cast(DataTypes.IntegerType).alias(ROI_SHOP_KEY),
from_unixtime(get_json_object(col("value"), "$.time").cast(DataTypes.IntegerType),"yyyy-MM-dd").alias("event_time"),
//get_json_object(xjson, '$.balanceData.money')/100 as money,
(get_json_object(col("value"), "$.balanceData.money").cast(DataTypes.DoubleType) / 100).alias(BUSINESS_AMOUNT),
get_json_object(col("value"), "$.shopData.id").cast(DataTypes.LongType).alias(DARK_ID),
get_json_object(col("value"), "$.balanceData.out_trade_no").cast(DataTypes.StringType).alias(OUT_TRADE_NO),
get_json_object(col("value"), "$.balanceData.type").cast(DataTypes.StringType).alias(BalanceData_type)
)

.where("(type = 'residue' and saleDataType = '4' and shop_id in ('8610022','5382783')) or type = 'promotion' " )
.select(col("*"))
.writeStream
.trigger(Trigger.ProcessingTime(5000))
.outputMode("Update")
.format("console")
.start()

=========

i find that its not while using this way to filter data 

anyone can help 

best wishes