how should I handle null value from spark

[Tasselmi]

Caused by: java.lang.Throwable: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected \t before: \\N\t1\t50\t\\N\t\\N\t\\N\t\\N\t9139\t6916\t9118\t482856\n2019-10-03\tv X21UD A\t1\t3.4.3.3\t10\t\\N\t1\t52\t\\N\t\\N\t\\N\t\\N\t195\t136\t180\t1261641\n2019-10-03\tv Y66i A\t1\t2.8.1.1\t15\t\\N\t1\t: (at row 1)

It seems that clickhouse4j parsed null value in spark as \\N

[doom369]

That's correct. \\N is value for the null in the clickhouse HTTP protocol.
Please clarify your question.

[Tasselmi]

protocol

path_direction is int :

scala> result.printSchema
root
 |-- dt: string (nullable = true)
 |-- model: string (nullable = true)
 |-- app_id: integer (nullable = true)
 |-- app_version_name: string (nullable = true)
 |-- session_type: integer (nullable = true)
 |-- path_direction: integer (nullable = true)

but an error is out ：

Caused by: java.lang.Throwable: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected \t before: \\N\t1\t13\t\\N\t\\N\t\\N\t\\N\t872\t588\t628\t41468596\n2019-10-03\tvivo Y13L\t1\t2.7.7.0\t15\t\\N\t1\t29\t\\N\t\\N\t\\N\t\\N\t42\t24\t40\t777177\n2019-10-03\tOPPO R9s Plus\t1\t3.0.0.0\t10\t\\N\t1\t1\t\\N\t\\: (at row 1)

Row 1:
Column 0,   name: dt,               type: Date,             parsed text: "2019-10-03"
Column 1,   name: model,            type: String,           parsed text: "vivo Y67A"
Column 2,   name: app_id,           type: UInt8,            parsed text: "1"
Column 3,   name: app_version_name, type: String,           parsed text: "3.3.0.0"
Column 4,   name: session_type,     type: UInt8,            parsed text: "10"
Column 5,   name: path_direction,   type: UInt8,            ERROR: text "<BACKSLASH>N<TAB>1<TAB>13<TAB><BACKSLASH>N" is not like UInt8

I do not know why

[einpare]

the same problem +1

[einpare]

@doom369 for help

[einpare]

ru.yandex.clickhouse has the same problem

ru.yandex.clickhouse.except.ClickHouseException (Cannot parse input: expected '\t' before: '\N\t\N\t1\tcampaign\t2021-03-05 10:59:24\t\N\t0\t2021-03-05 10:59:24\t1\t109978\': (at row 1)

Row 1:
Column 0, name: partition_time, type: UInt64, parsed text: "2021030510"
Column 1, name: trace_id, type: String, parsed text: "70c05820b62948b0bc276d4bb3aa836c"
Column 2, name: cid, type: UInt64, parsed text: "3106373507"
Column 3, name: aid, type: UInt64, ERROR: text "NN1ca" is not like UInt64

(version 20.7.2.30)

[TimmannaC]

Hi All,
I'm also facing the same problem.
Appreciate your response.

Thanks