[sqlserver] use nvarchar to load unicode data?

[nickolay]

• $ sling --version

    Version: 1.0.48
  

• file /c/temp/test_unicode.csv

    /c/temp/test_unicode.csv: Unicode text, UTF-8 text, with CRLF line terminators
  

• cat /c/temp/test_unicode.csv

    a;b;c
    aaa;bbb;ñññ
  

• cat /c/temp/test_unicode.csv | sling run --tgt-conn dev --tgt-object dbo.test

    INF connecting to target database (sqlserver)
    INF reading from stream (stdin)
    INF delimiter auto-detected: ";"
    INF writing to target database [mode: full-refresh]
    INF streaming data
    WRN bcp version 13 is old. This may give issues with sling, consider upgrading.
    INF dropped table "dbo"."test"
    INF created table "dbo"."test"
    INF inserted 1 rows in 0 secs [2 r/s]
    INF execution succeeded
  

This creates varchar columns:

CREATE TABLE [dbo].[test](
	[a] [varchar](255) NULL,
	[b] [varchar](255) NULL,
	[c] [varchar](255) NULL,
	[_sling_loaded_at] [bigint] NULL
) ON [PRIMARY]
GO

...which use the database default collation, corrupting any incompatible characters:

a	b	c
aaa	bbb	+-+-+-

Would it be possible to use the unicode nvarchar type instead of varchar?

[flarco]

Done.
Commit: flarco/dbio@7b87316
Releasing 1.0.50 in a bit.

[nickolay]

Thanks! I can see that the column type is now nvarchar, but the data is still not loaded correctly.

Will look into it further the next time I have something to load!

[flarco]

Interesting, it might be the encoding?
Feel free to try the transforms source option (source.options.transforms), provided as an array.
Maybe decode_latin1 or decode_windows1250?

See:

• https://github.com/slingdata-io/sling-cli/blob/main/core/sling/tranforms.go
• https://docs.slingdata.io/sling-cli/running-tasks#advanced-options

[nickolay]

Will look into it further the next time I have something to load!

Filed #518