SFTP to DB: wildcards in filter don't work

[pikachuev]

Issue Description

• Description of the issue: I'm trying to ingest files from SFTP server:

When I use wildcards in stream: /Address_*.csv.zip, I'm getting the error. With one file, for example /Address_20250130.csv.zip, it works fine.

• Sling version (sling --version): 1.4.2

• Operating System (linux, mac, windows): linux

• Replication Configuration:

source: SFTP
target: SNOWFLAKE

defaults:
  mode: incremental

streams:
  /Address_*.csv.zip:
    id: address
    object: 'test_sling.address'
    update_key: _sling_loaded_at
    columns:
      "*": string
    source_options:
      format: csv
      compression: zip
      delimiter: '|'
      header: true

env:
  SLING_LOADED_AT_COLUMN: "timestamp"
  SLING_STREAM_URL_COLUMN: "true"

• Log Output (please run command with -d):

2025-02-04 16:46:32 INF Sling CLI | https://slingdata.io
2025-02-04 16:46:32 DBG adding default private key (/home/pika/.ssh/id_rsa) as auth method for SFTP
2025-02-04 16:46:36 DBG opened "sftp" connection (conn-sftp-Y7C)
2025-02-04 16:46:36 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:36 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:36 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:36 INF Sling Replication | SFTP -> SNOWFLAKE | /Address_*.csv.zip
2025-02-04 16:46:36 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:36 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:36 DBG Sling version: 1.4.2 (linux amd64)
2025-02-04 16:46:36 DBG type is file-db
2025-02-04 16:46:36 DBG using: {"columns":[{"name":"*","type":"string"}],"mode":"incremental","select":null,"transforms":null}
2025-02-04 16:46:36 DBG using source options: {"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"zip","format":"csv","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","max_decimals":-1}
2025-02-04 16:46:36 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"}
2025-02-04 16:46:36 INF connecting to target database (snowflake)
2025-02-04 16:46:36 DBG opened "snowflake" connection (conn-snowflake-jVt)
2025-02-04 16:46:36 INF getting checkpoint value
2025-02-04 16:46:37 INF reading from source file system (sftp)
2025-02-04 16:46:37 DBG adding default private key (/home/pika/.ssh/id_rsa) as auth method for SFTP
2025-02-04 16:46:40 DBG opened "sftp" connection (conn-sftp-s0d)
2025-02-04 16:46:40 DBG opened "file" connection (conn-file-yDG)
2025-02-04 16:46:40 DBG closed "snowflake" connection (conn-snowflake-jVt)
2025-02-04 16:46:40 INF execution failed
2025-02-04 16:46:40 WRN Could not successfully get format values. Blank values for: stream_file_name, stream_file_path
2025-02-04 16:46:40 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
sftp: "Filename / is a directory" (SSH_FX_FAILURE)
2025-02-04 16:46:40 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name
2025-02-04 16:46:40 WRN Could not successfully get format values. Blank values for: stream_file_path, stream_file_name


fatal:
--- proc.go:271 main ---
--- sling_cli.go:482 main ---
--- sling_cli.go:518 cliInit ---
--- cli.go:286 CliProcess ---
~ failure running replication (see docs @ https://docs.slingdata.io/sling-cli)
--- sling_run.go:217 processRun ---

--------------------------- /Address_*.csv.zip ---------------------------
--- task_run.go:133 func2 ---
~ could not read from file
--- task_run.go:446 runFileToDB ---
~ Could not FileSysReadDataflow for sftp
--- task_run_read.go:316 ReadFromFile ---
~ could not get zip reader
--- fs.go:558 ReadDataflow ---
~ Unable to open /
--- fs_sftp.go:332 GetReader ---
sftp: "Filename / is a directory" (SSH_FX_FAILURE)

[flarco]

This is due to .zip extension.