Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: parse dates, times, and timestamps #30

Merged
merged 4 commits into from
Oct 17, 2023

Conversation

superhawk610
Copy link
Contributor

@superhawk610 superhawk610 commented Oct 13, 2023

This PR adds support for some Arrow temporal formats - dates, times, and timestamps. I'm primarily using ADBC with Snowflake, which doesn't support the full set of Arrow temporal formats (no timestamps with timezones, durations, or intervals).

These three formats can be tested using the following query:

db_opts = [driver: :snowflake, ..]
{:ok, db} = Adbc.Database.start_link(db_opts)
{:ok, conn} = Adbc.Connection.start_link(database: db)

query = """
select
  '2023-03-01T10:23:45.123456'::timestamp \"datetime\",
  '2023-03-01'::date \"date\",
  '10:23:45.123456'::time \"time\"
"""

Adbc.Connection.query!(conn, query)
# %Adbc.Result{
#   num_rows: 1,
#   data: %{
#     "date" => [~D[2023-03-01]],
#     "datetime" => [~N[2023-03-01 10:23:45.123456]],
#     "time" => [~T[10:23:45.123456]]
#   }
# }

@superhawk610
Copy link
Contributor Author

While working on parsing times, I ran into this Go panic

panic: arrow/array: column "time" type mismatch: got=float64, want=time64[ns]

goroutine 31 [running]:
github.com/apache/arrow/go/v13/arrow/array.NewRecord(0xc000577b00, {0xc0004efa80, 0x4, 0xc0004efa00?}, 0x1)
  /Users/runner/go/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/array/record.go:151 +0x173
github.com/apache/arrow-adbc/go/adbc/driver/snowflake.getRecTransformer.func1({0x113ffcf58, 0xc0008563c0}, {0x11401ab10, 0xc000524900})
  /Users/runner/work/arrow-adbc/arrow-adbc/adbc/go/adbc/driver/snowflake/record_reader.go:65 +0x1ad
github.com/apache/arrow-adbc/go/adbc/driver/snowflake.newRecordReader.func2()
  /Users/runner/work/arrow-adbc/arrow-adbc/adbc/go/adbc/driver/snowflake/record_reader.go:283 +0x1c3
golang.org/x/sync/errgroup.(*Group).Go.func1()
  /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
  /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

@josevalim
Copy link
Member

Hi @superhawk610! I believe the bug you are seeing is this one: apache/arrow-adbc#1021

You can try using v0.7.0-rc and see if it address it. https://github.com/apache/arrow-adbc/releases/tag/apache-arrow-adbc-0.7.0

I will add a guide on updating ADBC, but it should be:

  1. Copy root files and c/ directory from ADBC into 3rdparty
  2. Update the driver version: https://github.com/elixir-explorer/adbc/blob/main/lib/adbc_driver.ex#L9
  3. ...
  4. Profit

Thanks for the PR and let me know if there is anything else we can help with. :)

@superhawk610
Copy link
Contributor Author

That fixed it! Should I check in the driver version upgrade with this PR?

@superhawk610 superhawk610 force-pushed the feat/date-time branch 2 times, most recently from 8465b81 to 2bb5b96 Compare October 13, 2023 19:30
@superhawk610 superhawk610 changed the title [wip] parse dates, times, and timestamps feat: parse dates, times, and timestamps Oct 13, 2023
@superhawk610 superhawk610 marked this pull request as ready for review October 13, 2023 19:30
c_src/adbc_nif.cpp Outdated Show resolved Hide resolved
c_src/adbc_nif.cpp Outdated Show resolved Hide resolved
@josevalim
Copy link
Member

@superhawk610 I will do it in another PR. That's because if it is an external contribution, I need to review all code. :) I will ping you in the PR for double checking.

@josevalim
Copy link
Member

Btw, can you add tests using sqlite or PG? Or should I push them after merging?

@superhawk610
Copy link
Contributor Author

I can add tests early next week!

c_src/adbc_nif.cpp Outdated Show resolved Hide resolved
@josevalim josevalim merged commit 2990fd7 into elixir-explorer:main Oct 17, 2023
1 of 3 checks passed
@josevalim
Copy link
Member

💚 💙 💜 💛 ❤️

@superhawk610 superhawk610 deleted the feat/date-time branch October 17, 2023 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants