Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for decoding list views and large list views #89

Merged
merged 2 commits into from
Jun 2, 2024

Conversation

cocoa-xu
Copy link
Member

@cocoa-xu cocoa-xu commented Jun 2, 2024

This PR adds support for decoding list views and large list views.

The result by default is represented in the following format, where data is a map consisting 4 key-value pairs:

  • validity, indicating if the corresponding child is valid or not (nil for invalid ones)
  • offsets, the index where first element of the child list in values.data
  • sizes, the length of the corresponding child
  • values, the underlying values
iex> list_view = %Adbc.Column{
  name: "sample_list_view",
  type: :list_view,
  nullable: true,
  metadata: %{},
  data: %{
    validity: [true, false, true, true, true],
    offsets: [4, 7, 0, 0, 3],
    sizes: [3, 0, 4, 0, 2],
    values: %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: %{},
      data: [0, -127, 127, 50, 12, -7, 25]
    }
  }
}

To convert from the compact form to lists, users can use

iex> Adbc.Column.list_view_to_list(list_view)
%Adbc.Column{
  name: "sample_list_view",
  type: :list,
  nullable: true,
  metadata: %{},
  data: [
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: [12, -7, 25]
    },
    nil,
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: [0, -127, 127, 50]
    },
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: []
    },
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: ~c"2\f"
    }
  ]
}

The example is taken from https://arrow.apache.org/docs/format/Columnar.html#listview-layout

@cocoa-xu cocoa-xu merged commit 0ef3f0f into main Jun 2, 2024
3 checks passed
@cocoa-xu cocoa-xu deleted the cx-list-view branch June 2, 2024 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants