Skip to content

datafusion-contrib/datafusion-functions-extra

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

author
jatin
Sep 27, 2024
23ffedd · Sep 27, 2024

History

10 Commits
Sep 23, 2024
Sep 23, 2024
Sep 23, 2024
Sep 27, 2024
Sep 27, 2024
Sep 23, 2024
Sep 23, 2024
Sep 23, 2024
Sep 23, 2024
Sep 26, 2024
Sep 17, 2024
Sep 27, 2024
Sep 23, 2024

Repository files navigation

datafusion-functions-extra

CI

Note: This is not an official Apache Software Foundation release.

This crate provides extra functions for DataFusion, specifically focusing on advanced aggregations. These extensions are inspired by other projects like DuckDB and Spark SQL.

To use these functions, you'll just need to call:

datafusion_functions_extra::register_all_extra_functions(&mut ctx)?;

Examples

-- Create a table with various columns containing strings, integers, floats, dates, and times
CREATE TABLE test_table (
    utf8_col VARCHAR,
    int64_col INT,
    float64_col FLOAT,
    date64_col DATE,
    time64_col TIME
) AS VALUES
('apple', 1, 1.0, '2021-01-01', '01:00:00'),
('banana', 2, 2.0, '2021-01-02', '02:00:00'),
('apple', 2, 2.0, '2021-01-02', '02:00:00'),
('orange', 3, 3.0, '2021-01-03', '03:00:00'),
('banana', 3, 3.0, '2021-01-03', '03:00:00'),
('apple', 3, 3.0, '2021-01-03', '03:00:00');

-- Get the mode of the utf8_col column
SELECT mode(utf8_col) as mode_utf8 FROM test_table;
-- Results in
-- +----------+
-- | mode_utf8|
-- +----------+
-- | apple    |
-- +----------+

-- Get the mode of the date64_col column
SELECT mode(date64_col) as mode_date FROM test_table;
-- Results in
-- +-----------+
-- | mode_date |
-- +-----------+
-- | 2021-01-03|
-- +-----------+

-- Get the mode of the time64_col column
SELECT mode(time64_col) as mode_time FROM test_table;
-- Results in
-- +-----------+
-- | mode_time |
-- +-----------+
-- | 03:00:00  |
-- +-----------+

-- Get the x value associated with the maximum y value
SELECT max_by(x, y) FROM VALUES (1, 10), (2, 5), (3, 15), (4, 8) as tab(x, y);
-- Results in
-- +---------------------+
-- | max_by(tab.x,tab.y) |
-- +---------------------+
-- | 3                   |
-- +---------------------+

-- Get the x value associated with the minimum y value
SELECT min_by(x, y) FROM VALUES (1, 10), (2, 5), (3, 15), (4, 8) as tab(x, y);
-- Results in
-- +---------------------+
-- | min_by(tab.x,tab.y) |
-- +---------------------+
-- | 2                   |
-- +---------------------+

-- Get the kurtosis value of a column of numeric data.
SELECT kurtosis(col) FROM VALUES (1.0), (10.0), (100.0), (10.0), (1.0) as tab(col);
-- Results in
- +-------------------+
- "| kurtosis(tab.col) |"
- +-------------------+
- "| 4.777292927667962 |"
- +-------------------+

Done

  • mode(expression) -> scalar - Returns the most frequent (mode) value from a column of data.

  • max_by(expression1, expression2) -> scalar - Returns the value of expression1 associated with the maximum value of expression2.

  • min_by(expression1, expression2) -> scalar - Returns the value of expression1 associated with the minimum value of expression2.

  • kurtois_pop(expression) -> scalar - Computes the excess kurtosis (Fisher’s definition) without bias correction.

  • kurtosis(expression) -> scalar - Returns the kurtosis, a measure of the tailedness of the distribution, for the given numeric values in expression.