diff --git a/NOTICE b/NOTICE index d5061e4093b6..f884d5511464 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache Calcite -Copyright 2012-2023 The Apache Software Foundation +Copyright 2012-2024 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). diff --git a/README b/README index f94b69fc27e3..7d17e6ee4aac 100644 --- a/README +++ b/README @@ -1,4 +1,4 @@ -Apache Calcite release 1.36.0 +Apache Calcite release 1.37.0 This is a source distribution of Apache Calcite. diff --git a/site/_docs/history.md b/site/_docs/history.md index 2e2b8c74bb79..d2ae580fdbe5 100644 --- a/site/_docs/history.md +++ b/site/_docs/history.md @@ -29,25 +29,114 @@ Downloads are available on the [downloads page]({{ site.baseurl }}/downloads/). + +## 1.37.0 / 2024-05-06 +{: #v1-37-0} + +This release comes 5 months after [1.36.0](#v1-36-0), +contains contributions from 46 contributors, and resolves 138 issues. It's worth highlighting the +introduction of adapter for Apache Arrow ([CALCITE-2040]), +StarRocks dialect ([CALCITE-6257]). +The release also added support for lambda expressions in SQL ([CALCITE-3679]), +'Must-filter' columns ([CALCITE-6219]). +For table function calls it is now possible to use them without `TABLE()` wrapper in `FROM` ([CALCITE-6254]). +Furthermore, there is support for optional `FORMAT` of `CAST` operator from SQL:2016 ([CALCITE-6254]) +and more than 15 new SQL functions in various libraries such as BigQuery, PostgreSQL and Spark. + +Contributors to this release: +abhishekagarwal87, +Adam Kennedy, +Alessandro Solimando, +Barry Kelly, +Benchao Li, +Bruno Volpato, +caicancai, +chen768959, +Clint Wylie, +Corvin Kuebler, +Devaspati Krishnatri, +Dmitry Sysolyatin, +Dylan Chen, +Forward Xu, +Francis Chuang, +Hanumath Maduri, +Hongyu Guo, +James Duong, +Jerin John, +Jiajun Xie, +Julian Hyde, +Leonid Chistov, +maweibin, +Mihai Budiu, +Mingcan Wang, +Niels Pardon, +Norman Jordan, +Oliver Lee, +Paul Jackson, +Ran Tao, +Rob D'Hondt, +Ruben Quesada Lopez, +Sergey Nuyanzin (release manager), +Stamatis Zampetakis, +Tanner Clary, +Tim Nieradzik, +TJ Banghart, +Ulrich Kramer, +Will Noble, +xinqiu.hu, +Yingyu Wang, +YiwenWu, +Yubin Li, +Zhengqiang Duan, +zhujiang, +zstan. + #### Breaking Changes {: #breaking-1-37-0} -* In the context of [CALCITE-6015] the visibility of the method -`SqlCall.getCallSignature` has been converted from `protected` to `public`. +* In the context of [CALCITE-6015] +the visibility of the method `SqlCall.getCallSignature` has been converted from `protected` to `public`. Any subclass overriding it will need to be adjusted accordingly. * [CALCITE-6321] Add `copy(List)` method to `Window` class +* As a consequence of the support for lambda expressions + ([CALCITE-3679]) + new methods have been added to `RexVisitor`and `RexBiVisitor`; + any class implementing one of them will have to implement the new methods. Compatibility: This release is tested on Linux, macOS, Microsoft Windows; using JDK/OpenJDK versions 8 to 19; @@ -56,19 +145,291 @@ other software versions as specified in gradle.properties. #### New features {: #new-features-1-37-0} +* Supporting new functions + * [CALCITE-6205] + Add `BITAND_AGG`, `BITOR_AGG` functions (enabled in Snowflake library) + * [CALCITE-6156] + Add `ENDSWITH`, `STARTSWITH` functions (enabled in Postgres, Snowflake libraries) + * [CALCITE-6116] + Add `EXISTS` function (enabled in Spark library) + * [CALCITE-6182] + Add `LENGTH`/`LEN` functions (enabled in Snowflake library) + * [CALCITE-6224] + Add `LOG2` function (enabled in MySQL, Spark library) + * [CALCITE-6223] + Add `MAP_CONTAINS_KEY` function (enabled in Spark library) + * [CALCITE-6314] + Add `RANDOM` function (enabled in Postgres library) + * [CALCITE-6315] + Support PostgreSQL `TO_CHAR`, `TO_DATE`, `TO_TIMESTAMP` + * [CALCITE-6278] + Add `REGEXP`, `REGEXP_LIKE` function (enabled in Spark library) + * [CALCITE-6309] + Add `REGEXP_LIKE` function (enabled in MySQL, Oracle, PostgreSQL and Spark libraries) + * [CALCITE-6179] + Support `WEEKOFMONTH` function format and add test +* [CALCITE-6215] + Support century format datetime/timestamp in pg +* [CALCITE-6268] + Support implementing custom `JdbcSchema` +* [CALCITE-6255] + Support BigQuery-style `JSON_OBJECT` invocation syntax +* [CALCITE-6219] + 'Must-filter' columns +* [CALCITE-2980] + Implement the `FORMAT` clause of the `CAST` operator +* [CALCITE-3679] + Allow lambda expressions in SQL queries +* [CALCITE-3329] + Implement osquery for OS adapter +* [CALCITE-2040] + Create adapter for Apache Arrow +* [CALCITE-6257] + StarRocks dialect implementation +* [CALCITE-6254] + Support table function calls in `FROM` clause without `TABLE()` wrapper +* [CALCITE-6138] + Add parser support for `TIME WITH TIME ZONE` and `TIMESTAMP WITH TIME ZONE` as a data type #### Dependency version upgrade {: #dependency-1-37-0} +* [CALCITE-6124] + Bump json-path from 2.7.0 to 2.8.0 +* [CALCITE-6229] + Bump json-path from 2.8.0 to 2.9.0 +* [CALCITE-6378] + Bump Redis Docker image from 2.8.19 to 7.2.4 +* [CALCITE-6356] + Upgrade Calcite to Avatica 1.25.0 +* [CALCITE-6243] + Upgrade Cassandra to 4.1.3 and DataStax driver for Cassandra to 4.17.0 +* [CALCITE-6181] + Upgrade Janino from 3.1.9 to 3.1.11 +* [CALCITE-6119] + Upgrade testcontainers to 1.19.3 +* [CALCITE-6081] + Remove bouncycastle dependency #### Bug-fixes, API changes and minor enhancements {: #fixes-1-37-0} +* [CALCITE-6355] + `RelToSqlConverter[ORDER BY]` generates an incorrect order by when `NULLS LAST` is used in non-projected field +* [CALCITE-6210] + Cast to `VARBINARY` causes an assertion failure +* [CALCITE-5289] + Assertion failure in `MultiJoinOptimizeBushyRule` +* [CALCITE-6345] + Intervals with more than 100 years are not supported +* [CALCITE-6265] + Type coercion is failing for numeric values in prepared statements (follow-up) +* [CALCITE-6248] + Illegal dates are accepted by casts +* [CALCITE-6282] + Avatica ignores time precision when returning `TIME` results +* [CALCITE-6338] + `RelMdCollation#project` can return an incomplete list of collations in the presence of aliasing +* [CALCITE-5976] + Function `ARRAY_PREPEND`/`ARRAY_APPEND`/`ARRAY_INSERT` gives exception when inserted element type not equals array component type +* [CALCITE-6349] + `CoreRules.PROJECT_REDUCE_EXPRESSIONS` crashes on expressions with `ARRAY_REPEAT` +* [CALCITE-6333] + `NullPointerException` in `AggregateExpandDistinctAggregatesRule.doRewrite` when rewriting filtered distinct aggregation +* [CALCITE-6285] + Function `ARRAY_INSERT` produces an incorrect result for negative indices +* [CALCITE-6015] + `AssertionError` during optimization of `EXTRACT` expression +* [CALCITE-6317] + Incorrect constant replacement when group keys are `NULL` +* [CALCITE-6348] + `ARRAY_OVERLAP` with a `NULL` argument crashes the compiler +* [CALCITE-6347] + `ARRAY_REPEAT` with a string argument causes a compiler crash +* [CALCITE-6127] + The spark array function gives `NullPointerException` when element is row type +* [CALCITE-6074] + The size of `REAL`, `DOUBLE`, and `FLOAT` is not consistent +* [CALCITE-6115] + Interval type specifier with zero fractional second precision does not pass validation +* [CALCITE-5955] + BigQuery `PERCENTILE` functions are unparsed incorrectly +* [CALCITE-6048] + `ServerTest#testTruncateTable` fails intermittently due to method not found exception +* [CALCITE-5811] + Error messages produced for constant out-of-bounds arguments are confusing +* [CALCITE-6128] + `RelBuilder.limit` should apply offset and fetch to previous Sort operator, if possible +* [CALCITE-6118] + Missing empty `ARRAY` function usage in reference doc +* [CALCITE-6121] + Invalid unparse for `TIMESTAMP` with `SparkSqlDialect` +* [CALCITE-6109] + Linq4j `OptimizeShuttle` should not create new instances of `TernaryExpression` if it does not do any optimization +* [CALCITE-6095] + Arithmetic expression with `VARBINARY` value causes AssertionFailure +* [CALCITE-6150] + JDBC adapter for ClickHouse generates incorrect SQL for certain units in the `EXTRACT` function +* [CALCITE-6117] + Converting `SAFE_CAST` from `RexCall` to `SqlCall` fails to add the type as an argument +* [CALCITE-6211] + `SUBSTRING` with `Integer.MIN_VALUE` as a second parameter raise unexpected exception +* [CALCITE-6213] + The default behavior of `NullCollation` in Presto is `LAST` +* [CALCITE-6227] + `ELEMENT(NULL)` causes an assertion failure +* [CALCITE-6168] + `RexExecutor` can throw during compilation +* [CALCITE-5130] + `AssertionError`: "Conversion to relational algebra failed to preserve datatypes" when union `VARCHAR` literal and `CAST(null AS INTEGER)` +* [CALCITE-6178] + `WITH RECURSIVE` query when cloned using `SqlShuttle` looses `RECURSIVE` property +* [CALCITE-6332] + Optimization `CoreRules.AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN` produces incorrect results for aggregates with groupSets +* [CALCITE-6353] + Optimization `CoreRules.PROJECT_REDUCE_EXPRESSIONS` crashes while optimizing `ARRAY_CONCAT` expression +* [CALCITE-6262] + `CURRENT_TIMESTAMP(P)` ignores `DataTypeSystem#getMaxPrecision` +* [CALCITE-6283] + Function `ARRAY_APPEND` with a `NULL` array argument crashes with `NullPointerException` +* [CALCITE-6306] + JDBC adapter should not generate `FILTER` (`WHERE`) in MySQL and StarRocks dialect +* [CALCITE-5893] + Wrong `NULL` operand behavior of `ARRAY_CONTAINS`/`ARRAY_EXCEPT`/`ARRAY_INTERSECT` In Spark Library +* [CALCITE-6290] + Incorrect return type for BigQuery `TRUNC` +* [CALCITE-6252] + BigQuery `FORMAT_DATE` uses the wrong calendar for Julian dates +* [CALCITE-6214] + Remove `DISTINCT` in aggregate function if field is unique +* [CALCITE-6258] + Map value constructor is unparsed incorrectly for `PrestoSqlDialect` +* [CALCITE-6249] + `RelNode::estimatedRowCount` should not be used in `computeSelfCost` +* [CALCITE-6251] + `InnerEnumerator` in `EnumerableDefaults::correlateBatchJoin` is not closed +* [CALCITE-6247] + BigQuery `FORMAT_DATE` function handles incorrectly the `%e` format specifier +* [CALCITE-6238] + Exception while evaluating `ROUND`/`TRUNCATE` functions +* [CALCITE-6228] + `ELEMENT` function infers incorrect return type +* [CALCITE-5647] + `RelMdPopulationSize` should use `mq.getRowCount(rel)` instead of `rel.estimateRowCount(mq)` +* [CALCITE-6241] + Enable a few existing functions to Spark library +* [CALCITE-6094] + `Linq4j.ConstantExpression.write` crashes on special FP values +* [CALCITE-6190] + Incorrect precision derivation for negative numeric types +* [CALCITE-6202] + `sqlsh` does not print error message when query fails +* [CALCITE-6200] + `RelJson` throw `UnsupportedOperationException` for `RexDynamicParam` +* [CALCITE-6044] + `RelMetadataQuery` should regard single-row relational expressions as unique +* [CALCITE-5846] + Preserve filters on non-distinct aggCalls in `AggregateExpandWithinDistinctRule` +* [CALCITE-6100] + The `equalsDeep` of `SqlRowTypeNameSpec` should use `equalsDeep` for fieldTypes rather than reference comparison +* [CALCITE-6183] + The second parameter of `RexProgramBuilder#registerInternal` is always false +* [CALCITE-6149] + Unparse for `CAST` Nullable with `ClickHouseSqlDialect` +* [CALCITE-5649] + Get row count statistics from `ReflectiveSchema` +* [CALCITE-6220] + Rewrite `MIN`/`MAX(bool)` as `BOOL_AND`/`BOOL_OR` for Postgres, Redshift +* [CALCITE-6321] + Add `copy(List)` method to Window class +* [CALCITE-6337] + Distinguish naked measure support between inside and outside aggregation +* [CALCITE-6323] + Serialize return type during `RelJson.toJson(RexNode node)` for `SqlKind.SAFE_CAST` +* [CALCITE-6111] + Explicit cast from expression to numeric type doesn't check overflow +* [CALCITE-6162] + Add rule(s) to remove joins with constant single tuple relations +* [CALCITE-6192] + `DEFAULT` expression with `NULL` value throws unexpected exception +* [CALCITE-6147] + `CAST(CAST(EMPNO AS VARCHAR) AS INT)` should be simplified to `EMPNO` +* [CALCITE-6102] + `SqlWriter` in `SqlInsert`'s unparse start a list but does not end it +* [CALCITE-5607] + Serialize return type during `RelJson.toJson(RexNode node)` for `SqlKind.MINUS` +* [CALCITE-6269] + Fix missing/broken BigQuery date-time format elements +* [CALCITE-6231] + JDBC adapter generates `UNNEST` when it should generate `UNNEST ... WITH ORDINALITY` +* [CALCITE-6208] + Update `JSON_VALUE` return type inference to make explicit array return types be nullable with nullable elements +* [CALCITE-6199] + Trim unused fields for `SNAPSHOT` and `SAMPLE` if table has `VIRTUAL` column +* [CALCITE-6063] + If `ARRAY` subquery has `ORDER BY` (without `LIMIT`), rows are not sorted +* [CALCITE-6032] + Multilevel correlated query is failing in `RelDecorrelator` code path #### Build and test suite {: #build-1-37-0} +* [CALCITE-6103] + Use eclipse-temurin image to build and publish javadocs for the website +* [CALCITE-6131] + There are duplicate sample tests in `SqlTypeUtilTest` +* [CALCITE-6125] + Automate generation of contributor names in release notes by adding a git mailmap file +* [CALCITE-6165] + Add `DATE_ADD` test and `DATE_DIFF` test on `SqlOperatorTest` +* [CALCITE-6184] + Add `checkNullTest` on `SqlOperatorTest` +* [CALCITE-6187] + Linter should disallow tags such as `[MINOR]` in commit messages +* [CALCITE-6273] + Add sqrt negative test in `SqlOperatorTest` +* [CALCITE-6189] + Improve `FormatElementEnumTest` +* [CALCITE-6234] + Add tests on `SqlOperatorTest` for `to_char` function +* [CALCITE-6172] + Allow aliased operators to re-use existing tests +* [CALCITE-6359] + Update GitHub Actions workflows to use docker compose v2 +* [CALCITE-6092] + Skip breaking `CAST` String to `TIME` tests until fixed in Avatica 1.24.0 +* [CALCITE-6384] + Add ASF header to `buildcache.yml`, `gradle-wrapper-validation.yml` +* [CALCITE-6385] + LintTest fails when run in source distribution +* [CALCITE-6387] + Make Arrow adapter passing tests with jdk17+ +* [CALCITE-6390] + Exclude Arrow project on Windows builds #### Web site and documentation {: #site-1-37-0} ---> +* Site: Switch PMC Chair to Benchao Li +* Site: Troubleshooting/Website publishing improvements in release guide +* Site: Remove committer by request process +* [CALCITE-6083] + On web site, ensure contributors file is sorted +* [CALCITE-6098] + Update `LICENSE` and `NOTICE` for Jekyll website template +* [CALCITE-6250] + Limitations of MongoDB adapter are not documented +* [CALCITE-6256] + Incorrect rendering of HTML on InnoDB adapter page +* Add `.gitignore` and `.ratignore` for jenv +* [CALCITE-6097] + Gridism CSS dependency is mispelled in `LICENSE` +* [CALCITE-6096] + Remove obsolete html5shiv and respond entries from `LICENSE` +* [CALCITE-6194] + Contributor rules do not give instructions about how to quote commits +* [CALCITE-6212] + Config `locale = 'en_US'` for javadoc task +* [CALCITE-6316] + Update Javadoc for `RelWriterTest#testDeserializeMinusDateOperator` +* [CALCITE-6105] + Documentation does not specify the behavior of `SPLIT` function for empty string arguments ## 1.36.0 / 2023-11-10 {: #v1-36-0} diff --git a/site/_docs/howto.md b/site/_docs/howto.md index 15a7c6c3d8fa..26571eb5b363 100644 --- a/site/_docs/howto.md +++ b/site/_docs/howto.md @@ -31,7 +31,7 @@ adapters. ## Building from a source distribution -Prerequisite is Java (JDK 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 or 19) +Prerequisite is Java (JDK 8, 11, 17 or 19) and Gradle (version 7.6.1) on your path. Unpack the source distribution `.tar.gz` file, @@ -39,8 +39,8 @@ Unpack the source distribution `.tar.gz` file, then build using Gradle: {% highlight bash %} -$ tar xvfz apache-calcite-1.36.0-src.tar.gz -$ cd apache-calcite-1.36.0-src +$ tar xvfz apache-calcite-1.37.0-src.tar.gz +$ cd apache-calcite-1.37.0-src $ gradle build {% endhighlight %} @@ -51,7 +51,7 @@ tests (but you should use the `gradle` command rather than ## Building from Git Prerequisites are git -and Java (JDK 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 or 19) on your path. +and Java (JDK 8, 11, 17 or 19) on your path. Create a local copy of the GitHub repository, `cd` to its root directory,