-
Notifications
You must be signed in to change notification settings - Fork 109
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Website: Add blog post for 17.0.0 (#537)
Release blog post for 17.0.0. Release notes are here: https://arrow.apache.org/release/17.0.0.html Issues on the milestone are here: https://github.com/apache/arrow/milestone/62?closed=1 --------- Co-authored-by: David Li <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Felipe Oliveira Carvalho <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Bryce Mecum <[email protected]> Co-authored-by: Adam Reeve <[email protected]> Co-authored-by: Joel Lubinitsky <[email protected]> Co-authored-by: Dane Pitkin <[email protected]> Co-authored-by: Rossi Sun <[email protected]>
- Loading branch information
12 people
authored
Jul 19, 2024
1 parent
cd734aa
commit ab4042b
Showing
1 changed file
with
248 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| --- | ||
| layout: post | ||
| title: "Apache Arrow 17.0.0 Release" | ||
| date: "2024-07-16 00:00:00" | ||
| author: pmc | ||
| categories: [release] | ||
| --- | ||
| <!-- | ||
| {% comment %} | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to you under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| {% endcomment %} | ||
| --> | ||
|
|
||
|
|
||
| The Apache Arrow team is pleased to announce the 17.0.0 release. This covers | ||
| over 3 months of development work and includes [**331 resolved issues**][1] | ||
| on [**529 distinct commits**][2] from [**92 distinct contributors**][2]. | ||
| See the [Install Page](https://arrow.apache.org/install/) | ||
| to learn how to get the libraries for your platform. | ||
|
|
||
| The release notes below are not exhaustive and only expose selected highlights | ||
| of the release. Many other bugfixes and improvements have been made: we refer | ||
| you to the [complete changelog][3]. | ||
|
|
||
| ## Community | ||
|
|
||
| Since the 16.0.0 release, Dane Pitkin has been invited to be committer. | ||
| No new members have joined the Project Management Committee (PMC). | ||
|
|
||
| Thanks for your contributions and participation in the project! | ||
|
|
||
| ## Linux packages notes | ||
|
|
||
| - We dropped support for Debian GNU/Linux bullseye | ||
|
|
||
| ## C Data Interface notes | ||
|
|
||
| - `ArrowDeviceArrayStream` can now be imported and exported (GH-40078) | ||
|
|
||
| ## Arrow Flight RPC notes | ||
|
|
||
| - Flight SQL was formally stabilized (GH-39204). | ||
| - Flight SQL added a bulk ingestion command (GH-38255). | ||
| - The JDBC Flight SQL driver now accepts "catalog" as a connection parameter (GH-41947). | ||
| - "Stateless" prepared statements are now supported (GH-37220, GH-41262). | ||
| - Java added `FlightStatusCode.RESOURCE_EXHAUSTED` (GH-35888). | ||
| - C++ has some basic support for logging with OpenTelemetry (GH-39898). | ||
|
|
||
| ## C++ notes | ||
|
|
||
| For C++ notes refer to the full changelog. | ||
|
|
||
| ### Highlights | ||
|
|
||
| - Half-float values can now be parsed and formatted correctly (GH-41089). | ||
| - Record batches can now be converted to row-major tensors, not only column-major (GH-40866). | ||
| - The CSV writer is now able to write large string arrays that are larger than | ||
| 2 GiB (GH-40270). | ||
| - A possible invalid memory access in `BooleanArray.true_count()` has been fixed (GH-41016). | ||
| - A new method `FlattenRecursively` allows recursive nesting of list and | ||
| fixed-size list arrays (GH-41055). | ||
| - The scratch space in some `Scalar` subclasses is now immutable. This is required | ||
| for proper concurrent access to `Scalar` instances (GH-40069). | ||
| - Calling the `bit_width` or `byte_width` method of an extension type now defers | ||
| to the underlying storage type (GH-41353). | ||
| - Fixed a bug where `MapArray::FromArrays` would behave incorrectly if the given | ||
| offsets array has a non-zero offset (GH-40750). | ||
| - `MapArray::FromArrays` now accepts an optional null bitmap argument | ||
| (GH-41684). | ||
| - The `ARROW_NO_DEPRECATED_API` macro was unused and has been removed (GH-41343). | ||
|
|
||
| ### Acero | ||
|
|
||
| - The left anti join filter no longer crashes when the filter rows are empty (GH-41121). | ||
| - A race condition was fixed in the asof join (GH-41149). | ||
| - A potential stack overflow has been fixed (GH-41334, GH-41738). | ||
| - A potential crash on very large data has been fixed (GH-41813). | ||
| - Asof join and sort merge join now support single threaded mode (GH-41190). | ||
|
|
||
| ### Compute | ||
|
|
||
| - List views and maps are now supported by the `if_else`, `case_when` and | ||
| `coalesce` functions (GH-41418). | ||
| - List views are now supported by the functions `list_slice` (GH-42065), | ||
| `list_parent_indices` (GH-42235), `take` and `filter` (GH-42116). | ||
| - `list_flatten` can now be recursive based on new optional argument | ||
| (GH-41183, GH-41055) | ||
| - The `take` and `filter` functions have been made significantly faster on fixed-width | ||
| types, including fixed-size lists of fixed-width types (GH-39798). | ||
|
|
||
| ### Dataset | ||
|
|
||
| - Repeated scanning of an encrypted Parquet dataset now works correctly (GH-41431). | ||
|
|
||
| ### Filesystems | ||
|
|
||
| - Standard filesystem implementations are now tracked in a global registry which | ||
| also allows loading third-party filesystem implementations, for example from | ||
| runtime-loaded DLLs (GH-40342, | ||
| - Directory metadata operations on Azure filesystems are now more aligned with | ||
| the common expectations for filesystems (GH-41034). | ||
| - `CopyFile` is now supported for Azure filesystems with hierarchical namespace | ||
| enabled (GH-41095). | ||
| - Azure credentials can now be loaded explicitly from the environment (GH-39345), | ||
| or using the Azure CLI (GH-39344). | ||
| - A potential deadlock was fixed when closing an S3 output stream (GH-41862). | ||
|
|
||
| ### GPU | ||
|
|
||
| - Non-CPU data can now be pretty-printed (GH-41664). | ||
| - Non-CPU data with offsets, such as list and binary data, can now be properly | ||
| sent over IPC (GH-42198). | ||
|
|
||
| ### IPC | ||
|
|
||
| - Flatbuffers serialization is now more deterministic (GH-40361). | ||
|
|
||
| ### Parquet | ||
|
|
||
| - A crash was fixed when reading an invalid Parquet file where columns claim to | ||
| be of different lengths (GH-41317). | ||
| - Definition and repetition levels are now more strictly checked, avoiding later | ||
| crashes when reading an invalid Parquet file (GH-41321). | ||
| - A crash was fixed when reading an invalid encrypted Parquet file (GH-43070). | ||
| - Fixed a bug where `DeltaLengthByteArrayEncoder::EstimatedDataEncodedSize` could | ||
| return an invalid estimate in some situations (GH-41545). | ||
| - Delimiting records is now faster for columns with nested repeating (GH-41361). | ||
|
|
||
| ### Substrait | ||
|
|
||
| - Support for more Arrow data types was added: some temporal types, half floats, | ||
| large string and large binary (GH-40695). | ||
|
|
||
| ## C# notes | ||
|
|
||
| - The performance of building Decimal arrays using SqlDecimal values was improved for .NET 7+ (GH-41349) | ||
| - Scalar arrays now implement `ICollection<T?>` (GH-38692) | ||
| - Concatenating arrays with a non-zero offset with ArrowArrayConcatenator was fixed (GH-41164) | ||
| - Concatenating union arrays with ArrowArrayConcatenator was fixed (GH-41198) | ||
| - Accessing values of decimal arrays with a non-zero offset was fixed (GH-41199) | ||
|
|
||
| ## Go Notes | ||
|
|
||
| ### Bug Fixes | ||
|
|
||
| #### Arrow | ||
|
|
||
| - Prevent exposure of invalid Go pointers in CGO code ([GH-43062](https://github.com/apache/arrow/issues/43062)) | ||
| - Fix memory leak for 0-length C array imports ([GH=41534](https://github.com/apache/arrow/issues/41534)) | ||
| - Ensure statement handle is updated so stateless prepared statements work properly ([GH-41427](https://github.com/apache/arrow/issues/41427)) | ||
|
|
||
| #### Parquet | ||
|
|
||
| - Fix memory leak in BufferedPageWriter ([GH-41697](https://github.com/apache/arrow/issues/41697)) | ||
| - Fix performance regression in PooledBufferWriter ([GH-41541](https://github.com/apache/arrow/issues/41541)) | ||
|
|
||
| ### Enhancements | ||
|
|
||
| #### Arrow | ||
|
|
||
| - Arrow Schemas and Records can now be created from Protobuf messages ([GH-40494](https://github.com/apache/arrow/issues/40494)) | ||
|
|
||
| #### Parquet | ||
|
|
||
| - Performance improvement for BitWriter VlqInt ([GH-41160](https://github.com/apache/arrow/pull/41160)) | ||
|
|
||
| ## Java notes | ||
|
|
||
| **Some changes are coming up in the next version, Arrow 18. Java 8 support will be removed. The version of the flight-core artifact with shaded gRPC will no longer be distributed.** | ||
|
|
||
| - Basic support for ListView (GH-41287) and StringView (GH-40339) has been added. These types should still be considered experimental. | ||
|
|
||
| ## JavaScript notes | ||
|
|
||
| - General maintenance. Clean up packaging ([GH-39722](https://github.com/apache/arrow/issues/39722)), update dependencies ([GH-41905](https://github.com/apache/arrow/issues/41905)). | ||
|
|
||
| ## Python notes | ||
|
|
||
| ### Compatibility notes: | ||
| * To ensure Python 3.13 compatibility, _Py_IsFinalizing has been replaced with a public API (GH-41475). | ||
| * The C Data Interface now supports CUDA devices (GH-40384). | ||
|
|
||
| ### New features: | ||
| * Added support for Emscripten via Pyodide (GH-41910). | ||
|
|
||
| ### Other improvements: | ||
| * The ParquetWriter added the store_decimal_as_integer option (GH-42168). | ||
| * The Float16 logical type is supported in Parquet (GH-42016). | ||
| * Exposed bit_width and byte_width to extension types (GH-41389). | ||
| * Added bindings for Device and MemoryManager classes (GH-41126). | ||
| * The PyCapsule interface now exposes the device interface (GH-38325). | ||
| * Various PyArrow APIs have been updated to work with non-CPU architectures gracefully. (GH-42112, GH-41664, GH-41662, | ||
|
|
||
| ### Relevant bug fixes: | ||
| * Fixed Numpy 2.0 compatibility issues (GH-42170, GH-41924). | ||
| * Fixed sporadic as_of join failures (GH-41149). | ||
| * Fixed a bug in RecordBatch.filter() when passing a ChunkedArray, which would cause a segfault (GH-38770). | ||
| * Fixed a bug in RecordBatch.from_arrays() when passing a storage array, which would cause a segfault (GH-37669). | ||
| * Fixed a bug where constructing a MapArray from Array could drop nulls (GH-41684). | ||
| * FIxed a bug where RunEndEncodedArray.from_arrays fails if run_ends are pyarrow.Array (GH-40560). | ||
| * FIxed a regression introduced in PyArrow v16 in RecordBatchReader.cast() (GH-41884). | ||
|
|
||
| ## R notes | ||
|
|
||
| * R functions that users write that use functions that Arrow supports in dataset queries now can be used in queries too. Previously, only functions that used arithmetic operators worked. For example, `time_hours <- function(mins) mins / 60` worked, but `time_hours_rounded <- function(mins) round(mins / 60)` did not; now both work. These are automatic translations rather than true user-defined functions (UDFs); for UDFs, see `register_scalar_function()`. [GH-41223](https://github.com/apache/arrow/issues/41223) | ||
| * `mutate()` expressions can now include aggregations, such as `x - mean(x)`. [GH-41350](https://github.com/apache/arrow/pull/41350) | ||
| * `summarize()` supports more complex expressions, and correctly handles cases where column names are reused in expressions. [GH-41323](https://github.com/apache/arrow/issues/41323) | ||
| * The `na_matches` argument to the `dplyr::*_join()` functions is now supported. This argument controls whether `NA` values are considered equal when joining. [GH-41223](https://github.com/apache/arrow/issues/41358) | ||
| * R metadata, stored in the Arrow schema to support round-tripping data between R and Arrow/Parquet, is now serialized and deserialized more strictly. This makes it safer to load data from files from unknown sources into R data.frames. [GH-41223](https://github.com/apache/arrow/issues/41969) | ||
|
|
||
| For more on what’s in the 17.0.0 R package, see the [R changelog][4]. | ||
|
|
||
| ## Ruby and C GLib notes | ||
|
|
||
| ### Ruby | ||
|
|
||
| - Improved `Arrow::Table#to_s` format | ||
| - This is a breaking change | ||
|
|
||
| ### C GLib | ||
|
|
||
| - Added support for Microsoft Visual C++ | ||
| - Added `gadataset_dataset_to_record_batch_reader()` | ||
|
|
||
| ## Rust notes | ||
|
|
||
| The Rust projects have moved to separate repositories outside the | ||
| main Arrow monorepo. For notes on the latest release of the Rust | ||
| implementation, see the latest [Arrow Rust changelog][5]. | ||
|
|
||
| [1]: https://github.com/apache/arrow/milestone/62?closed=1 | ||
| [2]: {{ site.baseurl }}/release/17.0.0.html#contributors | ||
| [3]: {{ site.baseurl }}/release/17.0.0.html#changelog | ||
| [4]: {{ site.baseurl }}/docs/r/news/ | ||
| [5]: https://github.com/apache/arrow-rs/tags |