{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":494169982,"defaultBranch":"master","name":"delta","ownerLogin":"nkarpov","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2022-05-19T17:35:08.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/1390267?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1692124518.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"47f58fd63d87a90643ecf868e53afaccf8e47132","ref":"refs/heads/kernel_ug_package_name","pushedAt":"2023-08-15T18:35:18.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"nkarpov","name":"Nick","path":"/nkarpov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1390267?s=80&v=4"},"commit":{"message":"fix maven package name","shortMessageHtmlLink":"fix maven package name"}},{"before":"5ad64436ecc3218d6dc3c079afbfc0dc70325500","after":"5e90a97e0c4c0eed9dfca895c39658ff046ea091","ref":"refs/heads/master","pushedAt":"2023-08-15T18:32:16.000Z","pushType":"push","commitsCount":501,"pusher":{"login":"nkarpov","name":"Nick","path":"/nkarpov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1390267?s=80&v=4"},"commit":{"message":"[Kernel] API user guide\n\n## Description\r\nThis PR adds `USER_GUIDE.md` which explains how to use the Delta Kernel APIs to build Delta Lake connectors.","shortMessageHtmlLink":"[Kernel] API user guide"}},{"before":"d74cc6897730f4effb5d7272c21bd2554bdfacdb","after":"5ad64436ecc3218d6dc3c079afbfc0dc70325500","ref":"refs/heads/master","pushedAt":"2023-05-27T00:31:53.386Z","pushType":"push","commitsCount":33,"pusher":{"login":"nkarpov","name":"Nick","path":"/nkarpov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1390267?s=80&v=4"},"commit":{"message":"Handle FileAlreadyExistsException in S3DynamoDBLogStore\n\n## Description\n\nResolves #1734\n\nIt is possible that more than one concurrent reader/writers will try to fix the same incomplete entry in DynamoDB. This could result in some seeing a `FileAlreadyExistsException` when trying to copy the temp file to the delta log. We should not propagate this error to the end user since the recovery operation was successful and we can proceed. See #1734 for more details.\n\nNote, we attempt to copy a temp file in two places:\n1. As part of writing N.json [here](https://github.com/delta-io/delta/blob/master/storage-s3-dynamodb/src/main/java/io/delta/storage/BaseExternalLogStore.java#L249)\n2. In `fixDeltaLog` when performing a recovery for N-1.json as part of either a write or listFrom\n\nWe only need to catch the exception in scenario (2). In scenario (1) we already catch _all_ seen errors.\n\nThis is hard to test without manipulating the FS + external store a lot. We could manipulate `FailingFileSystem` to throw a `FileAlreadyExistsException`.\n\nCloses delta-io/delta#1776\n\nSigned-off-by: Allison Portis <allison.portis@databricks.com>\nGitOrigin-RevId: fcce5d5577d79dff4d071ebdd63b3ee837e5b645","shortMessageHtmlLink":"Handle FileAlreadyExistsException in S3DynamoDBLogStore"}},{"before":"5c3f4d37951ab4edf1cc3364ec6e8259844b331c","after":"d74cc6897730f4effb5d7272c21bd2554bdfacdb","ref":"refs/heads/master","pushedAt":"2023-05-11T19:37:17.459Z","pushType":"push","commitsCount":20,"pusher":{"login":"nkarpov","name":"Nick","path":"/nkarpov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1390267?s=80&v=4"},"commit":{"message":"Fix a test in DeltaVacuumSuite to pass locally\n\n\"vacuum after purging deletion vectors\" in `DeltaVacuumSuite` fails locally because the local filesystem only writes modification times to second accuracy.\n\nThis means a transaction might have timestamp `1683694325000` but the tombstone for a file removed in that transaction could have deletionTimestamp `1683694325372`.\n\n---> The test fails since we set the clock to the transaction timestamp + retention period, which isn't late enough to expire the tombstones in that transaction.\n\nGitOrigin-RevId: 63018c48524edb0f8edd9e40f1b21cc97bc546cc","shortMessageHtmlLink":"Fix a test in DeltaVacuumSuite to pass locally"}},{"before":"1c18c1d972e37d314711b3a485e6fb7c98fce96d","after":"5c3f4d37951ab4edf1cc3364ec6e8259844b331c","ref":"refs/heads/master","pushedAt":"2023-05-09T15:43:37.678Z","pushType":"push","commitsCount":133,"pusher":{"login":"nkarpov","name":"Nick","path":"/nkarpov","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1390267?s=80&v=4"},"commit":{"message":"Support Spark 3.4\n\n## Description\n\nMakes changes to support Spark 3.4. These include compile necessary changes, and test _and_ code changes due to changes in Spark behavior.\n\nSome of the bigger changes include\n- A lot of changes regarding error classes. These include...\n  - Spark 3.4 changed `class ErrorInfo` to private. This means the current approach in `DeltaThrowableHelper` can no longer work. We now use `ErrorClassJsonReader` (these are the changes to `DeltaThrowableHelper` and `DeltaThrowableSuite`\n  - Many error functions switched the first argument from `message: String` to `errorClass: String` which **does not** cause a compile error, but instead causes a \"SparkException-error not found\" when called. Some things affected include `ParseException(...)`, `a.failAnalysis(..)`.\n  - Supports error subclasses\n- Spark 3.4 supports insert-into-by-name and no longer reorders such queries to be insert-into-by-ordinal. See https://github.com/apache/spark/pull/39334. In `DeltaAnalysis.scala` we need to perform schema validation checks and schema evolution for such queries; right now we only match when `!isByName`\n- SPARK-27561 added support for lateral column alias. This broke our generation expression validation checks for generated columns. We now separately check for generated columns that reference other generated columns in `GeneratedColumn.scala`\n- `DelegatingCatalogExtension` deprecates `createTable(..., schema: StructType, ...)` in favor of `createTable(..., columns: Array[Column], ...)`\n- `_metadata.file_path` is not always encoded. We update `DeleteWithDeletionVectorsHelper.scala` to accomodate for this.\n- Support for SQL `REPLACE WHERE`. Tests are added to `DeltaSuite`.\n-  Misc test changes due to minor changes in Spark behavior or error messages\n\nResolves delta-io/delta#1696\n\nExisting tests should suffice since there are no major Delta behavior changes _besides_ support for `REPLACE WHERE` for which we have added tests.\n\n## Does this PR introduce _any_ user-facing changes?\n\nYes. Spark 3.4 will be supported. `REPLACE WHERE` is supported in SQL.\n\nGitOrigin-RevId: b282c95c4e6a7a1915c2a4ae9841b5e43ed4724d","shortMessageHtmlLink":"Support Spark 3.4"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wOC0xNVQxODozNToxOC4wMDAwMDBazwAAAANrSmlE","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wNS0wOVQxNTo0MzozNy42Nzg4MzdazwAAAAMpL4Na"}},"title":"Activity · nkarpov/delta"}