diff --git a/404.html b/404.html index 2ce110b91..9a09f6883 100644 --- a/404.html +++ b/404.html @@ -302,7 +302,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/assets/img/logos/red_hat_openshift_ai.png b/assets/img/logos/red_hat_openshift_ai.png new file mode 100644 index 000000000..c3df4c435 Binary files /dev/null and b/assets/img/logos/red_hat_openshift_ai.png differ diff --git a/assets/img/red-hat/OpenShiftAIDemoArchitecture.png b/assets/img/red-hat/OpenShiftAIDemoArchitecture.png new file mode 100644 index 000000000..946c5aaba Binary files /dev/null and b/assets/img/red-hat/OpenShiftAIDemoArchitecture.png differ diff --git a/assets/img/red-hat/OpenShiftDeploymentArchitecture.png b/assets/img/red-hat/OpenShiftDeploymentArchitecture.png new file mode 100644 index 000000000..c1ae3a5ba Binary files /dev/null and b/assets/img/red-hat/OpenShiftDeploymentArchitecture.png differ diff --git a/assets/js/search-data.json b/assets/js/search-data.json index 45dd6f8a0..17fb5c2ed 100644 --- a/assets/js/search-data.json +++ b/assets/js/search-data.json @@ -2248,7 +2248,7 @@ },"321": { "doc": "Integrations", "title": "Integrations with lakeFS", - "content": "You can use lakeFS with a wide range of tools and frameworks. lakeFS provides several clients directly, as well as an S3-compatible gateway. This gateway means that if you want to use something with lakeFS, so long as that technology can interface with S3, it can interface with lakeFS. See below for detailed instructions for using different technologies with lakeFS. | Airbyte | Amazon Athena | Amazon SageMaker | Apache Airflow | . | Apache Hive | Apache Iceberg | Apache Kafka | Apache Spark | . | AWS CLI | Cloudera | Delta Lake | Git | . | Dremio | DuckDB | Glue / Hive metastore | Kubeflow | . | Presto / Trino | Python | R | Vertex AI | . | HuggingFace Datasets | Databricks | | | . If there is a technology not listed here that you would like to use with lakeFS, please drop by our Slack and we’ll help you get started with it. ", + "content": "You can use lakeFS with a wide range of tools and frameworks. lakeFS provides several clients directly, as well as an S3-compatible gateway. This gateway means that if you want to use something with lakeFS, so long as that technology can interface with S3, it can interface with lakeFS. See below for detailed instructions for using different technologies with lakeFS. | Airbyte | Amazon Athena | Amazon SageMaker | Apache Airflow | . | Apache Hive | Apache Iceberg | Apache Kafka | Apache Spark | . | AWS CLI | Cloudera | Databricks | Delta Lake | . | Dremio | DuckDB | Git | Glue / Hive metastore | . | HuggingFace Datasets | Kubeflow | Presto / Trino | Python | . | R | Red Hat OpenShift AI | Vertex AI | | . If there is a technology not listed here that you would like to use with lakeFS, please drop by our Slack and we’ll help you get started with it. ", "url": "/integrations/#integrations-with-lakefs", "relUrl": "/integrations/#integrations-with-lakefs" @@ -3758,790 +3758,804 @@ "relUrl": "/security/rbac.html#pluggable-authentication-and-authorization" },"537": { + "doc": "Red Hat OpenShift AI", + "title": "Using lakeFS with Red Hat OpenShift AI", + "content": "Red Hat® OpenShift® is an enterprise-ready Kubernetes container platform with full-stack automated operations to manage hybrid cloud, multi-cloud, and edge deployments. OpenShift includes an enterprise-grade Linux operating system, container runtime, networking, monitoring, registry, and authentication and authorization solutions. Red Hat® OpenShift® AI is a flexible, scalable artificial intelligence (AI) and machine learning (ML) platform that enables enterprises to create and deliver AI-enabled applications at scale across hybrid cloud environments. Built using open source technologies, OpenShift AI provides trusted, operationally consistent capabilities for teams to experiment, serve models, and deliver innovative apps. OpenShift AI and lakeFS can be deployed in OpenShift cluster in 3 different architectures: . | OpenShift AI, lakeFS and object storage are delpoyed in OpenShift cluster | OpenShift AI and lakeFS are deployed in OpenShift cluster while object storage is external | OpenShift AI is deployed in OpenShift cluster while lakeFS and object storage are external | . Refer to an example in lakeFS-samples to deploy lakeFS, MinIO and OpenShift AI tutorial (Fraud Detection demo) in OpenShift cluster. Fraud detection demo is a step-by-step guide for using OpenShift AI to train an example model in JupyterLab, deploy the model, and refine the model by using automated pipelines. In this example, OpenShift AI is configured to connect over S3 interace to lakeFS, which will version the data in a backend MinIO instance. This is the architecture to run Fraud Detection demo with or without lakeFS: . lakeFS-samples also includes multiple Helm chart examples to deploy lakeFS and MinIO in different scenarios: . | lakefs-local.yaml: Bring up lakeFS using local object storage. This would be useful for a quick demo where MinIO is not included. | lakefs-minio.yaml: Bring up lakeFS configured to use MinIO as backend object storage. This will be used in the lakeFS demo. | minio-direct.yaml: This file would only be used if lakeFS is not in the picture and OpenShift AI will communicate directly with MinIO. It will bring up MinIO as it is in the default Fraud Detection demo, complete with configuring MinIO storage buckets and the OpenShift AI data connections. It may serve useful in debugging an issue. | minio-via-lakefs.yaml: Bring up MinIO for the modified Fraud Detection demo that includes lakeFS, complete with configuring MinIO storage buckets, but do NOT configure the OpenShift AI data connections. This will be used in the lakeFS demo. | . ", + "url": "/integrations/red_hat_openshift_ai.html#using-lakefs-with-red-hat-openshift-ai", + + "relUrl": "/integrations/red_hat_openshift_ai.html#using-lakefs-with-red-hat-openshift-ai" + },"538": { + "doc": "Red Hat OpenShift AI", + "title": "Red Hat OpenShift AI", + "content": " ", + "url": "/integrations/red_hat_openshift_ai.html", + + "relUrl": "/integrations/red_hat_openshift_ai.html" + },"539": { "doc": "Remote Authenticator", "title": "Remote Authenticator", "content": "Remote Authenticator is a pluggable architecture for lakeFS which allows you to use existing organizational identity policies and infrastructure with the authentication mechanism of lakeFS. The Remote Authenticator’s job is to abstract away the complexities of existing infrastructure and implement a standard interface, which lakeFS can use to resolve user identity and manage access to lakeFS. This loose coupling allows you to implement federated identity without providing lakeFS with direct access to your identity infrastructure. ", "url": "/security/remote-authenticator.html", "relUrl": "/security/remote-authenticator.html" - },"538": { + },"540": { "doc": "Remote Authenticator", "title": "Architecture", "content": "Here’s the authentication flow that lakeFS uses when configured with a remote authenticator: . sequenceDiagram participant A as lakeFS Client participant B as lakeFS Server participant C as Remote Authenticator participant D as IdP A->>B: Submit login form B->>C: POST user credentials C->>D: IdP request D->>C: IdP response C->>B: Auth response B->>A: auth JWT . ", "url": "/security/remote-authenticator.html#architecture", "relUrl": "/security/remote-authenticator.html#architecture" - },"539": { + },"541": { "doc": "Remote Authenticator", "title": "The Interface", "content": "To configure lakeFS to work with a Remote Authenticator add the following YAML to your lakeFS configuration: . auth: remote_authenticator: enabled: true endpoint: <url-to-remote-authenticator-endpoint> default_user_group: \"Developers\" ui_config: logout_url: /logout login_cookie_names: - internal_auth_session . | auth.remote_authenticator.enabled - set lakeFS to use the remote authenticator | auth.remote_authenticator.endpoint - an endpoint where the remote authenticator is able to receive a POST request from lakeFS | auth.remote_authenticator.default_user_group - the group assigned by default to new users | auth.ui_config.logout_url - the URL to redirect the browser when clicking the logout link in the user menu | auth.ui_config.login_cookie_names - the name of the cookie(s) lakeFS will set following a successful authentication. The value is the authenticated user’s JWT | . A Remote Authenticator implementation should expose a single endpoint, which expects the following JSON request: . { \"username\": \"testy.mctestface@example.com\", \"password\": \"Password1\" } . and returns a JSON response like this: . { \"external_user_identifier\": \"TestyMcTestface\" } . Example Request & Responses . Request . POST https://remote-authenticator.example.com/auth Content-Type: application/json { \"username\": \"testy.mctestface@example.com\", \"password\": \"Password1\" } . Successful Response . HTTP/1.1 200 OK Content-Type: application/json { \"external_user_identifier\": \"TestyMcTestface\" } . Unauthorized Response . HTTP/1.1 401 Unauthorized Content-Type: application/json { \"external_user_identifier\": \"\" } . If the Remote Authenticator returns any HTTP status in the 2xx range, lakeFS considers this a successful authentication. Any HTTP status < 200 or > 300 is considered a failed authentication. If the Remote Authenticator returns a non-empty value for the external_user_identifier property along with a success HTTP status, lakeFS will show this identifier instead of an internal lakeFS user identifier in the UI. ", "url": "/security/remote-authenticator.html#the-interface", "relUrl": "/security/remote-authenticator.html#the-interface" - },"540": { + },"542": { "doc": "Remote Authenticator", "title": "Sample Implementation", "content": "Here is a sample Remote Authenticator implemented using node and express and written in TypeScript. This example implementation doesn’t integrate with any real IdP but illustrates the expected request/response patterns that you need to implement. import dotenv from \"dotenv\"; import express, { Express, Request, Response } from \"express\"; import { StatusCodes } from \"http-status-codes\"; type AuthRequestBody = { username: string; password: string; }; type AuthResponseBody = { external_user_identifier: string; }; const DEFAULT_PORT = 80; dotenv.config(); const port = process.env.PORT || DEFAULT_PORT; const app: Express = express(); app.post( \"/auth\", (req: Request<AuthResponseBody, {}, AuthRequestBody>, res: Response) => { const { username, password } = req.body; if (!username?.length || !password?.length) { return res.status(StatusCodes.BAD_REQUEST).json({ external_user_identifier: \"\", }); } // 👇🏻 This is where you would implement your own authentication logic if ( username === \"testy.mctestface@example.com\" && password === \"Password1\" ) { return res.status(StatusCodes.OK).json({ external_user_identifier: \"TestyMcTestface\", }); } else { return res.status(StatusCodes.UNAUTHORIZED).json({ external_user_identifier: \"\", }); } } ); app.listen(port, () => { console.log(`Remote Authenticator listening on port ${port}`); }); . To run this service on the sub-domain idp.example.com, use a lakeFS configuration that looks like this: . auth: remote_authenticator: enabled: true endpoint: https://idp.example.com/auth default_user_group: \"Developers\" ui_config: logout_url: /logout login_cookie_names: - internal_auth_session . ", "url": "/security/remote-authenticator.html#sample-implementation", "relUrl": "/security/remote-authenticator.html#sample-implementation" - },"541": { + },"543": { "doc": "Reproducibility", "title": "The Benefits of Reproducible Data", "content": "Data changes frequently. This makes the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state. This has a negative impact on the work, as it becomes hard to: . | Debug a data issue. | Validate machine learning training accuracy (re-running a model over different data gives different results). | Comply with data audits. | . In comparison, lakeFS exposes a Git-like interface to data that allows keeping track of more than just the current state of data. This makes reproducing its state at any point in time straightforward. ", "url": "/understand/use_cases/reproducibility.html#the-benefits-of-reproducible-data", "relUrl": "/understand/use_cases/reproducibility.html#the-benefits-of-reproducible-data" - },"542": { + },"544": { "doc": "Reproducibility", "title": "Achieving Reproducibility with lakeFS", "content": "To make data reproducible, we recommend taking a new commit of your lakeFS repository every time the data in it changes. As long as there’s a commit taken, the process to reproduce a given state is as simple as reading the data from a path that includes the unique commit_id generated for each commit. To read data at it’s current state, we can use a static path containing the repository and branch names. To give an example, if you have a repository named example with a branch named main, reading the latest state of this data into a Spark Dataframe is always: . df = spark.read.parquet(‘s3://example/main/”) . Note: The code above assumes that all objects in the repository under this path are stored in parquet format. If a different format is used, the applicable Spark read method should be used. In a lakeFS repository, we are capable of taking many commits over the data, making many points in time reproducible. In the repository above, a new commit is taken each time a model training script is run, and the commit message includes the specific run number. If we wanted to re-run the model training script and reproduce the exact same results for a historical run, say run #435, we could copy the commit ID associated with the run and read the data into a dataframe like so: . df = spark.read.parquet(\"s3://example/296e54fbee5e176f3f4f4aeb7e087f9d57515750e8c3d033b8b841778613cb23/training_dataset/”) . The ability to reference a specific commit_id in code simplifies reproducing the specific state a data collection or even multiple collections. This has many applications that are common in data development, such as historical debugging, identifying deltas in a data collection, audit compliance, and more. ", "url": "/understand/use_cases/reproducibility.html#achieving-reproducibility-with-lakefs", "relUrl": "/understand/use_cases/reproducibility.html#achieving-reproducibility-with-lakefs" - },"543": { + },"545": { "doc": "Reproducibility", "title": "Reproducibility", "content": " ", "url": "/understand/use_cases/reproducibility.html", "relUrl": "/understand/use_cases/reproducibility.html" - },"544": { + },"546": { "doc": "Rollback", "title": "Rollbacks", "content": " ", "url": "/understand/use_cases/rollback.html#rollbacks", "relUrl": "/understand/use_cases/rollback.html#rollbacks" - },"545": { + },"547": { "doc": "Rollback", "title": "What Is a Rollback?", "content": "A rollback operation is used to to fix critical data errors immediately. What is a critical data error? Think of a situation where erroneous or misformatted data causes a significant issue with an important service or function. In such situations, the first thing to do is stop the bleeding. Rolling back returns data to a state in the past, before the error was present. You might not be showing all the latest data after a rollback, but at least you aren’t showing incorrect data or raising errors. ", "url": "/understand/use_cases/rollback.html#what-is-a-rollback", "relUrl": "/understand/use_cases/rollback.html#what-is-a-rollback" - },"546": { + },"548": { "doc": "Rollback", "title": "Why Rollbacks Are Useful", "content": "A Rollback is used as a stopgap measure to “put out the fire” as quickly as possible while RCA (root cause analysis) is performed to understand 1) exactly how the error happened, and 2) what can be done to prevent it from happening again. It can be a pressured, stressful situation to deal with a critical data error. Having the ability to employ a rollback relieves some of the pressure and makes it more likely you can figure out what happened without creating additional issues. As a real world example, the 14-day outage some Atlassian users experienced in May 2022 could have been an uninteresting minor incident had rolling back the deleted customer data been an option. ", "url": "/understand/use_cases/rollback.html#why-rollbacks-are-useful", "relUrl": "/understand/use_cases/rollback.html#why-rollbacks-are-useful" - },"547": { + },"549": { "doc": "Rollback", "title": "Performing Rollbacks with lakeFS", "content": "lakeFS lets you develop in your data lake in such a way that rollbacks are simple to perform. This starts by taking a commit over your lakeFS repository whenever a change to its state occurs. Using the lakeFS UI or CLI, you can set the current state, or HEAD, of a branch to any historical commit in seconds, effectively performing a rollback. To demonstrate how this works, let’s take the example of a lakeFS repo with the following commit history: . As can be inferred from the history, this repo is updated every minute with a data sync from some data source. An example data sync is a typical ETL job that replicates data from an internal database or any other data source. After each sync, a commit is taken in lakeFS to save a snapshot of data at that point in time. How to Rollback From a Bad Data Sync? . Say a situation occurs where one of the syncs had bad data and is causing downstream dashboards to fail to load. Since we took a commit of the repo right after the sync ran, we can use a revert operation to undo the changes introduced in that sync. Step 1: Copy the commit_id associated with the commit we want to revert. As the screenshot above shows, you can use the Copy ID to Clipboard button to do this. Step 2: Run the revert command using lakectl, the lakeFS CLI. In this example, the command will be as follows: . lakectl branch revert lakefs://example/main 9666d7c9daf37b3ba6964e733d08596ace2ec2c7bc3a4023ad8e80737a6c3e9d . This will undo the changes introduced by this commit, completing the rollback! . The rollback operation is that simple, even if many changes were introduced in a commit, spanning acrossmultiple data collections. In lakeFS, rolling back data is always a one-liner. ", "url": "/understand/use_cases/rollback.html#performing-rollbacks-with-lakefs", "relUrl": "/understand/use_cases/rollback.html#performing-rollbacks-with-lakefs" - },"548": { + },"550": { "doc": "Rollback", "title": "Rollback", "content": " ", "url": "/understand/use_cases/rollback.html", "relUrl": "/understand/use_cases/rollback.html" - },"549": { + },"551": { "doc": "5️⃣ Roll back Changes", "title": "Rolling back Changes in lakeFS", "content": "Our intrepid user (you) merged a change back into the main branch and realised that they had made a mistake 🤦🏻. The good news for them (you) is that lakeFS can revert changes made, similar to how you would in Git 😅. From your terminal window run lakectl with the revert command: . docker exec -it lakefs lakectl branch revert lakefs://quickstart/main main --parent-number 1 --yes . You should see a confirmation of a successful rollback: . Branch: lakefs://quickstart/main commit main successfully reverted . Back in the object page and the DuckDB query we can see that the original file is now back to how it was: . ", "url": "/quickstart/rollback.html#rolling-back-changes-in-lakefs", "relUrl": "/quickstart/rollback.html#rolling-back-changes-in-lakefs" - },"550": { + },"552": { "doc": "5️⃣ Roll back Changes", "title": "5️⃣ Roll back Changes", "content": " ", "url": "/quickstart/rollback.html", "relUrl": "/quickstart/rollback.html" - },"551": { + },"553": { "doc": "S3 Gateway API", "title": "S3-Supported API", "content": "The S3 Gateway emulates a subset of the API exposed by S3. This subset includes all API endpoints relevant to data systems. For more information, see architecture. lakeFS supports the following API operations: . | Identity and authorization . | SIGv2 | SIGv4 | . | Bucket operations: . | HEAD bucket | . | Object operations: . | DeleteObject | DeleteObjects | GetObject . | Support for caching headers, ETag | Support for range requests | No support for SSE | No support for SelectObject operations | . | HeadObject | PutObject . | Support multi-part uploads | No support for storage classes | No object level tagging | . | CopyObject | . | Object Listing: . | ListObjects | ListObjectsV2 | Delimiter support (for \"/\" only) | . | Multipart Uploads: . | AbortMultipartUpload | CompleteMultipartUpload | CreateMultipartUpload | ListParts Currently supported only on AWS S3. Link to tracked issue | Upload Part | UploadPartCopy | . | . ", "url": "/reference/s3.html#s3-supported-api", "relUrl": "/reference/s3.html#s3-supported-api" - },"552": { + },"554": { "doc": "S3 Gateway API", "title": "S3 Gateway API", "content": " ", "url": "/reference/s3.html", "relUrl": "/reference/s3.html" - },"553": { + },"555": { "doc": "Amazon SageMaker", "title": "Using lakeFS with Amazon SageMaker", "content": "Amazon SageMaker helps to prepare, build, train and deploy ML models quickly by bringing together a broad set of capabilities purpose-built for ML. ", "url": "/integrations/sagemaker.html#using-lakefs-with-amazon-sagemaker", "relUrl": "/integrations/sagemaker.html#using-lakefs-with-amazon-sagemaker" - },"554": { + },"556": { "doc": "Amazon SageMaker", "title": "Table of contents", "content": ". | Initializing session and client | Usage Examples | . ", "url": "/integrations/sagemaker.html#table-of-contents", "relUrl": "/integrations/sagemaker.html#table-of-contents" - },"555": { + },"557": { "doc": "Amazon SageMaker", "title": "Initializing session and client", "content": "Initialize a Sagemaker session and an S3 client with lakeFS as the endpoint: . import sagemaker import boto3 endpoint_url = '<LAKEFS_ENDPOINT>' aws_access_key_id = '<LAKEFS_ACCESS_KEY_ID>' aws_secret_access_key = '<LAKEFS_SECRET_ACCESS_KEY>' repo = 'example-repo' sm = boto3.client('sagemaker', endpoint_url=endpoint_url, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key) s3_resource = boto3.resource('s3', endpoint_url=endpoint_url, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key) session = sagemaker.Session(boto3.Session(), sagemaker_client=sm, default_bucket=repo) session.s3_resource = s3_resource . ", "url": "/integrations/sagemaker.html#initializing-session-and-client", "relUrl": "/integrations/sagemaker.html#initializing-session-and-client" - },"556": { + },"558": { "doc": "Amazon SageMaker", "title": "Usage Examples", "content": "Upload train and test data . Let’s use the created session for uploading data to the ‘main’ branch: . prefix = \"/prefix-within-branch\" branch = 'main' train_file = 'train_data.csv'; train_data.to_csv(train_file, index=False, header=True) train_data_s3_path = session.upload_data(path=train_file, key_prefix=branch + prefix + \"/train\") test_file = 'test_data.csv'; test_data_no_target.to_csv(test_file, index=False, header=False) test_data_s3_path = session.upload_data(path=test_file, key_prefix=branch + prefix + \"/test\") . Download objects . You can use the integration with lakeFS to download a portion of the data you see fit: . repo = 'example-repo' prefix = \"/prefix-to-download\" branch = 'main' localpath = './' + branch session.download_data(path=localpath, bucket=repo, key_prefix = branch + prefix) . Note: Advanced AWS SageMaker features, like Autopilot jobs, are encapsulated and don’t have the option to override the S3 endpoint. However, it is possible to export the required inputs from lakeFS to S3. If you’re using SageMaker features that aren’t supported by lakeFS, we’d love to hear from you. ", "url": "/integrations/sagemaker.html#usage-examples", "relUrl": "/integrations/sagemaker.html#usage-examples" - },"557": { + },"559": { "doc": "Amazon SageMaker", "title": "Amazon SageMaker", "content": " ", "url": "/integrations/sagemaker.html", "relUrl": "/integrations/sagemaker.html" - },"558": { + },"560": { "doc": "System for Cross-domain Identity Management (SCIM)", "title": "System for Cross-domain Identity Management (SCIM)", "content": "lakeFS Cloud . lakeFS Enterprise . lakeFS Cloud includes an SCIM v2.0 compliant server, which can integrate with SCIM clients (IdPs) to automate provisioning/de-provisioning of users and groups. ", "url": "/howto/scim.html", "relUrl": "/howto/scim.html" - },"559": { + },"561": { "doc": "System for Cross-domain Identity Management (SCIM)", "title": "Table of contents", "content": ". | Officially Supported Clients (IdPs), Capabilities, and Limitations | Enabling SCIM in lakeFS Cloud | Setting Up SCIM Provisioning in Entra ID (a.k.a Azure AD) | . ", "url": "/howto/scim.html#table-of-contents", "relUrl": "/howto/scim.html#table-of-contents" - },"560": { + },"562": { "doc": "System for Cross-domain Identity Management (SCIM)", "title": "Officially Supported Clients (IdPs), Capabilities, and Limitations", "content": "Supported Clients (IdPs) . Currently, the lakeFS Cloud SCIM server has been tested and validated with Entra ID (a.k.a Azure AD). However, with SCIM v2.0 being an accepted standard, any SCIM-compliant IdP should be able to integrate with lakeFS Cloud. Capabilities . User Provisioning . | Create users: Users and members of groups assigned to the application will be provisioned in lakeFS Cloud | Update user attributes: Changes to supported user attributes are synced to lakeFS Cloud | Deactivate users: Deactivating a user or removing their assignment to the application will disable them in lakeFS Cloud | User adoption: Users that are already found in lakeFS Cloud will be “adopted” by the IdP and not re-created | . Group Provisioning . | Create groups: Groups assigned to the application are created in lakeFS Cloud and any group user members are created and added to the group in lakeFS Cloud | Update group name: When a synced group is renamed in the IdP, it will be renamed in lakeFS Cloud | Add/remove members: When members are added/removed from an assigned group, they will be added/removed from the group in lakeFS Cloud | Group adoption: Groups that already exist in lakeFS Cloud will be “adopted” by the IdP and not re-created | . User Attributes and Consent . The lakeFS Cloud SCIM server requires the minimum set of user attributes required for provisioning. The required attributes are a sub-set of the basic user profile, which is exchanged during federated authentication/SSO login. User consent is requested by the IdP upon first login to lakeFS Cloud. Known Limitations . | User and group policies can only be managed in lakeFS This means groups and users newly created via SCIM only have basic read permissions. The lakeFS UI or API must be used to attach policies to those users and groups. However, if a user is created and added to an existing group with an attached policy, that user will receive the permissions allowed by the policy attached to the group. | Only direct group memberships are provisioned via SCIM Both Okta and Entra ID only support syncing direct group membership via SCIM. This means that if you assign a group to the application, only its user members will be provisioned via SCIM. SCIM provisioning will not cascade to member groups and their members, and so forth. | . ", "url": "/howto/scim.html#officially-supported-clients-idps-capabilities-and-limitations", "relUrl": "/howto/scim.html#officially-supported-clients-idps-capabilities-and-limitations" - },"561": { + },"563": { "doc": "System for Cross-domain Identity Management (SCIM)", "title": "Enabling SCIM in lakeFS Cloud", "content": "To enable SCIM support in lakeFS Cloud, you need to log into the cloud admin. In the cloud admin, SCIM settings are under Access > Settings. SCIM is not enabled by default, so to enable SCIM for the organization, click the Setup Provisioning Button. Clicking the button will enable SCIM for the organization and provide the details you’ll need to set up your IdP to work with lakeFS Cloud SCIM. To set up your IdP, you’ll need the lakeFS Cloud SCIM provisioning endpoint and you’ll also need to generate an integration token. When creating a new integration token, you can optionally provide a description for future reference. Note: The token value is only presented once, right after creation. Make sure to copy the token, as its value isn’t stored and cannot be retrieved after the initial creation. ", "url": "/howto/scim.html#enabling-scim-in-lakefs-cloud", "relUrl": "/howto/scim.html#enabling-scim-in-lakefs-cloud" - },"562": { + },"564": { "doc": "System for Cross-domain Identity Management (SCIM)", "title": "Setting Up SCIM Provisioning in Entra ID (a.k.a Azure AD)", "content": "Note: This guide assumes you’ve already set up an Entra ID enterprise application for federated authentication to lakeFS Cloud. In the Entra ID admin dashboard, go to Enterprise Applications and choose the lakeFS Cloud enterprise application from the list. Then click Provisioning in the sidebar and then Get Started. | In the provisioning settings set mode to Automatic | In Tenant URL enter the URL from the lakeFS Cloud provisioning settings | In Secret Token paste the token you copied in the previous step. If you haven’t created a token yet, you may do so now | Click Test Connection | If the test fails, please ensure you’ve entered the correct SCIM endpoint URL from lakeFS Cloud and copied the token correctly. Otherwise, click “Save” at the top of the settings panel | . Note: lakeFS Cloud is designed to work with the default attribute mapping for users and groups provided by Entra ID. If your organization has customized the user and/or group entities in Entra ID, you might want to set mappings in accordance with those. You can find details of how this is done in the Entra ID documentation. Incorrectly modifying these mappings can break provisioning functionality, so it’s advised to do so cautiously and only when necessary. ", "url": "/howto/scim.html#setting-up-scim-provisioning-in-entra-id-aka-azure-ad", "relUrl": "/howto/scim.html#setting-up-scim-provisioning-in-entra-id-aka-azure-ad" - },"563": { + },"565": { "doc": "Sizing Guide", "title": "Sizing guide", "content": "Note: For a scalable managed lakeFS service with guaranteed SLAs, try lakeFS Cloud . ", "url": "/howto/sizing-guide.html#sizing-guide", "relUrl": "/howto/sizing-guide.html#sizing-guide" - },"564": { + },"566": { "doc": "Sizing Guide", "title": "Table of contents", "content": ". | System Requirements | Scaling factors | Benchmarks | Important metrics | Reference architectures | . ", "url": "/howto/sizing-guide.html#table-of-contents", "relUrl": "/howto/sizing-guide.html#table-of-contents" - },"565": { + },"567": { "doc": "Sizing Guide", "title": "System Requirements", "content": "Operating Systems and ISA . lakeFS can run on MacOS and Linux. Windows binaries are available but not rigorously tested - we don’t recommend deploying lakeFS to production on Windows. x86_64 and arm64 architectures are supported for both MacOS and Linux. Memory and CPU requirements . lakeFS servers require a minimum of 512mb of RAM and 1 CPU core. For high throughput, additional CPUs help scale requests across different cores. “Expensive” operations such as large diff or commit operations can take advantage of multiple cores. Network . If using the data APIs such as the S3 Gateway, lakeFS will require enough network bandwidth to support the planned concurrent network upload/download operations. For most cloud providers, more powerful machines (i.e., more expensive and usually containing more CPU cores) also provide increased network bandwidth. If using only the metadata APIs (for example, only using the Hadoop/Spark clients), network bandwidth is minimal, at roughly 1Kb per request. Disk . lakeFS greatly benefits from fast local disks. A lakeFS instance doesn’t require any strong durability guarantees from the underlying storage, as the disk is only ever used as a local caching layer for lakeFS metadata and not for long-term storage. lakeFS is designed to work with ephemeral disks - these are usually based on NVMe and are tied to the machine’s lifecycle. Using ephemeral disks lakeFS can provide a very high throughput/cost ratio, probably the best that could be achieved on a public cloud, so we recommend those. A local cache of at least 512 MiB should be provided. For large installations (managing >100 concurrently active branches, with >100M objects per commit), we recommend allocating at least 10 GiB - since it’s a caching layer over a relatively slow storage (the object store), see Important metrics below to understand how to size this: it should be big enough to hold all commit metadata for actively referenced commits. lakeFS KV Store . lakeFS uses a key-value database to manage branch references, authentication and authorization information and to keep track of currently uncommitted data across branches. Please refer to the relevant driver tab for best practices, requirements and benchmarks. Storage . The dataset stored in the metadata store is relatively modest as most metadata is pushed down into the object store. Required storage is mostly a factor of the amount of uncommitted writes across all branches at any given point in time: in the range of 150 MiB per every 100,000 uncommitted writes. We recommend starting at 10 GiB for a production deployment, as it will likely be more than enough. | PostgreSQL | DynamoDB | . RAM Since the data size is small, it’s recommended to provide enough memory to hold the vast majority of that data in RAM. Cloud providers will save you the need to tune this parameter - it will be set to a fixed percentage the chosen instance’s available RAM (25% on AWS RDS, 30% on Google Cloud SQL). It is recommended that you check with your selected cloud provider for configuration and provisioning information for you database. For self-managed database instances follow these best practices . Ideally, configure the shared_buffers of your PostgreSQL instances to be large enough to contain the currently active dataset. Pick a database instance with enough RAM to accommodate this buffer size at roughly x4 the size given for shared_buffers. For example, if an installation has ~500,000 uncommitted writes at any given time, it would require about 750 MiB of shared_buffers that would require about 3 GiB of RAM. CPU PostgreSQL CPU cores help scale concurrent requests. 1 CPU core for every 5,000 requests/second is ideal. lakeFS will create a DynamoDB table for you, defaults to on-demand capacity setting. No need to specify how much read and write throughput you expect your application to perform, as DynamoDB instantly accommodates your workloads as they ramp up or down. You can customize the table settings to provisioned capacity which allows you to manage and optimize your costs by allocating read/write capacity in advance (see Benchmarks) . Notes: . | Using DynamoDB on-demand capacity might generate unwanted costs if the table is abused, if you’d like to cap your costs, make sure to change the table to use provisioned capacity instead. | lakeFS doesn’t manage the DynamoDB’s table lifecycle, we’ve included the table creation in order to help evaluating the system with minimal effort, any change to the table beyond the table creation - will need to be handled manually or by 3rd party tools. | . RAM Managed by AWS. CPU Managed by AWS. ", "url": "/howto/sizing-guide.html#system-requirements", "relUrl": "/howto/sizing-guide.html#system-requirements" - },"566": { + },"568": { "doc": "Sizing Guide", "title": "Scaling factors", "content": "Scaling lakeFS, like most data systems, moves across two axes: throughput of requests (amount per given timeframe) and latency (time to complete a single request). Understanding latency and throughput considerations . Most lakeFS operations are designed to be very low in latency. Assuming a well-tuned local disk cache (see Storage above), most critical path operations (writing objects, requesting objects, deleting objects) are designed to complete in <25ms at p90. Listing objects obviously requires accessing more data, but should always be on-par with what the underlying object store can provide, and in most cases, it’s actually faster. At the worst case, for directory listing with 1,000 common prefixes returned, expect a latency of 75ms at p90. Managing branches (creating them, listing them and deleting them) are all constant-time operations, generally taking <30ms at p90. Committing and merging can take longer, as they are proportional to the amount of changes introduced. This is what makes lakeFS optimal for large Data Lakes - the amount of changes introduced per commit usually stays relatively stable while the entire data set usually grows over time. This means lakeFS will provide predictable performance: committing 100 changes will take roughly the same amount of time whether the resulting commit contains 500 or 500 million objects. See Data Model for more information. Scaling throughput depends very much on the amount of CPU cores available to lakeFS. In many cases, it’s easier to scale lakeFS across a fleet of smaller cloud instances (or containers) than scale up with machines that have many cores. In fact, lakeFS works well in both cases. Most critical path operations scale very well across machines. ", "url": "/howto/sizing-guide.html#scaling-factors", "relUrl": "/howto/sizing-guide.html#scaling-factors" - },"567": { + },"569": { "doc": "Sizing Guide", "title": "Benchmarks", "content": ". | PostgreSQL | DynamoDB | . PostgresSQL . All benchmarks below were measured using 2 x c5ad.4xlarge instances on AWS us-east-1. Similar results can be achieved on Google Cloud using a c2-standard-16 machine type, with an attached local SSD. On Azure, you can use a Standard_F16s_v2 virtual machine. The PostgreSQL instance that was used is a db.m6g.2xlarge (8 vCPUs, 32 GB RAM). Equivalent machines on Google Cloud or Azure should yield similar results. The example repository we tested against contains the metadata of a large lakeFS installation, where each commit contains ~180,000,000 objects (representing ~7.5 Petabytes of data). All tests are reproducible using the lakectl abuse command, so use it to properly size and tune your setup. All tests are accompanied by the relevant lakectl abuse command that generated them. Random reads . This test generates random read requests to lakeFS, in a given commit. Paths are requested randomly from a file containing a set of preconfigured (and existing) paths. command executed: . lakectl abuse random-read \\ --from-file randomly_selected_paths.txt \\ --amount 500000 \\ --parallelism 128 \\ lakefs://example-repo/<commit hash> . Note lakeFS version <= v0.33.1 uses ‘@’ (instead of ‘/’) as separator between repository and commit hash. Result Histogram (raw): . Histogram (ms): 1 0 2 0 5 37945 7 179727 10 296964 15 399682 25 477502 50 499625 75 499998 100 499998 250 500000 350 500000 500 500000 750 500000 1000 500000 5000 500000 min 3 max 222 total 500000 . So 50% of all requests took <10ms, while 99.9% of them took <50ms . throughput: . Average throughput during the experiment was 10851.69 requests/second . Random Writes . This test generates random write requests to a given lakeFS branch. All the paths are pre-generated and don’t overwrite each other (as overwrites are relatively rare in a Data Lake setup). command executed: . lakectl abuse random-write \\ --amount 500000 \\ --parallelism 64 \\ lakefs://example-repo/main . Note lakeFS version <= v0.33.1 uses ‘@’ (instead of ‘/’) as separator between repository and branch. Result Histogram (raw): . Histogram (ms): 1 0 2 0 5 30715 7 219647 10 455807 15 498144 25 499535 50 499742 75 499784 100 499802 250 500000 350 500000 500 500000 750 500000 1000 500000 5000 500000 min 3 max 233 total 500000 . So, 50% of all requests took <10ms, while 99.9% of them took <25ms. throughput: . The average throughput during the experiment was 7595.46 requests/second. Branch creation . This test creates branches from a given reference. command executed: . lakectl abuse create-branches \\ --amount 500000 \\ --branch-prefix \"benchmark-\" \\ --parallelism 256 \\ lakefs://example-repo/<commit hash> . Note lakeFS version <= v0.33.1 uses ‘@’ (instead of ‘/’) as separator between repository and commit hash. Result Histogram (raw): . Histogram (ms): 1 0 2 1 5 5901 7 39835 10 135863 15 270201 25 399895 50 484932 75 497180 100 499303 250 499996 350 500000 500 500000 750 500000 1000 500000 5000 500000 min 2 max 304 total 500000 . So, 50% of all requests took <15ms, while 99.9% of them took <100ms. throughput: . The average throughput during the experiment was 7069.03 requests/second. DynamoDB . All benchmarks below were measured using m5.xlarge instance on AWS us-east-1. The DynamoDB table that was used was provisioned with 500/1000 read/write capacity. The example repository we tested against contains the metadata of a large lakeFS installation, where each commit contains ~100,000,000 objects (representing ~3.5 Petabytes of data). All tests are reproducible using the lakectl abuse command, so use it to properly size and tune your setup. All tests are accompanied by the relevant lakectl abuse command that generated them. Random reads . This test generates random read requests to lakeFS, in a given commit. Paths are requested randomly from a file containing a set of preconfigured (and existing) paths. command executed: . lakectl abuse random-read \\ --from-file randomly_selected_paths.txt \\ --amount 500000 \\ --parallelism 128 \\ lakefs://example-repo/<commit hash> . Result Histogram (raw): Provisioned read capacity units = 1000 Provisioned write capacity units = 1000 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 0 25 122 50 47364 75 344489 100 460404 250 497912 350 498016 500 498045 750 498111 1000 498176 5000 499478 min 18 max 52272 total 500000 . Result Histogram (raw): Provisioned read capacity units = 500 Provisioned write capacity units = 500 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 1 25 2672 50 239661 75 420171 100 470146 250 486603 350 486715 500 486789 750 487443 1000 488113 5000 493201 min 14 max 648085 total 499998 . Random Writes . This test generates random write requests to a given lakeFS branch. All the paths are pre-generated and don’t overwrite each other (as overwrites are relatively rare in a Data Lake setup). command executed: . lakectl abuse random-write \\ --amount 500000 \\ --parallelism 64 \\ lakefs://example-repo/main . Result Histogram (raw): Provisioned read capacity units = 1000 Provisioned write capacity units = 1000 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 0 25 24 50 239852 75 458504 100 485225 250 493687 350 493872 500 493960 750 496239 1000 499194 5000 500000 min 23 max 4437 total 500000 . Result Histogram (raw): Provisioned read capacity units = 500 Provisioned write capacity units = 500 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 0 25 174 50 266460 75 462641 100 484486 250 490633 350 490856 500 490984 750 492973 1000 495605 5000 498920 min 21 max 50157 total 500000 . Branch creation . This test creates branches from a given reference. command executed: . lakectl abuse create-branches \\ --amount 500000 \\ --branch-prefix \"benchmark-\" \\ --parallelism 256 \\ lakefs://example-repo/<commit hash> . Result Histogram (raw): Provisioned read capacity units = 1000 Provisioned write capacity units = 1000 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 0 25 0 50 628 75 26153 100 58099 250 216160 350 307078 500 406165 750 422898 1000 431332 5000 475848 min 41 max 430725 total 490054 . Result Histogram (raw): Provisioned read capacity units = 500 Provisioned write capacity units = 500 . Histogram (ms): 1 0 2 0 5 0 7 0 10 0 15 0 25 0 50 3132 75 155570 100 292745 250 384224 350 397258 500 431141 750 441360 1000 445597 5000 469538 min 39 max 760626 total 497520 . ", "url": "/howto/sizing-guide.html#benchmarks", "relUrl": "/howto/sizing-guide.html#benchmarks" - },"568": { + },"570": { "doc": "Sizing Guide", "title": "Important metrics", "content": "lakeFS exposes metrics using the Prometheus protocol. Every lakeFS instance exposes a /metrics endpoint that could be used to extract them. Here are a few notable metrics to keep track of when sizing lakeFS: . api_requests_total - Tracks throughput of API requests over time. api_request_duration_seconds - Histogram of latency per operation type. gateway_request_duration_seconds - Histogram of latency per S3 Gateway operation. | PostgreSQL | DynamoDB | . dynamo_request_duration_seconds - Time spent doing DynamoDB requests. dynamo_consumed_capacity_total - The capacity units consumed by operation. dynamo_failures_total - The total number of errors while working for kv store. ", "url": "/howto/sizing-guide.html#important-metrics", "relUrl": "/howto/sizing-guide.html#important-metrics" - },"569": { + },"571": { "doc": "Sizing Guide", "title": "Reference architectures", "content": "Below are a few example architectures for lakeFS deployment. Reference Architecture: Data Science/Research environment . Use case: Manage Machine learning or algorithms development. Use lakeFS branches to achieve both isolation and reproducibility of experiments. Data being managed by lakeFS is both structured tabular data, as well as unstructured sensor and image data used for training. Assuming a team of 20-50 researchers, with a dataset size of 500 TiB across 20M objects. Environment: lakeFS will be deployed on Kubernetes. managed by AWS EKS with PostgreSQL on AWS RDS Aurora . Sizing: Since most of the work is done by humans (vs. automated pipelines), most experiments tend to be small in scale, reading and writing 10s to 1000s of objects. The expected amount of branches active in parallel is relatively low, around 1-2 per user, each representing a small amount of uncommitted changes at any given point in time. Let’s assume 5,000 uncommitted writes per branch = ~500k. To support the expected throughput, a single moderate lakeFS instance should be more than enough, since requests per second would be on the order of 10s to 100s. For high availability, we’ll deploy 2 pods with 1 CPU core and 1 GiB of RAM each. Since the PostgreSQL instance is expected to hold a very small dataset (at 500k, expected dataset size is 150MiB (for 100k records) * 5 = 750MiB). To ensure we have enough RAM to hold this, we’ll need 3 GiB of RAM, so, a very moderate Aurora instance db.t3.large (2 vCPUs, 8 GB RAM) will be more than enough. An equivalent database instance on GCP or Azure should give similar results. Reference Architecture: Automated Production Pipelines . Use case: Manage multiple concurrent data pipelines using Apache Spark and Airflow. Airflow DAGs start by creating a branch for isolation and for CI/CD. Data being managed by lakeFS is structured, tabular data. The total dataset size is 10 PiB, spanning across 500M objects. The expected throughput is 10k reads/second + 2k writes per second across 100 concurrent branches. Environment: lakeFS will be deployed on Kubernetes. managed by AWS EKS with PostgreSQL on AWS RDS . Sizing: Data pipelines tend to be bursty in nature: reading in a lot of objects concurrently, doing some calculation or aggregation, and then writing many objects concurrently. The expected amount of branches active in parallel is high, with many Airflow DAGs running per day, each representing a moderate amount of uncommitted changes at any given point in time. Let’s assume 1,000 uncommitted writes/branch * 2500 branches = ~2.5M records. To support the expected throughput, looking the benchmarking numbers above, we’re doing roughly 625 requests/core, so 24 cores should cover our peak traffic. We can deploy 6 * 4 CPU core pods. On to the PostgreSQL instance - at 500k, the expected dataset size is 150MiB (for 100k records) * 25 = 3750 MiB. To ensure we have enough RAM to hold this, we’ll need at least 15 GiB of RAM, so we’ll go with a db.r5.xlarge (4 vCPUs, 32GB RAM) Aurora instance. An equivalent database instance on GCP or Azure should give similar results. ", "url": "/howto/sizing-guide.html#reference-architectures", "relUrl": "/howto/sizing-guide.html#reference-architectures" - },"570": { + },"572": { "doc": "Sizing Guide", "title": "Sizing Guide", "content": " ", "url": "/howto/sizing-guide.html", "relUrl": "/howto/sizing-guide.html" - },"571": { + },"573": { "doc": "Spark Client", "title": "lakeFS Spark Metadata Client", "content": "Utilize the power of Spark to interact with the metadata on lakeFS. Possible use cases include: . | Creating a DataFrame for listing the objects in a specific commit or branch. | Computing changes between two commits. | Exporting your data for consumption outside lakeFS. | Bulk operations on the underlying storage. | . ", "url": "/reference/spark-client.html#lakefs-spark-metadata-client", "relUrl": "/reference/spark-client.html#lakefs-spark-metadata-client" - },"572": { + },"574": { "doc": "Spark Client", "title": "Getting Started", "content": "Please note that Spark 2 is no longer supported with the lakeFS metadata client. The Spark metadata client is compiled for Spark 3.1.2 with Hadoop 3.2.1, but can work for other Spark versions and higher Hadoop versions. | PySpark, spark-shell, spark-submit, spark-sql | DataBricks | . Start Spark Shell / PySpark with the --packages flag, for instance: . spark-shell --packages io.lakefs:lakefs-spark-client_2.12:0.14.1 . Alternatively use the assembled jar (an “Überjar”) on S3, from s3://treeverse-clients-us-east/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar by passing its path to --jars. The assembled jar is larger but shades several common libraries. Use it if Spark complains about bad classes or missing methods. Include this assembled jar (an “Überjar”) from S3, from s3://treeverse-clients-us-east/lakefs-spark-client/0.14.1/lakefs-spark-client-assembly-0.14.1.jar. ", "url": "/reference/spark-client.html#getting-started", "relUrl": "/reference/spark-client.html#getting-started" - },"573": { + },"575": { "doc": "Spark Client", "title": "Configuration", "content": ". | To read metadata from lakeFS, the client should be configured with your lakeFS endpoint and credentials, using the following Hadoop configurations: . | Configuration | Description | . | spark.hadoop.lakefs.api.url | lakeFS API endpoint, e.g: http://lakefs.example.com/api/v1 | . | spark.hadoop.lakefs.api.access_key | The access key to use for fetching metadata from lakeFS | . | spark.hadoop.lakefs.api.secret_key | Corresponding lakeFS secret key | . | The client will also directly interact with your storage using Hadoop FileSystem. Therefore, your Spark session must be able to access the underlying storage of your lakeFS repository. There are various ways to do this, but for a non-production environment you can use the following Hadoop configurations: . | Configuration | Description | . | spark.hadoop.fs.s3a.access.key | Access key to use for accessing underlying storage on S3 | . | spark.hadoop.fs.s3a.secret.key | Corresponding secret key to use with S3 access key | . Assuming role on S3 (Hadoop 3 only) . The client includes support for assuming a separate role on S3 when running on Hadoop 3. It uses the same configuration used by S3AFileSystem to assume the role on S3A. Apache Hadoop AWS documentation has details under “Working with IAM Assumed Roles”. You will need to use the following Hadoop configurations: . | Configuration | Description | . | fs.s3a.aws.credentials.provider | Set to org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider | . | fs.s3a.assumed.role.arn | Set to the ARN of the role to assume | . | . ", "url": "/reference/spark-client.html#configuration", "relUrl": "/reference/spark-client.html#configuration" - },"574": { + },"576": { "doc": "Spark Client", "title": "Examples", "content": ". | Get a DataFrame for listing all objects in a commit: . import io.treeverse.clients.LakeFSContext val commitID = \"a1b2c3d4\" val df = LakeFSContext.newDF(spark, \"example-repo\", commitID) df.show /* output example: +------------+--------------------+--------------------+-------------------+----+ | key | address| etag| last_modified|size| +------------+--------------------+--------------------+-------------------+----+ | file_1 |791457df80a0465a8...|7b90878a7c9be5a27...|2021-03-05 11:23:30| 36| file_2 |e15be8f6e2a74c329...|95bee987e9504e2c3...|2021-03-05 11:45:25| 36| file_3 |f6089c25029240578...|32e2f296cb3867d57...|2021-03-07 13:43:19| 36| file_4 |bef38ef97883445c8...|e920efe2bc220ffbb...|2021-03-07 13:43:11| 13| +------------+--------------------+--------------------+-------------------+----+ */ . | Run SQL queries on your metadata: . df.createOrReplaceTempView(\"files\") spark.sql(\"SELECT DATE(last_modified), COUNT(*) FROM files GROUP BY 1 ORDER BY 1\") /* output example: +----------+--------+ | dt|count(1)| +----------+--------+ |2021-03-05| 2|2021-03-07| 2| +----------+--------+ */ . | Search by user metadata: . import io.treeverse.clients.LakeFSContext val namespace = \"s3://bucket/repo/path/\" val df = LakeFSContext.newDF(spark, namespace) val key = \"SomeKey\" val searchedValue = \"val3\" df.select(\"key\", \"user_metadata\") .filter(_.getMap[String, String](1).toMap.get(s\"X-Amz-Meta-${key}\").getOrElse(\"\") == searchedValue) .show() /* output example: +---------+-----------------------------------------------------+ |key |user_metadata | +---------+-----------------------------------------------------+ |file1.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> blue} |file8.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> green}| +---------+-----------------------------------------------------+ */ . | . ", "url": "/reference/spark-client.html#examples", "relUrl": "/reference/spark-client.html#examples" - },"575": { + },"577": { "doc": "Spark Client", "title": "Spark Client", "content": " ", "url": "/reference/spark-client.html", "relUrl": "/reference/spark-client.html" - },"576": { + },"578": { "doc": "Apache Spark", "title": "Using lakeFS with Apache Spark", "content": "There are several ways to use lakeFS with Spark: . | The S3-compatible API: Scalable and best to get started. All Storage Vendors | The lakeFS FileSystem: Direct data flow from client to storage, highly scalable. AWS S3 . | lakeFS FileSystem in Presigned mode: Best of both worlds. AWS S3Azure Blob | . | . See how SimilarWeb is using lakeFS with Spark to manage algorithm changes in data pipelines. ", "url": "/integrations/spark.html#using-lakefs-with-apache-spark", "relUrl": "/integrations/spark.html#using-lakefs-with-apache-spark" - },"577": { + },"579": { "doc": "Apache Spark", "title": "Table of contents", "content": ". | S3-compatible API | lakeFS Hadoop FileSystem | Hadoop FileSystem in Presigned mode | . ", "url": "/integrations/spark.html#table-of-contents", "relUrl": "/integrations/spark.html#table-of-contents" - },"578": { + },"580": { "doc": "Apache Spark", "title": "S3-compatible API", "content": "lakeFS has an S3-compatible endpoint which you can point Spark at to get started quickly. You will access your data using S3-style URIs, e.g. s3a://example-repo/example-branch/example-table. You can use the S3-compatible API regardless of where your data is hosted. Configuration . To configure Spark to work with lakeFS, we set S3A Hadoop configuration to the lakeFS endpoint and credentials: . | fs.s3a.access.key: lakeFS access key | fs.s3a.secret.key: lakeFS secret key | fs.s3a.endpoint: lakeFS S3-compatible API endpoint (e.g. https://example-org.us-east-1.lakefscloud.io) | fs.s3a.path.style.access: true | . Here is how to do it: . | CLI | Scala | XML Configuration | EMR | . spark-shell --conf spark.hadoop.fs.s3a.access.key='AKIAlakefs12345EXAMPLE' \\ --conf spark.hadoop.fs.s3a.secret.key='abc/lakefs/1234567bPxRfiCYEXAMPLEKEY' \\ --conf spark.hadoop.fs.s3a.path.style.access=true \\ --conf spark.hadoop.fs.s3a.endpoint='https://example-org.us-east-1.lakefscloud.io' ... spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.access.key\", \"AKIAlakefs12345EXAMPLE\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.endpoint\", \"https://example-org.us-east-1.lakefscloud.io\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.path.style.access\", \"true\") . Add these into a configuration file, e.g. $SPARK_HOME/conf/hdfs-site.xml: . <?xml version=\"1.0\"?> <configuration> <property> <name>fs.s3a.access.key</name> <value>AKIAlakefs12345EXAMPLE</value> </property> <property> <name>fs.s3a.secret.key</name> <value>abc/lakefs/1234567bPxRfiCYEXAMPLEKEY</value> </property> <property> <name>fs.s3a.endpoint</name> <value>https://example-org.us-east-1.lakefscloud.io</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property> </configuration> . Use the below configuration when creating the cluster. You may delete any app configuration that is not suitable for your use case: . [ { \"Classification\": \"spark-defaults\", \"Properties\": { \"spark.sql.catalogImplementation\": \"hive\" } }, { \"Classification\": \"core-site\", \"Properties\": { \"fs.s3.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.path.style.access\": \"true\", \"fs.s3a.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.path.style.access\": \"true\" } }, { \"Classification\": \"emrfs-site\", \"Properties\": { \"fs.s3.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.path.style.access\": \"true\", \"fs.s3a.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.path.style.access\": \"true\" } }, { \"Classification\": \"presto-connector-hive\", \"Properties\": { \"hive.s3.aws-access-key\": \"AKIAIOSFODNN7EXAMPLE\", \"hive.s3.aws-secret-key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"hive.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"hive.s3.path-style-access\": \"true\", \"hive.s3-file-system-type\": \"PRESTO\" } }, { \"Classification\": \"hive-site\", \"Properties\": { \"fs.s3.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.path.style.access\": \"true\", \"fs.s3a.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.path.style.access\": \"true\" } }, { \"Classification\": \"hdfs-site\", \"Properties\": { \"fs.s3.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.path.style.access\": \"true\", \"fs.s3a.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.path.style.access\": \"true\" } }, { \"Classification\": \"mapred-site\", \"Properties\": { \"fs.s3.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.path.style.access\": \"true\", \"fs.s3a.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.path.style.access\": \"true\" } } ] . Alternatively, you can pass these configuration values when adding a step. For example: . aws emr add-steps --cluster-id j-197B3AEGQ9XE4 \\ --steps=\"Type=Spark,Name=SparkApplication,ActionOnFailure=CONTINUE, \\ Args=[--conf,spark.hadoop.fs.s3a.access.key=AKIAIOSFODNN7EXAMPLE, \\ --conf,spark.hadoop.fs.s3a.secret.key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY, \\ --conf,spark.hadoop.fs.s3a.endpoint=https://example-org.us-east-1.lakefscloud.io, \\ --conf,spark.hadoop.fs.s3a.path.style.access=true, \\ s3a://<lakefs-repo>/<lakefs-branch>/path/to/jar]\" . Per-bucket configuration . The above configuration will use lakeFS as the sole S3 endpoint. To use lakeFS in parallel with S3, you can configure Spark to use lakeFS only for specific bucket names. For example, to configure only example-repo to use lakeFS, set the following configurations: . | CLI | Scala | XML Configuration | EMR | . spark-shell --conf spark.hadoop.fs.s3a.bucket.example-repo.access.key='AKIAlakefs12345EXAMPLE' \\ --conf spark.hadoop.fs.s3a.bucket.example-repo.secret.key='abc/lakefs/1234567bPxRfiCYEXAMPLEKEY' \\ --conf spark.hadoop.fs.s3a.bucket.example-repo.endpoint='https://example-org.us-east-1.lakefscloud.io' \\ --conf spark.hadoop.fs.s3a.path.style.access=true . spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.bucket.example-repo.access.key\", \"AKIAlakefs12345EXAMPLE\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.bucket.example-repo.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.bucket.example-repo.endpoint\", \"https://example-org.us-east-1.lakefscloud.io\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.path.style.access\", \"true\") . Add these into a configuration file, e.g. $SPARK_HOME/conf/hdfs-site.xml: . <?xml version=\"1.0\"?> <configuration> <property> <name>fs.s3a.bucket.example-repo.access.key</name> <value>AKIAlakefs12345EXAMPLE</value> </property> <property> <name>fs.s3a.bucket.example-repo.secret.key</name> <value>abc/lakefs/1234567bPxRfiCYEXAMPLEKEY</value> </property> <property> <name>fs.s3a.bucket.example-repo.endpoint</name> <value>https://example-org.us-east-1.lakefscloud.io</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property> </configuration> . Use the below configuration when creating the cluster. You may delete any app configuration that is not suitable for your use case: . [ { \"Classification\": \"spark-defaults\", \"Properties\": { \"spark.sql.catalogImplementation\": \"hive\" } }, { \"Classification\": \"core-site\", \"Properties\": { \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.bucket.example-repo.path.style.access\": \"true\", \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.bucket.example-repo.path.style.access\": \"true\" } }, { \"Classification\": \"emrfs-site\", \"Properties\": { \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.bucket.example-repo.path.style.access\": \"true\", \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.bucket.example-repo.path.style.access\": \"true\" } }, { \"Classification\": \"presto-connector-hive\", \"Properties\": { \"hive.s3.aws-access-key\": \"AKIAIOSFODNN7EXAMPLE\", \"hive.s3.aws-secret-key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"hive.s3.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"hive.s3.path-style-access\": \"true\", \"hive.s3-file-system-type\": \"PRESTO\" } }, { \"Classification\": \"hive-site\", \"Properties\": { \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.bucket.example-repo.path.style.access\": \"true\", \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.bucket.example-repo.path.style.access\": \"true\" } }, { \"Classification\": \"hdfs-site\", \"Properties\": { \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.bucket.example-repo.path.style.access\": \"true\", \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.bucket.example-repo.path.style.access\": \"true\" } }, { \"Classification\": \"mapred-site\", \"Properties\": { \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3.bucket.example-repo.path.style.access\": \"true\", \"fs.s3a.bucket.example-repo.access.key\": \"AKIAIOSFODNN7EXAMPLE\", \"fs.s3a.bucket.example-repo.secret.key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\", \"fs.s3a.bucket.example-repo.endpoint\": \"https://example-org.us-east-1.lakefscloud.io\", \"fs.s3a.bucket.example-repo.path.style.access\": \"true\" } } ] . Alternatively, you can pass these configuration values when adding a step. For example: . aws emr add-steps --cluster-id j-197B3AEGQ9XE4 \\ --steps=\"Type=Spark,Name=SparkApplication,ActionOnFailure=CONTINUE, \\ Args=[--conf,spark.hadoop.fs.s3a.bucket.example-repo.access.key=AKIAIOSFODNN7EXAMPLE, \\ --conf,spark.hadoop.fs.s3a.bucket.example-repo.secret.key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY, \\ --conf,spark.hadoop.fs.s3a.bucket.example-repo.endpoint=https://example-org.us-east-1.lakefscloud.io, \\ --conf,spark.hadoop.fs.s3a.path.style.access=true, \\ s3a://<lakefs-repo>/<lakefs-branch>/path/to/jar]\" . With this configuration set, you read S3A paths with example-repo as the bucket will use lakeFS, while all other buckets will use AWS S3. Usage . Here’s an example for reading a Parquet file from lakeFS to a Spark DataFrame: . val repo = \"example-repo\" val branch = \"main\" val df = spark.read.parquet(s\"s3a://${repo}/${branch}/example-path/example-file.parquet\") . Here’s how to write some results back to a lakeFS path: . df.write.partitionBy(\"example-column\").parquet(s\"s3a://${repo}/${branch}/output-path/\") . The data is now created in lakeFS as new changes in your branch. You can now commit these changes or revert them. Configuring Azure Databricks with the S3-compatible API . If you use Azure Databricks, you can take advantage of the lakeFS S3-compatible API with your Azure account and the S3A FileSystem. This will require installing the hadoop-aws package (with the same version as your hadoop-azure package) to your Databricks cluster. Define your FileSystem configurations in the following way: . spark.hadoop.fs.lakefs.impl=org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.lakefs.access.key=‘AKIAlakefs12345EXAMPLE’ // The access key to your lakeFS server spark.hadoop.fs.lakefs.secret.key=‘abc/lakefs/1234567bPxRfiCYEXAMPLEKEY’ // The secret key to your lakeFS server spark.hadoop.fs.lakefs.path.style.access=true spark.hadoop.fs.lakefs.endpoint=‘https://example-org.us-east-1.lakefscloud.io’ // The endpoint of your lakeFS server . For more details about Mounting cloud object storage on Databricks. Configuring Databricks SQL Warehouse with the S3-compatible API . A SQL warehouse is a compute resource that lets you run SQL commands on data objects within Databricks SQL. If you use Databricks SQL warehouse, you can take advantage of the lakeFS S3-compatible API with the S3A FileSystem. Define your SQL Warehouse configurations in the following way: . | In the top right, select Admin Settings and then Compute, and SQL warehouses. | Under Data Access Properties add the following key-value pairs for each lakeFS repository you want to access: . | . spark.hadoop.fs.s3a.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.bucket.example-repo.access.key AKIAIOSFODNN7EXAMPLE // The access key to your lakeFS server spark.hadoop.fs.s3a.bucket.example-repo.secret.key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY // The secret key to your lakeFS server spark.hadoop.fs.s3a.bucket.example-repo.endpoint https://example-org.us-east-1.lakefscloud.io // The endpoint of your lakeFS server spark.hadoop.fs.s3a.bucket.example-repo.path.style.access true . | Changes are applied automatically after the SQL Warehouse restarts. | You can now use the lakeFS S3-compatible API with your SQL Warehouse, e.g.: | . SELECT * FROM delta.`s3a://example-repo/main/datasets/delta-table/` LIMIT 100 . ⚠️ Experimental: Pre-signed mode for S3A . In Hadoop 3.1.4 version and above (as tested using our lakeFS Hadoop FS), it is possible to use pre-signed URLs as return values from the lakeFS S3 Gateway. This has the immediate benefit of reducing the amount of traffic that has to go through the lakeFS server thus improving IO performance. To read more about pre-signed URLs, see this guide. Here’s an example Spark configuration to enable this support: . spark.hadoop.fs.s3a.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3a.bucket.example-repo.access.key AKIAIOSFODNN7EXAMPLE // The access key to your lakeFS server spark.hadoop.fs.s3a.bucket.example-repo.secret.key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY // The secret key to your lakeFS server spark.hadoop.fs.s3a.path.style.access true spark.hadoop.fs.s3a.bucket.example-repo.signing-algorithm QueryStringSignerType spark.hadoop.fs.s3a.bucket.example-repo.user.agent.prefix s3RedirectionSupport . user.agent.prefix should contain the string s3RedirectionSupport but does not have to match the string exactly. Once configured, requests will include the string s3RedirectionSupport in the User-Agent HTTP header sent with GetObject requests, resulting in lakeFS responding with a pre-signed URL. Setting the signing-algorithm to QueryStringSignerType is required to stop S3A from signing a pre-signed URL, since the existence of more than one signature method will return an error from S3. ℹ This feature requires a lakeFS server of version >1.18.0 . ", "url": "/integrations/spark.html#s3-compatible-api", "relUrl": "/integrations/spark.html#s3-compatible-api" - },"579": { + },"581": { "doc": "Apache Spark", "title": "lakeFS Hadoop FileSystem", "content": "If you’re using lakeFS on top of S3, this mode will enhance your application’s performance. In this mode, Spark will read and write objects directly from S3, reducing the load on the lakeFS server. It will still access the lakeFS server for metadata operations. After configuring the lakeFS Hadoop FileSystem below, use URIs of the form lakefs://example-repo/ref/path/to/data to interact with your data on lakeFS. Installation . | Spark Standalone | Databricks | Cloudera Spark | . Add the package to your spark-submit command: . --packages io.lakefs:hadoop-lakefs-assembly:0.2.4 . In your cluster settings, under the Libraries tab, add the following Maven package: . io.lakefs:hadoop-lakefs-assembly:0.2.4 . Once installed, it should look something like this: . Add the package to your pyspark or spark-submit command: . --packages io.lakefs:hadoop-lakefs-assembly:0.2.4 . Add the configuration to access the S3 bucket used by lakeFS to your pyspark or spark-submit command or add this configuration at the Cloudera cluster level (see below): . --conf spark.yarn.access.hadoopFileSystems=s3a://bucket-name . Add the configuration to access the S3 bucket used by lakeFS at the Cloudera cluster level: . | Log in to the CDP (Cloudera Data Platform) web interface. | From the CDP home screen, click the Management Console icon. | In the Management Console, select Data Hub Clusters from the navigation pane. | Select the cluster you want to configure. Click on CM-UI link under Services: | In Cloudera Manager web interface, click on Clusters from the navigation pane and click on spark_on_yarn option: | Click on Configuration tab and search for spark.yarn.access.hadoopFileSystems in the search box: | Add S3 bucket used by lakeFS s3a://bucket-name in the spark.yarn.access.hadoopFileSystems list: | . Configuration . Set the fs.lakefs.* Hadoop configurations to point to your lakeFS installation: . | fs.lakefs.impl: io.lakefs.LakeFSFileSystem | fs.lakefs.access.key: lakeFS access key | fs.lakefs.secret.key: lakeFS secret key | fs.lakefs.endpoint: lakeFS API URL (e.g. https://example-org.us-east-1.lakefscloud.io/api/v1) | . Configure the lakeFS client to use a temporary token instead of static credentials: . | fs.lakefs.auth.provider: The default is basic_auth with fs.lakefs.access.key and fs.lakefs.secret.key for basic authentication. Can be set to io.lakefs.auth.TemporaryAWSCredentialsLakeFSTokenProvider for using temporary AWS credentials, you can read more about it here. | . When using io.lakefs.auth.TemporaryAWSCredentialsLakeFSTokenProvider as the auth provider the following configuration are relevant: . | fs.lakefs.token.aws.access.key: AWS assumed role access key | fs.lakefs.token.aws.secret.key: AWS assumed role secret key | fs.lakefs.token.aws.session.token: AWS assumed role temporary session token | fs.lakefs.token.aws.sts.endpoint: AWS STS regional endpoint for generated the presigned-url (i.e https://sts.us-west-2.amazonaws.com) | fs.lakefs.token.aws.sts.duration_seconds: Optional, the duration in seconds for the initial identity token (default is 60) | fs.lakefs.token.duration_seconds: Optional, the duration in seconds for the lakeFS token (default is set in the lakeFS configuration auth.login_duration) | fs.lakefs.token.sts.additional_headers: Optional, comma separated list of header:value to attach when generating presigned sts request. Default is X-Lakefs-Server-ID:fs.lakefs.endpoint. | . Configure the S3A FileSystem to access your S3 storage, for example using the fs.s3a.* configurations (these are not your lakeFS credentials): . | fs.s3a.access.key: AWS S3 access key | fs.s3a.secret.key: AWS S3 secret key | . Here are some configuration examples: . | CLI | Scala | PySpark | XML Configuration | Databricks | . spark-shell --conf spark.hadoop.fs.s3a.access.key='AKIAIOSFODNN7EXAMPLE' \\ --conf spark.hadoop.fs.s3a.secret.key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' \\ --conf spark.hadoop.fs.s3a.endpoint='https://s3.eu-central-1.amazonaws.com' \\ --conf spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem \\ --conf spark.hadoop.fs.lakefs.access.key=AKIAlakefs12345EXAMPLE \\ --conf spark.hadoop.fs.lakefs.secret.key=abc/lakefs/1234567bPxRfiCYEXAMPLEKEY \\ --conf spark.hadoop.fs.lakefs.endpoint=https://example-org.us-east-1.lakefscloud.io/api/v1 \\ --packages io.lakefs:hadoop-lakefs-assembly:0.2.4 \\ io.example.ExampleClass . spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.access.key\", \"AKIAIOSFODNN7EXAMPLE\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.secret.key\", \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\") spark.sparkContext.hadoopConfiguration.set(\"fs.s3a.endpoint\", \"https://s3.eu-central-1.amazonaws.com\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.impl\", \"io.lakefs.LakeFSFileSystem\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.access.key\", \"AKIAlakefs12345EXAMPLE\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.endpoint\", \"https://example-org.us-east-1.lakefscloud.io/api/v1\") . sc._jsc.hadoopConfiguration().set(\"fs.s3a.access.key\", \"AKIAIOSFODNN7EXAMPLE\") sc._jsc.hadoopConfiguration().set(\"fs.s3a.secret.key\", \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\") sc._jsc.hadoopConfiguration().set(\"fs.s3a.endpoint\", \"https://s3.eu-central-1.amazonaws.com\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.impl\", \"io.lakefs.LakeFSFileSystem\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.access.key\", \"AKIAlakefs12345EXAMPLE\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.endpoint\", \"https://example-org.us-east-1.lakefscloud.io/api/v1\") . Make sure that you load the lakeFS FileSystem into Spark by running it with --packages or --jars, and then add these into a configuration file, e.g., $SPARK_HOME/conf/hdfs-site.xml: . <?xml version=\"1.0\"?> <configuration> <property> <name>fs.s3a.access.key</name> <value>AKIAIOSFODNN7EXAMPLE</value> </property> <property> <name>fs.s3a.secret.key</name> <value>wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY</value> </property> <property> <name>fs.s3a.endpoint</name> <value>https://s3.eu-central-1.amazonaws.com</value> </property> <property> <name>fs.lakefs.impl</name> <value>io.lakefs.LakeFSFileSystem</value> </property> <property> <name>fs.lakefs.access.key</name> <value>AKIAlakefs12345EXAMPLE</value> </property> <property> <name>fs.lakefs.secret.key</name> <value>abc/lakefs/1234567bPxRfiCYEXAMPLEKEY</value> </property> <property> <name>fs.lakefs.endpoint</name> <value>https://example-org.us-east-1.lakefscloud.io/api/v1</value> </property> </configuration> . Add the following the cluster’s configuration under Configuration ➡️ Advanced options: . spark.hadoop.fs.lakefs.impl io.lakefs.LakeFSFileSystem spark.hadoop.fs.lakefs.access.key AKIAlakefs12345EXAMPLE spark.hadoop.fs.lakefs.secret.key abc/lakefs/1234567bPxRfiCYEXAMPLEKEY spark.hadoop.fs.s3a.access.key AKIAIOSFODNN7EXAMPLE spark.hadoop.fs.s3a.secret.key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY spark.hadoop.fs.s3a.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.lakefs.endpoint https://example-org.us-east-1.lakefscloud.io/api/v1 . Alternatively, follow this step by step Databricks integration tutorial, including lakeFS Hadoop File System, Python client and lakeFS SPARK client. ⚠️ If your bucket is on a region other than us-east-1, you may also need to configure fs.s3a.endpoint with the correct region. Amazon provides S3 endpoints you can use. Usage with TemporaryAWSCredentialsLakeFSTokenProvider . An initial setup is required - you must have AWS Auth configured with lakeFS. The TemporaryAWSCredentialsLakeFSTokenProvider depends on the caller to provide AWS credentials (e.g Assumed Role Key,Secret and Token) as input to the lakeFS client. ⚠️ Configure sts.endpoint with a valid sts regional service endpoint and it must be be equal to the region that is used for authentication first place. The only exception is us-east-1 which is the default region for STS. ⚠️ Using the current provider the lakeFS token will not renew upon expiry and the user will need to re-authenticate. PySpark example using TemporaryAWSCredentialsLakeFSTokenProvider with boto3 and AWS session credentials: . import boto3 session = boto3.session.Session() # AWS credentials used s3a to access lakeFS bucket sc._jsc.hadoopConfiguration().set(\"fs.s3a.access.key\", \"AKIAIOSFODNN7EXAMPLE\") sc._jsc.hadoopConfiguration().set(\"fs.s3a.secret.key\", \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\") sc._jsc.hadoopConfiguration().set(\"fs.s3a.endpoint\", \"https://s3.us-west-2.amazonaws.com\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.impl\", \"io.lakefs.LakeFSFileSystem\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.endpoint\", \"https://example-org.us-west-2.lakefscloud.io/api/v1\") sc._jsc.hadoopConfiguration().set(\"spark.hadoop.fs.s3a.path.style.access\", \"true\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.auth.provider\", \"io.lakefs.auth.TemporaryAWSCredentialsLakeFSTokenProvider\") # AWS tempporary session credentials to use with lakeFS sc._jsc.hadoopConfiguration().set(\"fs.lakefs.token.aws.access.key\", session.get_credentials().access_key) sc._jsc.hadoopConfiguration().set(\"fs.lakefs.token.aws.secret.key\", session.get_credentials().secret_key) sc._jsc.hadoopConfiguration().set(\"fs.lakefs.token.aws.session.token\", session.get_credentials().token) sc._jsc.hadoopConfiguration().set(\"fs.lakefs.token.aws.sts.endpoint\", \"https://sts.us-west-2.amazonaws.com\") . Usage . Hadoop FileSystem paths use the lakefs:// protocol, with paths taking the form lakefs://<repository>/<ref>/path/to/object. <ref> can be a branch, tag, or commit ID in lakeFS. Here’s an example for reading a Parquet file from lakeFS to a Spark DataFrame: . val repo = \"example-repo\" val branch = \"main\" val df = spark.read.parquet(s\"lakefs://${repo}/${branch}/example-path/example-file.parquet\") . Here’s how to write some results back to a lakeFS path: . df.write.partitionBy(\"example-column\").parquet(s\"lakefs://${repo}/${branch}/output-path/\") . The data is now created in lakeFS as new changes in your branch. You can now commit these changes or revert them. ", "url": "/integrations/spark.html#lakefs-hadoop-filesystem", "relUrl": "/integrations/spark.html#lakefs-hadoop-filesystem" - },"580": { + },"582": { "doc": "Apache Spark", "title": "Hadoop FileSystem in Presigned mode", "content": "Available starting version 0.1.13 of the FileSystem . In this mode, the lakeFS server is responsible for authenticating with your storage. The client will still perform data operations directly on the storage. To do so, it will use pre-signed storage URLs provided by the lakeFS server. When using this mode, you don’t need to configure the client with access to your storage: . | CLI | Scala | PySpark | XML Configuration | Databricks | . spark-shell --conf spark.hadoop.fs.lakefs.access.mode=presigned \\ --conf spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem \\ --conf spark.hadoop.fs.lakefs.access.key=AKIAlakefs12345EXAMPLE \\ --conf spark.hadoop.fs.lakefs.secret.key=abc/lakefs/1234567bPxRfiCYEXAMPLEKEY \\ --conf spark.hadoop.fs.lakefs.endpoint=https://example-org.us-east-1.lakefscloud.io/api/v1 \\ --packages io.lakefs:hadoop-lakefs-assembly:0.2.4 . spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.access.mode\", \"presigned\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.impl\", \"io.lakefs.LakeFSFileSystem\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.access.key\", \"AKIAlakefs12345EXAMPLE\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") spark.sparkContext.hadoopConfiguration.set(\"fs.lakefs.endpoint\", \"https://example-org.us-east-1.lakefscloud.io/api/v1\") . sc._jsc.hadoopConfiguration().set(\"fs.lakefs.access.mode\", \"presigned\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.impl\", \"io.lakefs.LakeFSFileSystem\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.access.key\", \"AKIAlakefs12345EXAMPLE\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.secret.key\", \"abc/lakefs/1234567bPxRfiCYEXAMPLEKEY\") sc._jsc.hadoopConfiguration().set(\"fs.lakefs.endpoint\", \"https://example-org.us-east-1.lakefscloud.io/api/v1\") . Make sure that you load the lakeFS FileSystem into Spark by running it with --packages or --jars, and then add these into a configuration file, e.g., $SPARK_HOME/conf/hdfs-site.xml: . <?xml version=\"1.0\"?> <configuration> <property> <name>fs.lakefs.access.mode</name> <value>presigned</value> </property> <property> <name>fs.lakefs.impl</name> <value>io.lakefs.LakeFSFileSystem</value> </property> <property> <name>fs.lakefs.access.key</name> <value>AKIAlakefs12345EXAMPLE</value> </property> <property> <name>fs.lakefs.secret.key</name> <value>abc/lakefs/1234567bPxRfiCYEXAMPLEKEY</value> </property> <property> <name>fs.lakefs.endpoint</name> <value>https://example-org.us-east-1.lakefscloud.io/api/v1</value> </property> </configuration> . Add the following the cluster’s configuration under Configuration ➡️ Advanced options: . spark.hadoop.fs.lakefs.access.mode presigned spark.hadoop.fs.lakefs.impl io.lakefs.LakeFSFileSystem spark.hadoop.fs.lakefs.access.key AKIAlakefs12345EXAMPLE spark.hadoop.fs.lakefs.secret.key abc/lakefs/1234567bPxRfiCYEXAMPLEKEY spark.hadoop.fs.lakefs.endpoint https://example-org.us-east-1.lakefscloud.io/api/v1 . ", "url": "/integrations/spark.html#hadoop-filesystem-in-presigned-mode", "relUrl": "/integrations/spark.html#hadoop-filesystem-in-presigned-mode" - },"581": { + },"583": { "doc": "Apache Spark", "title": "Apache Spark", "content": " ", "url": "/integrations/spark.html", "relUrl": "/integrations/spark.html" - },"582": { + },"584": { "doc": "Single Sign On (SSO)", "title": "Single Sign On (SSO)", "content": "lakeFS Cloud . lakeFS Enterprise . SSO is available for lakeFS Cloud and lakeFS Enterprise. If you’re using the open-source version of lakeFS you can read more about the authentication options available. ", "url": "/security/sso.html", "relUrl": "/security/sso.html" - },"583": { + },"585": { "doc": "Single Sign On (SSO)", "title": "SSO for lakeFS Cloud", "content": "lakeFS Cloud uses Auth0 for authentication and thus support the same identity providers as Auth0 including Active Directory/LDAP, ADFS, Azure Active Directory Native, Google Workspace, OpenID Connect, Okta, PingFederate, SAML, and Azure Active Directory. | Okta | AD FS | Azure AD | . ", "url": "/security/sso.html#sso-for-lakefs-cloud", "relUrl": "/security/sso.html#sso-for-lakefs-cloud" - },"584": { + },"586": { "doc": "Single Sign On (SSO)", "title": "Okta", "content": "This guide is based on Okta’s Create OIDC app integrations guide. Steps: . | Login to your Okta account | Select Applications > Applications, then Create App Integration. | Select Create New App and enter the following: . | For Sign-in method, choose OIDC. | Under Application type, choose Web app. | Select Next. | . | Under General Settings: . | App integration name, enter a name for your application. (i.e lakeFS Cloud) | . | In the Sign-in redirect URIs field, enter https://lakefs-cloud.us.auth0.com/login (United States) or https://lakefs-cloud.eu.auth0.com/login (Europe). | Under Sign-in redirect URIs, click Add URI, enter https://lakefs-cloud.us.auth0.com/login/callback (United States) or https://lakefs-cloud.eu.auth0.com/login/callback (Europe). | Under Assignments, choose the wanted Controlled access. (i.e Allow everyone in your organization to access) | Uncheck Enable immediate access with Federation Broker Mode. | Select Save. | . Once you finish registering your application with Okta, save the Client ID, Client Secret and your Okta Domain, send this to Treeverse’s team to finish the integration. ", "url": "/security/sso.html#okta", "relUrl": "/security/sso.html#okta" - },"585": { + },"587": { "doc": "Single Sign On (SSO)", "title": "Active Directory Federation Services (AD FS)", "content": "Prerequisites: . | Client’s AD FS server should be exposed publicly or to Auth0’s IP ranges (either directly or using Web Application Proxy) | . Steps: . | Connect to the AD FS server | Open AD FS’ PowerShell CLI as Administrator through the server manager | Execute the following: . (new-object Net.WebClient -property @{Encoding = [Text.Encoding]::UTF8}).DownloadString(\"https://raw.github.com/auth0/adfs-auth0/master/adfs.ps1\") | iex AddRelyingParty \"urn:auth0:lakefs-cloud\" \"https://lakefs-cloud.us.auth0.com/login/callback\" . Note: If your organization data is located in Europe, use lakefs-cloud.eu.auth0.com instead of lakefs-cloud.us.auth0.com. | . Once you finish registering lakeFS Cloud with AD FS, save the AD FS URL and send this to Treeverse’s team to finish the integration. ", "url": "/security/sso.html#active-directory-federation-services-ad-fs", "relUrl": "/security/sso.html#active-directory-federation-services-ad-fs" - },"586": { + },"588": { "doc": "Single Sign On (SSO)", "title": "Azure Active Directory (AD)", "content": "Prerequisites: . | Azure account with permissions to manage applications in Azure Active Directory | . Note: If you’ve already set up lakeFS Cloud with your Azure account, you can skip the Register lakeFS Cloud with Azure and Add client secret and go directly to Add a redirect URI. Register lakeFS Cloud with Azure . Steps: . | Sign in to the Azure portal. | If you have access to multiple tenants, use the Directories + subscriptions filter in the top menu to switch to the tenant in which you want to register the application. | Search for and select Azure Active Directory. | Under Manage, select App registrations > New registration. | Enter a display Name for your application. Users of your application might see the display name when they use the app, for example during sign-in. You can change the display name at any time and multiple app registrations can share the same name. The app registration’s automatically generated Application (client) ID, not its display name, uniquely identifies your app within the identity platform. | Specify who can use the application, sometimes called its sign-in audience. Note: don’t enter anything for Redirect URI (optional). You’ll configure a redirect URI in the next section. | Select Register to complete the initial app registration. | . When registration finishes, the Azure portal displays the app registration’s Overview pane. You see the Application (client) ID. Also called the client ID, this value uniquely identifies your application in the Microsoft identity platform. Important: new app registrations are hidden to users by default. When you are ready for users to see the app on their My Apps page you can enable it. To enable the app, in the Azure portal navigate to Azure Active Directory > Enterprise applications and select the app. Then on the Properties page toggle Visible to users? to Yes. Add a secret . Sometimes called an application password, a client secret is a string value your app can use in place of a certificate to identity itself. Steps: . | In the Azure portal, in App registrations, select your application. | Select Certificates & secrets > Client secrets > New client secret. | Add a description for your client secret. | Select an expiration for the secret or specify a custom lifetime. | Client secret lifetime is limited to two years (24 months) or less. You can’t specify a custom lifetime longer than 24 months. | Microsoft recommends that you set an expiration value of less than 12 months. | . | Select Add. | Record the secret’s value for use in your client application code. This secret value is never displayed again after you leave this page. | . Add a redirect URI . A redirect URI is the location where the Microsoft identity platform redirects a user’s client and sends security tokens after authentication. You add and modify redirect URIs for your registered applications by configuring their platform settings. Enter https://lakefs-cloud.us.auth0.com/login/callback as your redirect URI. Settings for each application type, including redirect URIs, are configured in Platform configurations in the Azure portal. Some platforms, like Web and Single-page applications, require you to manually specify a redirect URI. For other platforms, like mobile and desktop, you can select from redirect URIs generated for you when you configure their other settings. Steps: . | In the Azure portal, in App registrations, select your application. | Under Manage, select Authentication. | Under Platform configurations, select Add a platform. | Under Configure platforms, select the web option. | Select Configure to complete the platform configuration. | . Once you finish registering lakeFS Cloud with Azure AD send the following items to the Treeverse’s team: . | Client ID | Client Secret | Azure AD Domain | Identity API Version (v1 for Azure AD or v2 for Microsoft Identity Platform/Entra) | . ", "url": "/security/sso.html#azure-active-directory-ad", "relUrl": "/security/sso.html#azure-active-directory-ad" - },"587": { + },"589": { "doc": "Single Sign On (SSO)", "title": "SSO for lakeFS Enterprise", "content": "Authentication in lakeFS Enterprise is handled by a secondary service which runs side-by-side with lakeFS. With a nod to Hogwarts and their security system, we’ve named this service Fluffy. Details for configuring the supported identity providers with Fluffy are shown below. In addition, please review the necessary Helm configuration to configure Fluffy. | Active Directory Federation Services (AD FS) (using SAML) | OpenID Connect | LDAP | . If you’re using an authentication provider that is not listed please contact us for further assistance. | AD FS | OpenID Connect | LDAP | . ", "url": "/security/sso.html#sso-for-lakefs-enterprise", "relUrl": "/security/sso.html#sso-for-lakefs-enterprise" - },"588": { + },"590": { "doc": "Single Sign On (SSO)", "title": "Active Directory Federation Services (AD FS) (using SAML)", "content": "AD FS integration uses certificates to sign & encrypt requests going out from Fluffy and decrypt incoming requests from AD FS server. In order for Fluffy to work, the following values must be configured. Update (or override) the following attributes in the chart’s values.yaml file. | Replace fluffy.saml_rsa_public_cert and fluffy.saml_rsa_private_key with real certificate values | Replace fluffyConfig.auth.saml.idp_metadata_url with the metadata URL of the AD FS provider (e.g adfs-auth.company.com) | Replace fluffyConfig.auth.saml.external_user_id_claim_name with the claim name representing user id name in AD FS | Replace lakefs.company.com with your lakeFS server URL. | . If you’d like to generate the certificates using OpenSSL, you can take a look at the following example: . openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj \"/CN=lakefs.company.com\" - . lakeFS Server Configuration (Update in helm’s values.yaml file): . auth: cookie_auth_verification: auth_source: saml friendly_name_claim_name: displayName persist_friendly_name: true external_user_id_claim_name: samName default_initial_groups: - \"Developers\" logout_redirect_url: \"https://lakefs.company.com/logout-saml\" encrypt: secret_key: shared-secrey-key ui_config: login_url: \"https://lakefs.company.com/sso/login-saml\" logout_url: \"https://lakefs.company.com/sso/logout-saml\" login_cookie_names: - internal_auth_session - saml_auth_session . Fluffy Configuration (Update in helm’s values.yaml file): . logging: format: \"json\" level: \"INFO\" audit_log_level: \"INFO\" output: \"=\" auth: encrypt: secret_key: shared-secrey-key logout_redirect_url: https://lakefs.company.com post_login_redirect_url: https://lakefs.company.com saml: enabled: true sp_root_url: https://lakefs.company.com sp_x509_key_path: '/etc/saml_certs/rsa_saml_private.cert' sp_x509_cert_path: '/etc/saml_certs/rsa_saml_public.pem' sp_sign_request: true sp_signature_method: \"http://www.w3.org/2001/04/xmldsig-more#rsa-sha256\" idp_metadata_url: \"https://adfs-auth.company.com/federationmetadata/2007-06/federationmetadata.xml\" # idp_authn_name_id_format: \"urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified\" external_user_id_claim_name: samName # idp_metadata_file_path: # idp_skip_verify_tls_cert: true . OpenID Connect . In order for Fluffy to work, the following values must be configured. Update (or override) the following attributes in the chart’s values.yaml file. | Replace lakefsConfig.friendly_name_claim_name with the right claim name. | Replace lakefsConfig.default_initial_groups with desired claim name (See pre-configured groups for enterprise) | Replace fluffyConfig.auth.logout_redirect_url with your full OIDC logout URL (e.g https://oidc-provider-url.com/logout/path) | Replace fluffyConfig.auth.oidc.url with your OIDC provider URL (e.g https://oidc-provider-url.com) | Replace fluffyConfig.auth.oidc.logout_endpoint_query_parameters with parameters you’d like to pass to the OIDC provider for logout. | Replace fluffyConfig.auth.oidc.client_id and fluffyConfig.auth.oidc.client_secret with the client ID & secret for OIDC. | Replace fluffyConfig.auth.oidc.logout_client_id_query_parameter with the query parameter that represent the client_id, note that it should match the the key/query param that represents the client id and required by the specific OIDC provider. | Replace lakefs.company.com with the lakeFS server URL. | . lakeFS Server Configuration (Update in helm’s values.yaml file): . # Important: make sure to include the rest of your lakeFS Configuration here! auth: encrypt: secret_key: shared-secrey-key oidc: friendly_name_claim_name: \"name\" persist_friendly_name: true default_initial_groups: [\"Developers\"] ui_config: login_url: /oidc/login logout_url: /oidc/logout login_cookie_names: - internal_auth_session - oidc_auth_session . Fluffy Configuration (Update in helm’s values.yaml file): . logging: format: \"json\" level: \"INFO\" audit_log_level: \"INFO\" output: \"=\" installation: fixed_id: fluffy-authenticator auth: post_login_redirect_url: / logout_redirect_url: https://oidc-provider-url.com/logout/url oidc: enabled: true url: https://oidc-provider-url.com/ client_id: <oidc-client-id> client_secret: <oidc-client-secret> callback_base_url: https://lakefs.company.com is_default_login: true logout_client_id_query_parameter: client_id logout_endpoint_query_parameters: - returnTo - https://lakefs.company.com/oidc/login encrypt: secret_key: shared-secrey-key . ", "url": "/security/sso.html#active-directory-federation-services-ad-fs-using-saml", "relUrl": "/security/sso.html#active-directory-federation-services-ad-fs-using-saml" - },"589": { + },"591": { "doc": "Single Sign On (SSO)", "title": "LDAP", "content": "Fluffy is incharge of providing LDAP authentication for lakeFS Enterprise. The authentication works by querying the LDAP server for user information and authenticating the user based on the provided credentials. Important: An administrative bind user must be configured. It should have search permissions for the LDAP server that will be used to query the LDAP server for user information. For Helm: set the following attributes in the Helm chart values, for lakeFS lakefsConfig.* and fluffyConfig.* for fluffy. No Helm: If not using Helm use the YAML below to directly update the configuration file for each service. lakeFS Configuration: . | Replace auth.remote_authenticator.enabled with true | Replace auth.remote_authenticator.endpoint with the fluffy authentication server URL combined with the api/v1/ldap/login suffix (e.g http://lakefs.company.com/api/v1/ldap/login) | . fluffy Configuration: . See Fluffy configuration reference. | Repalce auth.ldap.remote_authenticator.server_endpoint with your LDAP server endpoint (e.g ldaps://ldap.ldap-address.com:636) | Replace auth.ldap.remote_authenticator.bind_dn with the LDAP bind user/permissions to query your LDAP server. | Replace auth.ldap.remote_authenticator.user_base_dn with the user base to search users in. | . lakeFS Server Configuration file: . $lakefs run -c ./lakefs.yaml . # Important: make sure to include the rest of your lakeFS Configuration here! auth: remote_authenticator: enabled: true endpoint: http://<Fluffy URL>:<Fluffy http port>/api/v1/ldap/login default_user_group: \"Developers\" # Value needs to correspond with an existing group in lakeFS ui_config: logout_url: /logout login_cookie_names: - internal_auth_session . Fluffy Configuration file: . $fluffy run -c ./fluffy.yaml . logging: format: \"json\" level: \"INFO\" audit_log_level: \"INFO\" output: \"=\" installation: fixed_id: fluffy-authenticator auth: post_login_redirect_url: / ldap: server_endpoint: 'ldaps://ldap.company.com:636' bind_dn: uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com bind_password: '<ldap pwd>' username_attribute: uid user_base_dn: ou=<some-ou>,o=<org-id>,dc=<company>,dc=com user_filter: (objectClass=inetOrgPerson) connection_timeout_seconds: 15 request_timeout_seconds: 7 . ", "url": "/security/sso.html#ldap", "relUrl": "/security/sso.html#ldap" - },"590": { + },"592": { "doc": "Single Sign On (SSO)", "title": "Troubleshooting LDAP issues", "content": "Inspecting Logs . If you encounter LDAP connection errors, you should inspect the fluffy container logs to get more information. Authentication issues . Auth issues (e.g. user not found, invalid credentials) can be debugged with the ldapwhoami CLI tool. The Examples are based on the fluffy config above: . To verify that the main bind user can connect: . ldapwhoami -H ldap://ldap.company.com:636 -D \"uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -x -W . To verify that a specific lakeFS user dev-user can connect: . ldapwhoami -H ldap://ldap.company.com:636 -D \"uid=dev-user,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -x -W . User not found issue . Upon a login request in fluffy, the bind user will search for the user in the LDAP server. If the user is not found it will be presented in the logs. We can search the user using ldapsearch CLI tool. Search ALL users in the base DN (no filters): . Note: -b is the user_base_dn, -D is bind_dn and -w is bind_password from the fluffy configuration. ldapsearch -H ldap://ldap.company.com:636 -x -b \"ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -D \"uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -w '<bind_user_pwd>' . If the user is found, we should now use filters for the specific user the same way fluffy does it and expect to see the user. For example, to repdocue the same search as fluffy does: . | user dev-user set from uid attribute in LDAP | Fluffy configuration values: user_filter: (objectClass=inetOrgPerson) and username_attribute: uid | . ldapsearch -H ldap://ldap.company.com:636 -x -b \"ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -D \"uid=<bind-user-name>,ou=<some-ou>,o=<org-id>,dc=<company>,dc=com\" -w '<bind_user_pwd>' \"(&(uid=dev-user)(objectClass=inetOrgPerson))\" . Helm . In order to use lakeFS Enterprise and Fluffy, we provided out of the box setup, see lakeFS Helm chart configuration. Notes: . | Check the examples on GitHub we provide for each authentication method (oidc/adfs/ldap + rbac). | The examples are provisioned with a Postgres pod for quick-start, make sure to replace that to a stable database once ready. | The encrypt secret key secrets.authEncryptSecretKey is shared between fluffy and lakeFS for authentication. | The lakeFS image.tag must be >= 1.0.0 | The fluffy image.tag must be >= 0.2.7 | Change the ingress.hosts[0] from lakefs.company.com to a real host (usually same as lakeFS), also update additional references in the file (note: URL path after host if provided should stay unchanged). | Update the ingress configuration with other optional fields if used | Fluffy docker image: replace the fluffy.image.privateRegistry.secretToken with real token to dockerhub for the fluffy docker image. | . ", "url": "/security/sso.html#troubleshooting-ldap-issues", "relUrl": "/security/sso.html#troubleshooting-ldap-issues" - },"591": { + },"593": { "doc": "Standalone Garbage Collection", "title": "Standalone Garbage Collection", "content": "lakeFS Enterprise . experimental . Standalone GC is only available for lakeFS Enterprise. Standalone GC is experimental and offers limited capabilities compared to the Spark-backed GC. For large scale environments, we recommend using the Spark-backed solution. ", "url": "/howto/garbage-collection/standalone-gc.html", "relUrl": "/howto/garbage-collection/standalone-gc.html" - },"592": { + },"594": { "doc": "Standalone Garbage Collection", "title": "Table of contents", "content": ". | What is Standalone GC? | Limitations | Installation . | Step 1: Obtain Dockerhub token | Step 2: Login to Dockerhub with this token | Step 3: Download the docker image | . | Setup . | Permissions | Credentials | Using S3-compatible clients | Configuration | . | How to Run Standalone GC? . | Command line reference | Get the List of Objects Marked for Deletion | Delete marked objects | . | Lab tests | . ", "url": "/howto/garbage-collection/standalone-gc.html#table-of-contents", "relUrl": "/howto/garbage-collection/standalone-gc.html#table-of-contents" - },"593": { + },"595": { "doc": "Standalone Garbage Collection", "title": "What is Standalone GC?", "content": "Standalone GC is a simplified version of the Spark-backed GC that runs without any external dependencies, delivered as a standalone docker image. It supports S3 and self-managed S3 compatible storages such as MinIO. ", "url": "/howto/garbage-collection/standalone-gc.html#what-is-standalone-gc", "relUrl": "/howto/garbage-collection/standalone-gc.html#what-is-standalone-gc" - },"594": { + },"596": { "doc": "Standalone Garbage Collection", "title": "Limitations", "content": ". | No horizontal scalability: Only a single instance of lakefs-sgc can operate on a given repository at a time. | Mark phase only: Standalone GC supports only the mark phase, identifying objects for deletion but not executing the sweep stage to delete them. It functions similarly to the GC’s mark-only mode. | Only supports AWS S3 and S3-compatible object storages. However, supporting Azure blob and GCS are in our roadmap. | . ", "url": "/howto/garbage-collection/standalone-gc.html#limitations", "relUrl": "/howto/garbage-collection/standalone-gc.html#limitations" - },"595": { + },"597": { "doc": "Standalone Garbage Collection", "title": "Installation", "content": "Step 1: Obtain Dockerhub token . lakeFS Enterprise customers . Contact your account manager to verify that Standalone GC is included in your license. Then use your dockerhub token for the externallakefs user. New to lakeFS Enterprise . Please contact us to get trial access to Standalone GC. Step 2: Login to Dockerhub with this token . docker login -u <token> . Step 3: Download the docker image . Download the image from the lakefs-sgc repository: . docker pull treeverse/lakefs-sgc:<tag> . ", "url": "/howto/garbage-collection/standalone-gc.html#installation", "relUrl": "/howto/garbage-collection/standalone-gc.html#installation" - },"596": { + },"598": { "doc": "Standalone Garbage Collection", "title": "Setup", "content": "Permissions . To run lakefs-sgc, you need both AWS (or S3-compatible) storage and lakeFS user permissions as outlined below: . Storage permissions . The minimum required permissions for AWS or S3-compatible storage are: . { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"s3:PutObject\", \"s3:GetObject\" ], \"Resource\": [ \"arn:aws:s3:::some-bucket/some/prefix/*\" ] }, { \"Effect\": \"Allow\", \"Action\": [ \"s3:ListBucket\" ], \"Resource\": [ \"arn:aws:s3:::some-bucket\" ] }, { \"Effect\": \"Allow\", \"Action\": [ \"s3:ListAllMyBuckets\" ], \"Resource\": [ \"arn:aws:s3:::*\" ] } ] } . In this example, the repository storage namespace is s3://some-bucket/some/prefix. lakeFS permissions . The minimum required permissions for lakeFS are: . { \"statement\": [ { \"action\": [ \"retention:PrepareGarbageCollectionCommits\", \"retention:PrepareGarbageCollectionUncommitted\", \"fs:ReadConfig\", \"fs:ReadRepository\", \"fs:ListObjects\", \"fs:ReadConfig\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::repository/<repository>\" } ] } . Credentials . Standalone GC supports S3 and S3-compatible storage backends and relies on AWS credentials for authentication. To set up credentials on the lakefs-sgc docker container, follow AWS guidelines, such as those outlined in this guide. For details on how to pass credentials to lakefs-sgc, refer to the instructions in How to Run Standalone GC. Using S3-compatible clients . lakefs-sgc leverages AWS credentials to work seamlessly with S3-compatible storage solutions, such as MinIO. Follow the steps below to set up and use lakefs-sgc with an S3-compatible client: . | Add a profile to your ~/.aws/config file: [profile minio] region = us-east-1 endpoint_url = <MinIO URL> s3 = signature_version = s3v4 . | Add an access and secret keys to your ~/.aws/credentials file: [minio] aws_access_key_id = <MinIO access key> aws_secret_access_key = <MinIO secret key> . | Run the lakefs-sgc docker image and pass it the minio profile - see example below. | . Configuration . The following configuration keys are available: . | Key | Description | Default value | Possible values | . | logging.format | Logs output format | “text” | “text”,”json” | . | logging.level | Logs level | “info” | “error”,”warn”,info”,”debug”,”trace” | . | logging.output | Where to output the logs to | ”-“ | ”-“ (stdout), “=” (stderr), or any string for file path | . | cache_dir | Directory to use for caching data during run | ~/.lakefs-sgc/data | string | . | aws.max_page_size | Max number of items per page when listing objects in AWS | 1000 | number | . | aws.s3.addressing_path_style | Whether or not to use path-style when reading objects from AWS | true | boolean | . | lakefs.endpoint_url | The URL to the lakeFS installation - should end with /api/v1 | NOT SET | URL | . | lakefs.access_key_id | Access key to the lakeFS installation | NOT SET | string | . | lakefs.secret_access_key | Secret access key to the lakeFS installation | NOT SET | string | . These keys can be provided in the following ways: . | Config file: Create a YAML file with the keys, each . is a new nesting level. For example, logging.level will be: logging: level: <value> # info,debug... Then, pass it to the program using the --config path/to/config.yaml argument. | Environment variables: by setting LAKEFS_SGC_<KEY>, with uppercase letters and .s converted to _s. For example logging.level will be: export LAKEFS_SGC_LOGGING_LEVEL=info . | . Example (minimalistic) config file: . logging: level: debug lakefs: endpoint_url: https://your.url/api/v1 access_key_id: <lakeFS access key> secret_access_key: <lakeFS secret key> . ", "url": "/howto/garbage-collection/standalone-gc.html#setup", "relUrl": "/howto/garbage-collection/standalone-gc.html#setup" - },"597": { + },"599": { "doc": "Standalone Garbage Collection", "title": "How to Run Standalone GC?", "content": "Command line reference . Flags . | -c, --config: config file to use (default is $HOME/.lakefs-sgc.yaml) | . Commands . run . Usage: lakefs-sgc run <repository> . Flags: . | --cache-dir: directory to cache read files (default is $HOME/.lakefs-sgc/data/) | --parallelism: number of parallel downloads for metadata files (default 10) | --presign: use pre-signed URLs when downloading/uploading data (recommended) (default true) | . To run standalone GC, choose the method you prefer to pass AWS credentials and invoke the commands below. Directly passing in credentials parsed from ~/.aws/credentials . docker run \\ -e AWS_REGION=<region> \\ -e AWS_SESSION_TOKEN=\"$(grep 'aws_session_token' ~/.aws/credentials | awk -F' = ' '{print $2}')\" \\ -e AWS_ACCESS_KEY_ID=\"$(grep 'aws_access_key_id' ~/.aws/credentials | awk -F' = ' '{print $2}')\" \\ -e AWS_SECRET_ACCESS_KEY=\"$(grep 'aws_secret_access_key' ~/.aws/credentials | awk -F' = ' '{print $2}')\" \\ -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<lakefs endpoint URL> \\ -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<lakefs accesss key> \\ -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<lakefs secret key> \\ -e LAKEFS_SGC_LOGGING_LEVEL=debug \\ treeverse/lakefs-sgc:<tag> run <repository> . Mounting the ~/.aws directory . When working with S3-compatible clients, it’s often more convenient to mount the ~/.aws directory and pass in the desired profile. First, change the permissions for ~/.aws/* to allow the docker container to read this directory: . chmod 644 ~/.aws/* . Then, run the docker image and mount ~/.aws to the lakefs-sgc home directory on the docker container: . docker run \\ --network=host \\ -v ~/.aws:/home/lakefs-sgc/.aws \\ -e AWS_REGION=us-east-1 \\ -e AWS_PROFILE=<profile> \\ -e LAKEFS_SGC_LAKEFS_ENDPOINT_URL=<lakefs endpoint URL> \\ -e LAKEFS_SGC_LAKEFS_ACCESS_KEY_ID=<lakefs accesss key> \\ -e LAKEFS_SGC_LAKEFS_SECRET_ACCESS_KEY=<lakefs secret key> \\ -e LAKEFS_SGC_LOGGING_LEVEL=debug \\ treeverse/lakefs-sgc:<tag> run <repository> . Get the List of Objects Marked for Deletion . lakefs-sgc will write its reports to <REPOSITORY_STORAGE_NAMESPACE>/_lakefs/retention/gc/reports/<RUN_ID>/. RUN_ID is generated during runtime by the Standalone GC. You can find it in the logs: . \"Marking objects for deletion\" ... run_id=gcoca17haabs73f2gtq0 . In this prefix, you’ll find 2 objects: . | deleted.csv - Containing all marked objects in a CSV containing one address column. Example: address \"data/gcnobu7n2efc74lfa5ug/csfnri7n2efc74lfa69g,_e7P9j-1ahTXtofw7tWwJUIhTfL0rEs_dvBrClzc_QE\" \"data/gcnobu7n2efc74lfa5ug/csfnri7n2efc74lfa78g,mKZnS-5YbLzmK0pKsGGimdxxBlt8QZzCyw1QeQrFvFE\" ... | summary.json - A small json summarizing the GC run. Example: { \"run_id\": \"gcoca17haabs73f2gtq0\", \"success\": true, \"first_slice\": \"gcss5tpsrurs73cqi6e0\", \"start_time\": \"2024-10-27T13:19:26.890099059Z\", \"cutoff_time\": \"2024-10-27T07:19:26.890099059Z\", \"num_deleted_objects\": 33000 } . | . Delete marked objects . We recommend starting by backing up the marked objects to a different bucket before deleting them. After ensuring the backup is complete, you can proceed to delete the objects directly from the backup location. Use the following script to backup marked objects to another bucket: . # Update these variables with your actual values storage_ns=<storage namespace (s3://...)> output_bucket=<output bucket (s3://...)> run_id=<GC run id> # Download the CSV file aws s3 cp \"$storage_ns/_lakefs/retention/gc/reports/$run_id/deleted.csv\" \"./run_id-$run_id.csv\" # Move all addresses to the output bucket under the \"run_id=$run_id\" prefix cat run_id-$run_id.csv | tail -n +2 | xargs -I {} aws s3 mv \"$storage_ns/{}\" \"$output_bucket/run_id=$run_id/\" . To delete the marked objects, use the following script: . # Update these variables with your actual values output_bucket=<output bucket (s3://...)> run_id=<GC run id> aws s3 rm $output_bucket/run_id=$run_id --recursive . Tip: Remember to periodically delete the backups to actually reduce storage costs. ", "url": "/howto/garbage-collection/standalone-gc.html#how-to-run-standalone-gc", "relUrl": "/howto/garbage-collection/standalone-gc.html#how-to-run-standalone-gc" - },"598": { + },"600": { "doc": "Standalone Garbage Collection", "title": "Lab tests", "content": "Standalone GC was tested on the lakeFS setup below. Repository spec . | 100k objects | 250 commits | 100 branches | . Machine spec . | 4GiB RAM | 8 CPUs | . Testing results . | Time: < 5m | Disk space: 123MB | . ", "url": "/howto/garbage-collection/standalone-gc.html#lab-tests", "relUrl": "/howto/garbage-collection/standalone-gc.html#lab-tests" - },"599": { + },"601": { "doc": "Short-lived token (STS like) Authentication for lakeFS", "title": "Short-lived token (STS like)", "content": "lakeFS Cloud . lakeFS Enterprise . STS Login is available in lakeFS Cloud and lakeFS Enterprise. currently only tested on lakeFS Enterprise. Secure Token Service (STS) authentication in lakeFS enables users to authenticate to lakeFS using temporary credentials obtained from an Identity Provider (IdP) via the OpenID Connect (OIDC) Authentication workflow. This document outlines the process of setting up the STS authentication flow and using the temporary credentials to interact with lakeFS through the high-level Python SDK. ", "url": "/security/sts-login.html#short-lived-token-sts-like", "relUrl": "/security/sts-login.html#short-lived-token-sts-like" - },"600": { + },"602": { "doc": "Short-lived token (STS like) Authentication for lakeFS", "title": "Login", "content": "Initiate a client session with temporary credentials using the high-level Python SDK: . import lakefs my_client = lakefs.client.from_web_identity(code = '<CODE_FROM_IDP>', state = '<STATE_FROM_IDP>' , redirect_uri = '<URI_USED_FOR_REDIRECT_FROM_IDP>', ttl_seconds = 7200) . ", "url": "/security/sts-login.html#login", "relUrl": "/security/sts-login.html#login" - },"601": { + },"603": { "doc": "Short-lived token (STS like) Authentication for lakeFS", "title": "Setup", "content": "Prerequisites . Ensure you have a way to generate the code, redirect_uri, and state values that are required to initiate a new client session using the STS login feature. For a reference implementation, see the sample implementation section. Configuration . To enable STS authentication, configure your lakeFS instance with the endpoint of the external Authentication Service. auth: authentication_api: endpoint: <url-to-remote-authenticator-endpoint> . The endpoint value should point to the external Authentication Service described at authentication.yml. Make sure to replace with the actual URL of your Authentication Service. Sample implementation to generate the code, redirect_uri, and state . The following code snippet demonstrates how to generate the values that are required to initiate a new client session using the STS login feature. Replace <your-authorize-endpoint> with the path to your IdP authorize endpoint. Examples: Auth0: The authorize endpoint will be https://<your-auth0-domain>/authorize Entra ID: The authorize endpoint will be https://<your-entra-domain>/oauth2/v2.0/authorize . import crypto from 'crypto'; import express from 'express'; import axios from 'axios'; import url from 'url'; import jsonwebtoken from 'jsonwebtoken'; const app = express(); // the local script will will spin up the server and the IdP provider will return to this endpoint the response. const callback = \"http://localhost:8080/oidc/callback\" const authorizeEndpoint = \"<your-authorize-endpoint>\" // step 1 // Create a code_verifier, which is a cryptographically-random, Base64-encoded key that will eventually be sent to Auth0 to request tokens. function base64URLEncode(str) { return str.toString('base64') .replace(/\\+/g, '-') .replace(/\\//g, '_') .replace(/=/g, ''); } var verifier = base64URLEncode(crypto.randomBytes(32)); console.log(`verifier: ${verifier}`); // step 2 // Generate a code_challenge from the code_verifier that will be sent to Auth0 to request an authorization_code. function sha256(buffer) { return crypto.createHash('sha256').update(buffer).digest(); } var challenge = base64URLEncode(sha256(verifier)); console.log(`challenge: ${challenge}`); const authorizeURL = `${authorizeEndpoint}?response_type=code&code_challenge=${challenge}&code_challenge_method=S256&client_id=${auth0ClientId}&redirect_uri=${callback}&scope=openid&state=${verifier}` console.log(`authorizeURL: ${authorizeURL}`) // Endpoint for OIDC callback app.get('/oidc/callback', async (req, res) => { try { const code = req.query.code; const state = req.query.state; console.log(`code: ${code}`); console.log(`state: ${state}`); // Return a success response res.status(200).json({ code, state, redirect_uri: callback, python-cmd: `lakefs.client.from_web_identity(code = ${code} redirect_uri = ${callback} state = ${state}, ttl_seconds = 7200) ` }); return } catch (err) { console.error(err); res.status(500).json({ message: 'Internal server error' }); } }); // Start the server const PORT = 8080; app.listen(PORT, () => { console.log(`Server is running on port ${PORT}`); }); . ", "url": "/security/sts-login.html#setup", "relUrl": "/security/sts-login.html#setup" - },"602": { + },"604": { "doc": "Short-lived token (STS like) Authentication for lakeFS", "title": "Architecture", "content": "The STS authentication flow involves several components that facilitate secure communication between the lakeFS client, lakeFS server, the remote authenticator, and the IdP. sequenceDiagram participant A as lakeFS Client participant B as lakeFS Server participant C as Remote Authenticator participant D as IdP A->>B: Call STS login endpoint B->>C: POST idp code state and redirect uri C->>D: IdP request D->>C: IdP response C->>B: Auth response B->>A: auth JWT . | lakeFS Client: Initiates the authentication process by providing IdP credentials. | lakeFS Server: Facilitates the authentication request between the client and the remote authenticator. | Remote Authenticator: Acts as a bridge between lakeFS and the IdP, handling credential validation. | IdP (Identity Provider): Validates the provided credentials and returns the authentication status. | . ", "url": "/security/sts-login.html#architecture", "relUrl": "/security/sts-login.html#architecture" - },"603": { + },"605": { "doc": "Short-lived token (STS like) Authentication for lakeFS", "title": "Short-lived token (STS like) Authentication for lakeFS", "content": " ", "url": "/security/sts-login.html", "relUrl": "/security/sts-login.html" - },"604": { + },"606": { "doc": "Troubleshooting lakeFS Enterprise", "title": "Troubleshooting lakeFS Enterprise", "content": " ", "url": "/enterprise/troubleshooting.html", "relUrl": "/enterprise/troubleshooting.html" - },"605": { + },"607": { "doc": "Troubleshooting lakeFS Enterprise", "title": "Table of contents", "content": ". | The flare command | What Information Does the flare Command Collect? | Usage - Collect and Send Flare | . A lakeFS Enterprise deployment has multiple moving parts that must all be deployed and configured correctly. This is especially true during initial setup. To help troubleshoot issues, both lakeFS and fluffy include the flare command. ", "url": "/enterprise/troubleshooting.html#table-of-contents", "relUrl": "/enterprise/troubleshooting.html#table-of-contents" - },"606": { + },"608": { "doc": "Troubleshooting lakeFS Enterprise", "title": "The flare command", "content": "Synopsis . Both lakefs and fluffy include the flare command . lakefs flare [flags] fluffy flare [flags] . Flags . --env-var-filename the name of the file environment variables will be written to (default: lakefs-env.txt) --include-env-vars should environment variables be collected by flare (default: true) -o, --output Output path relative to the current path -p, --package Package generated artifacts into a .zip file (default: false) --stdout Output to stdout instead of files (default: false) --zip-filename The name of the zip file created when the -p option is used (default: lakefs-flare.zip) . Example Usage . # This will run flare with output set to stdout, which is redirected into a file $ ./lakefs flare --stdout > lakefs.flare # The same works for fluffy $ ./fluffy flare --stdout > fluffy.flare . ", "url": "/enterprise/troubleshooting.html#the-flare-command", "relUrl": "/enterprise/troubleshooting.html#the-flare-command" - },"607": { + },"609": { "doc": "Troubleshooting lakeFS Enterprise", "title": "What Information Does the flare Command Collect?", "content": "Configuration . Both lakeFS and fluffy allow configuration to be supplied in multiple ways: configuration file, environment variables, .env files, and command flags. The flare command collects the fully resolved final configuration used by the lakeFS/fluffy process. Environment Variables . When troubleshooting, it’s important to get a view of the environment in which lakeFS/fluffy are running. This is especially true for container-based deployment environments, like Kubernetes, where env vars are used extensively. The flare command collects environment variables with the following prefixes: . | LAKEFS_ | FLUFFY_ | HTTP_ | HOSTNAME | . Sanitization and Secret Redaction . Both configuration and env var include sensitive secrets. The flare command has a multi-step process for redacting secrets from what it collects. The flare command is able to detect the following types of secrets: . | AWS static credentials | Azure storage keys | Basic HTTP auth username/password | JWTs | Bearer auth headers | GitHub tokens | Certificate private keys | . Aside from the specific secret type listed above, flare also has the ability to detect and redact generic high-entropy strings, which are likely to be secrets. Redacted secrets are replaced by a SHA512 hash of the value. This allows comparing them (e.g., between lakeFS and fluffy) without exposing the actual values. ", "url": "/enterprise/troubleshooting.html#what-information-does-the-flare-command-collect", "relUrl": "/enterprise/troubleshooting.html#what-information-does-the-flare-command-collect" - },"608": { + },"610": { "doc": "Troubleshooting lakeFS Enterprise", "title": "Usage - Collect and Send Flare", "content": "The following script is intended to be run locally and assumes that lakeFS and fluffy are deployed to a Kubernetes cluster, since this is the recommended setup. Running this script requires that kubectl be installed on the machine it is being run from and that kubectl is configured with the correct context and credentials to access the cluster. Aside from running the flare command on both lakeFS and fluffy, this script also fetches the logs from all running pods of lakeFS and fluffy. Step 1 - Set Script Variables . At the top of the script you’ll find the Variables block. It is important to change these values according to how lakeFS is deployed in your cluster. NAMESPACE - The K8s namespace where lakeFS and fluffy are deployed LAKEFS_DEPLOYMENT - The name of the lakeFS K8s deployment FLUFFY_DEPLOYMENT - The name of the fluffy K8s deployment LAKEFS_LOGS_OUTPUT_FILE - The name of the local file where lakeFS logs will be saved FLUFFY_LOGS_OUTPUT_FILE - The name of the local file where fluffy logs will be saved LAKEFS_FLARE_FILE - The name of the local file where the lakeFS flare result will be saved FLUFFY_FLARE_FILE - The name of the local file where the fluffy flare result will be saved . Step 2 - Execute the Script . #!/bin/bash RED='\\033[0;31m' NC='\\033[0m' # Variables NAMESPACE=lakefs-prod LAKEFS_DEPLOYMENT=lakefs-server FLUFFY_DEPLOYMENT=lakefs-fluffy LAKEFS_LOGS_OUTPUT_FILE=lakefs.log FLUFFY_LOGS_OUTPUT_FILE=fluffy.log LAKEFS_FLARE_FILE=lakefs.flare FLUFFY_FLARE_FILE=fluffy.flare # Find kubectl KUBECTLCMD=$(which kubectl) if [ -z \"$KUBECTLCMD\" ] then echo -e \"${RED}Couldn't find kubectl in path${NC}\" exit 1 fi $KUBECTLCMD get pods -o name -n $NAMESPACE | grep pod/$LAKEFS_DEPLOYMENT | xargs -I {} $KUBECTLCMD logs -n $NAMESPACE --all-containers=true --prefix --ignore-errors --timestamps {} > $LAKEFS_LOGS_OUTPUT_FILE $KUBECTLCMD get pods -o name -n $NAMESPACE | grep pod/$FLUFFY_DEPLOYMENT | xargs -I {} $KUBECTLCMD logs -n $NAMESPACE --all-containers=true --prefix --ignore-errors --timestamps {} > $FLUFFY_LOGS_OUTPUT_FILE $KUBECTLCMD exec deployment/$LAKEFS_DEPLOYMENT -- ./lakefs flare --stdout > $LAKEFS_FLARE_FILE $KUBECTLCMD exec deployment/$FLUFFY_DEPLOYMENT -- ./lakefs flare --stdout > $FLUFFY_FLARE_FILE . Step 3 - Inspect the Output Files . After executing the script you should have four files: lakeFS/fluffy logs and lakeFS/fluffy flare output. Before sharing these files, please review them to make sure all secrets were correctly redacted and that all the collected information is shareable. Step 4 - Zip Output Files and Attach to Support Ticket . Once you’re done inspecting the files, you can zip them into a single file and attach it to a Support Ticket in the support portal. New Ticket . When filing a new ticket, you can attach the zip file using the file upload input at the bottom of the new ticket form. Existing Ticket . To add a file to an existing ticket, click the ticket subject in the support portal. On the ticket details attach a file as a new response. You can also add text to the response, if you’d like to add further details. ", "url": "/enterprise/troubleshooting.html#usage---collect-and-send-flare", "relUrl": "/enterprise/troubleshooting.html#usage---collect-and-send-flare" - },"609": { + },"611": { "doc": "Unity Catalog", "title": "Using lakeFS with the Unity Catalog", "content": " ", "url": "/integrations/unity-catalog.html#using-lakefs-with-the-unity-catalog", "relUrl": "/integrations/unity-catalog.html#using-lakefs-with-the-unity-catalog" - },"610": { + },"612": { "doc": "Unity Catalog", "title": "Table of contents", "content": ". | Overview | Prerequisites . | Databricks authentication | . | Guide . | Table descriptor definition | Write some data | The Unity Catalog exporter script | Action configuration | Databricks Integration | . | . ", "url": "/integrations/unity-catalog.html#table-of-contents", "relUrl": "/integrations/unity-catalog.html#table-of-contents" - },"611": { + },"613": { "doc": "Unity Catalog", "title": "Overview", "content": "Databricks Unity Catalog serves as a centralized data governance platform for your data lakes. Through the Unity Catalog, you can search for and locate data assets across workspaces via a unified catalog. Leveraging the external tables feature within Unity Catalog, you can register a Delta Lake table exported from lakeFS and access it through the unified catalog. The subsequent step-by-step guide will lead you through the process of configuring a Lua hook that exports Delta Lake tables from lakeFS, and subsequently registers them in Unity Catalog. Currently, Unity Catalog export feature exclusively supports AWS S3 and Azure ADLS Gen2 as the underlying storage solution. It’s planned to support other cloud providers soon. Reference Guide: lakeFS + Unity Catalog Integration: Step-by-Step Tutorial . ", "url": "/integrations/unity-catalog.html#overview", "relUrl": "/integrations/unity-catalog.html#overview" - },"612": { + },"614": { "doc": "Unity Catalog", "title": "Prerequisites", "content": "Before starting, ensure you have the following: . | Access to Unity Catalog | An active lakeFS installation with S3 as the backing storage, and a repository in this installation. | A Databricks SQL warehouse. | AWS Credentials with S3 access. | lakeFS credentials with access to your Delta Tables. | . Supported from lakeFS v1.4.0 . Databricks authentication . Given that the hook will ultimately register a table in Unity Catalog, authentication with Databricks is imperative. Make sure that: . | You have a Databricks Service Principal. | The Service principal has token usage permissions, and an associated token configured. | The service principal has the Service principal: Manager privilege over itself (Workspace: Admin console -> Service principals -> <service principal> -> Permissions -> Grant access (<service principal>: Service principal: Manager), with Workspace access and Databricks SQL access checked (Admin console -> Service principals -> <service principal> -> Configurations). | Your SQL warehouse allows the service principal to use it (SQL Warehouses -> <SQL warehouse> -> Permissions -> <service principal>: Can use). | The catalog grants the USE CATALOG, USE SCHEMA, CREATE SCHEMA permissions to the service principal(Catalog -> <catalog name> -> Permissions -> Grant -> <service principal>: USE CATALOG, USE SCHEMA, CREATE SCHEMA). | You have an External Location configured, and the service principal has the CREATE EXTERNAL TABLE permission over it (Catalog -> External Data -> External Locations -> Create location). | . ", "url": "/integrations/unity-catalog.html#prerequisites", "relUrl": "/integrations/unity-catalog.html#prerequisites" - },"613": { + },"615": { "doc": "Unity Catalog", "title": "Guide", "content": "Table descriptor definition . To guide the Unity Catalog exporter in configuring the table in the catalog, define its properties in the Delta Lake table descriptor. The table descriptor should include (at minimum) the following fields: . | name: The table name. | type: Should be delta. | catalog: The name of the catalog in which the table will be created. | path: The path in lakeFS (starting from the root of the branch) in which the Delta Lake table’s data is found. | . Let’s define the table descriptor and upload it to lakeFS: . Save the following as famous-people-td.yaml: . --- name: famous_people type: delta catalog: my-catalog-name path: tables/famous-people . It’s recommended to create a Unity catalog with the same name as your repository . Upload the table descriptor to _lakefs_tables/famous-people-td.yaml and commit: . lakectl fs upload lakefs://repo/main/_lakefs_tables/famous-people-td.yaml -s ./famous-people-td.yaml && \\ lakectl commit lakefs://repo/main -m \"add famous people table descriptor\" . Write some data . Insert data into the table path, using your preferred method (e.g. Spark), and commit upon completion. We shall use Spark and lakeFS’s S3 gateway to write some data as a Delta table: . pyspark --packages \"io.delta:delta-spark_2.12:3.0.0,org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262\" \\ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \\ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \\ --conf spark.hadoop.fs.s3a.aws.credentials.provider='org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider' \\ --conf spark.hadoop.fs.s3a.endpoint='<LAKEFS_SERVER_URL>' \\ --conf spark.hadoop.fs.s3a.access.key='<LAKEFS_ACCESS_KEY>' \\ --conf spark.hadoop.fs.s3a.secret.key='<LAKEFS_SECRET_ACCESS_KEY>' \\ --conf spark.hadoop.fs.s3a.path.style.access=true . data = [ ('James','Bond','England','intelligence'), ('Robbie','Williams','England','music'), ('Hulk','Hogan','USA','entertainment'), ('Mister','T','USA','entertainment'), ('Rafael','Nadal','Spain','professional athlete'), ('Paul','Haver','Belgium','music'), ] columns = [\"firstname\",\"lastname\",\"country\",\"category\"] df = spark.createDataFrame(data=data, schema = columns) df.write.format(\"delta\").mode(\"overwrite\").partitionBy(\"category\", \"country\").save(\"s3a://repo/main/tables/famous-people\") . The Unity Catalog exporter script . For code references check delta_exporter and unity_exporter docs. Create unity_exporter.lua: . local aws = require(\"aws\") local formats = require(\"formats\") local databricks = require(\"databricks\") local delta_export = require(\"lakefs/catalogexport/delta_exporter\") local unity_export = require(\"lakefs/catalogexport/unity_exporter\") local sc = aws.s3_client(args.aws.access_key_id, args.aws.secret_access_key, args.aws.region) -- Export Delta Lake tables export: local delta_client = formats.delta_client(args.lakefs.access_key_id, args.lakefs.secret_access_key, args.aws.region) local delta_table_details = delta_export.export_delta_log(action, args.table_defs, sc.put_object, delta_client, \"_lakefs_tables\") -- Register the exported table in Unity Catalog: local databricks_client = databricks.client(args.databricks_host, args.databricks_token) local registration_statuses = unity_export.register_tables(action, \"_lakefs_tables\", delta_table_details, databricks_client, args.warehouse_id) for t, status in pairs(registration_statuses) do print(\"Unity catalog registration for table \\\"\" .. t .. \"\\\" completed with commit schema status : \" .. status .. \"\\n\") end . Upload the lua script to the main branch under scripts/unity_exporter.lua and commit: . lakectl fs upload lakefs://repo/main/scripts/unity_exporter.lua -s ./unity_exporter.lua && \\ lakectl commit lakefs://repo/main -m \"upload unity exporter script\" . Action configuration . Define an action configuration that will run the above script after a commit is completed (post-commit) over the main branch. Create unity_exports_action.yaml: . --- name: unity_exports on: post-commit: branches: [\"main\"] hooks: - id: unity_export type: lua properties: script_path: scripts/unity_exporter.lua args: aws: access_key_id: <AWS_ACCESS_KEY_ID> secret_access_key: <AWS_SECRET_ACCESS_KEY> region: <AWS_REGION> lakefs: # provide credentials of a user that has access to the script and Delta Table access_key_id: <LAKEFS_ACCESS_KEY_ID> secret_access_key: <LAKEFS_SECRET_ACCESS_KEY> table_defs: # an array of table descriptors used to be defined in Unity Catalog - famous-people-td databricks_host: <DATABRICKS_HOST_URL> databricks_token: <DATABRICKS_SERVICE_PRINCIPAL_TOKEN> warehouse_id: <WAREHOUSE_ID> . Upload the action configurations to _lakefs_actions/unity_exports_action.yaml and commit: . Once the commit will finish its run, the action will start running since we’ve configured it to run on post-commit events on the main branch. lakectl fs upload lakefs://repo/main/_lakefs_actions/unity_exports_action.yaml -s ./unity_exports_action.yaml && \\ lakectl commit lakefs://repo/main -m \"upload action and run it\" . The action has run and exported the famous_people Delta Lake table to the repo’s storage namespace, and has register the table as an external table in Unity Catalog under the catalog my-catalog-name, schema main (as the branch’s name) and table name famous_people: my-catalog-name.main.famous_people. Databricks Integration . After registering the table in Unity, you can leverage your preferred method to query the data from the exported table under my-catalog-name.main.famous_people, and view it in the Databricks’s Catalog Explorer, or retrieve it using the Databricks CLI with the following command: . databricks tables get my-catalog-name.main.famous_people . ", "url": "/integrations/unity-catalog.html#guide", "relUrl": "/integrations/unity-catalog.html#guide" - },"614": { + },"616": { "doc": "Unity Catalog", "title": "Unity Catalog", "content": " ", "url": "/integrations/unity-catalog.html", "relUrl": "/integrations/unity-catalog.html" - },"615": { + },"617": { "doc": "Unity Delta Sharing", "title": "Unity Delta Sharing", "content": "lakeFS Cloud . Please note, as of June 15th 2024, the Unity Delta Sharing feature will be removed. To integrate lakeFS with Unity Catalog, refer to the Unity integration docs. ", "url": "/howto/unity-delta-sharing.html", "relUrl": "/howto/unity-delta-sharing.html" - },"616": { + },"618": { "doc": "Unity Delta Sharing", "title": "Introduction", "content": "lakeFS Unity Delta Sharing provides a read-only experience from Unity Catalog for lakeFS customers. Currently, this is available as a private preview. It provides full read-only functionality for Unity Catalog. It does not provide a “self-service” experience to set up the service. ", "url": "/howto/unity-delta-sharing.html#introduction", "relUrl": "/howto/unity-delta-sharing.html#introduction" - },"617": { + },"619": { "doc": "Unity Delta Sharing", "title": "Setup", "content": "This guide explains how to set up and use lakeFS Delta Sharing. Currently, you will have to configure lakeFS Delta Sharing in collaboration with Treeverse Customer Success. Once set up is complete, you will of course be able to use lakeFS Delta Sharing on existing and on new tables without further assistance. 1. Collect data and initial setup . | Select a Delta Sharing configuration URL. This is a single location on lakeFS to hold the top-level configuration of lakeFS Delta Sharing across all repositories of your organization. Typically, it will have the form lakefs://REPO/main/lakefs_delta_sharing.yaml for one of your repositories. A longer path may be supplied - however, we do recommend keeping it on the main branch, as this object represents state for the entire installation. | Create a user lakefs-delta-sharing-service for lakeFS Delta Sharing, and an access key for that user. It should have at least read permissions for the configuration URL and for all repositories and all data accesses by Unity. lakeFS Delta Sharing will these credentials to communicate with lakeFS. | . Communicate these items to Customer Success: . | Configuration URL | Access key ID and secret access key for user lakefs-delta-sharing-service. | . Note: All YAML files extensions used in this guide must be yaml. Do not use a yml extension instead. 2. Initial configuration . Select a secret authorization token to share Unity catalog. Unity catalog will use to authenticate to the lakeFS Delta Sharing server. You might use this command on Linux: . head -c 50 /dev/random | base64 . Create a file lakefs_delta_sharing.yaml and place it at the config URL selected above. It should look like this: . authorization_token: \"GENERATED TOKEN\" # Map lakeFS repositories to Unity shares repositories: - id: sample-repo share_name: undev # List branches and prefixes to export. Each of these branches (and only # these branches) will be available as a schema on Unity. branches: - main - staging - dev_* - id: repo2 share_name: share_two branches: - \"*\" . Note that a plain “*” line must be quoted in YAML. Upload it to your config URL. For instance if the config URL is lakefs://repo/main/lakefs_delta_sharing.yaml, you might use: . lakectl fs upload -s ./lakefs_delta_sharing.yaml lakefs://repo/main/lakefs_delta_sharing.yaml . 3. Connect Unity to lakeFS Delta Sharing! . You now need to configure Unity to use the lakeFS Delta Sharing server. Create a share provider file config.share.json; see the Delta Sharing manual: . { \"shareCredentialsVersion\": 1, \"endpoint\": \"https://ORG_ID.REGION.lakefscloud.io/service/delta-sharing/v1\", \"bearerToken\": \"GENERATED TOKEN\", \"expirationTime\": \"2030-01-01T00:00:00.0Z\" } . “GENERATED TOKEN” is the secret authorization token use above. Install the databricks cli. We will use it to create Delta Share on Unity. Follow the instructions to configure it. Run the provider creation command: . databricks unity-catalog providers create \\ --name lakefs-cloud \\ --recipient-profile-json-file config.share.json . Go to “Data » Delta Sharing” on the DataBricks environment. Once Treeverse have configured lakeFS Delta Sharing on your account with your config URL, the “lakefs-cloud” provider should appear under “Shared with me”. Click the provider to see its shares. You can now create a catalog from these shares. And you can see schemas for each of the branches that you configured in the share. Here branch name dev_experiment1 matches the pattern dev_* that we defined in the configuration object lakefs-delta-sharing.yaml, so it appears as a schema. At this point you have configured Delta Sharing on lakeFS, and DataBricks to communicate with lakeFS delta sharing. No further Treeverse involvement is required. Updates to lakefs_delta_sharing.yaml will update within a minute of uploading a new version. 4. Configure tables . Everything is ready: lakeFS repositories are configured as shares, and branches are configured as schemas. Now you can define tables! Once a repository is shared, its tables are configured as a table descriptor object on the repository on the path _lakefs_tables/TABLE.yaml. Delta Lake tables . Delta Lake format includes full metadata, so you only need to configure the prefix: . name: users type: delta path: path/to/users/ . Note: The filename of the ‘yaml’ file containing the table definition must match the ‘name’ of the table itself. In the example above, ‘_lakefs_tables/users.yaml’. When placed inside _lakefs_tables/users.yaml this defines a table users on the prefix path/to/users/ (so path/to/users/ holds the prefix _delta_log). Hive tables . Hive metadata server tables are essentially just a set of objects that share a prefix, with no table metadata stored on the object store. You need to configure prefix, partitions, and schema. name: clicks type: hive path: path/to/clicks/ partition_columns: ['year'] schema: type: struct fields: - name: year type: integer nullable: false metadata: {} - name: page type: string nullable: false metadata: {} - name: site type: string nullable: true metadata: comment: a comment about this column . Useful types recognized by DataBricks Photon include integer, long, short, string, double, float, date, and timestamp. For exact type mappings, and whether to specify a field as nullable: false, refer to DataBricks Photon documentation. ", "url": "/howto/unity-delta-sharing.html#setup", "relUrl": "/howto/unity-delta-sharing.html#setup" - },"618": { + },"620": { "doc": "Upgrade lakeFS", "title": "Upgrading lakeFS", "content": "Note: For a fully managed lakeFS service with guaranteed SLAs, try lakeFS Cloud . Upgrading lakeFS from a previous version usually just requires re-deploying with the latest image (or downloading the latest version if you’re using the binary). If you’re upgrading, check whether the release requires a migration. ", "url": "/howto/deploy/upgrade.html#upgrading-lakefs", "relUrl": "/howto/deploy/upgrade.html#upgrading-lakefs" - },"619": { + },"621": { "doc": "Upgrade lakeFS", "title": "When DB migrations are required", "content": "lakeFS 0.103.0 or greater . Version 0.103.0 added support for rolling KV upgrade. This means that users who already migrated to the KV ref-store (versions 0.80.0 and above) no longer have to pass through specific versions for migration. This includes ACL migration which was introduced in lakeFS version 0.98.0. Running lakefs migrate up on the latest lakeFS version will perform all the necessary migrations up to that point. lakeFS 0.80.0 or greater (KV Migration) . Starting with version 0.80.2, lakeFS has transitioned from using a PostgreSQL based database implementation to a Key-Value datastore interface supporting multiple database implementations. More information can be found here. Users upgrading from a previous version of lakeFS must pass through the KV migration version (0.80.2) before upgrading to newer versions of lakeFS. IMPORTANT: Pre Migrate Requirements . | Users using OS environment variables for database configuration must define the connection_string explicitly or as environment variable before proceeding with the migration. | Database storage free capacity of at least twice the amount of the currently used capacity | It is strongly recommended to perform these additional steps: . | Commit all uncommitted data on branches | Create a snapshot of your database | . | By default, old database tables are not being deleted by the migration process, and should be removed manually after a successful migration. To enable table drop as part of the migration, set the database.drop_tables configuration param to true | . Migration Steps . For each lakeFS instance currently running with the database . | Modify the database section under lakeFS configuration yaml: . | Add type field with \"postgres\" as value | Copy the current configuration parameters to a new section called postgres | . --- database: type: \"postgres\" connection_string: \"postgres://localhost:5432/postgres?sslmode=disable\" max_open_connections: 20 postgres: connection_string: \"postgres://localhost:5432/postgres?sslmode=disable\" max_open_connections: 20 . | Stop all lakeFS instances | Using the lakefs binary for the new version (0.80.2), run the following: . lakefs migrate up . | lakeFS will run the migration process, which in the end should display the following message with no errors: . time=\"2022-08-10T14:46:25Z\" level=info msg=\"KV Migration took 717.629563ms\" func=\"pkg/logging.(*logrusEntryWrapper).Infof\" file=\"build/pkg/logging/logger.go:246\" TempDir=/tmp/kv_migrate_2913402680 . | It is now possible to remove the old database configuration. The updated configuration should look as such: . --- database: type: \"postgres\" postgres: connection_string: \"postgres://localhost:5432/postgres?sslmode=disable\" max_open_connections: 20 . | Deploy (or run) the new version of lakeFS. | . lakeFS 0.30.0 or greater . In case migration is required, you first need to stop the running lakeFS service. Using the lakefs binary for the new version, run the following: . lakefs migrate up . Deploy (or run) the new version of lakeFS. Note that an older version of lakeFS cannot run on a migrated database. Prior to lakeFS 0.30.0 . Note: with lakeFS < 0.30.0, you should first upgrade to 0.30.0 following this guide. Then, proceed to upgrade to the newest version. Starting version 0.30.0, lakeFS handles your committed metadata in a new way, which is more robust and has better performance. To move your existing data, you will need to run the following upgrade commands. Verify lakeFS version == 0.30.0 (can skip if using Docker) . lakefs --version . Migrate data from the previous format: . lakefs migrate db . Or migrate using Docker image: . docker run --rm -it -e LAKEFS_DATABASE_CONNECTION_STRING=<database connection string> treeverse/lakefs:rocks-migrate migrate db . Once migrated, it is possible to now use more recent lakeFS versions. Please refer to their release notes for more information on ugrading and usage). If you want to start over, discarding your existing data, you need to explicitly state this in your lakeFS configuration file. To do so, add the following to your configuration (relevant only for 0.30.0): . cataloger: type: rocks . ", "url": "/howto/deploy/upgrade.html#when-db-migrations-are-required", "relUrl": "/howto/deploy/upgrade.html#when-db-migrations-are-required" - },"620": { + },"622": { "doc": "Upgrade lakeFS", "title": "Data Migration for Version v0.50.0", "content": "If you are using a version before 0.50.0, you must first perform the previous upgrade to that version. {: note: .note-warning } . ", "url": "/howto/deploy/upgrade.html#data-migration-for-version-v0500", "relUrl": "/howto/deploy/upgrade.html#data-migration-for-version-v0500" - },"621": { + },"623": { "doc": "Upgrade lakeFS", "title": "Upgrade lakeFS", "content": " ", "url": "/howto/deploy/upgrade.html", "relUrl": "/howto/deploy/upgrade.html" - },"622": { + },"624": { "doc": "Upgrade", "title": "Upgrade", "content": "For upgrading from lakeFS enterprise to a newer version see lakefs migration. ", "url": "/enterprise/upgrade.html", "relUrl": "/enterprise/upgrade.html" - },"623": { + },"625": { "doc": "Versioning Internals", "title": "Versioning Internals", "content": " ", "url": "/understand/how/versioning-internals.html", "relUrl": "/understand/how/versioning-internals.html" - },"624": { + },"626": { "doc": "Versioning Internals", "title": "Table of contents", "content": ". | Overview | SSTable File Format (“Graveler File”) | Constructing a consistent view of the keyspace (i.e., a commit) | Representing references and uncommitted metadata | . ", "url": "/understand/how/versioning-internals.html#table-of-contents", "relUrl": "/understand/how/versioning-internals.html#table-of-contents" - },"625": { + },"627": { "doc": "Versioning Internals", "title": "Overview", "content": "Since commits in lakeFS are immutable, they are easy to store on an immutable object store. Older commits are rarely accessed, while newer commits are accessed very frequently, a tiered storage approach can work very well - the object store is the source of truth, while local disk and even RAM can be used to cache the more frequently accessed ones. Since they are immutable - once cached, you only need to evict them when space is running out. There’s no complex invalidation that needs to happen. In terms of storage format, commits are be stored as SSTables, compatible with RocksDB. SSTables were chosen as a storage format for 3 major reasons: . | Extremely high read throughput on modern hardware: using commits representing a 200m object repository (modeled after the S3 inventory of one of our design partners), we were able to achieve close to 500k random GetObject calls / second. This provides a very high throughput/cost ratio, probably as high as can be achieved on public clouds. | Being a known storage format means it’s relatively easy to generate and consume. Storing it in the object store makes it accessible to data engineering tools for analysis and distributed computation, effectively reducing the silo effect of storing it in an operational database. | The SSTable format supports delta encoding for keys which makes them very space efficient for data lakes where many keys share the same common prefixes. | . Each lakeFS commit is represented as a set of contiguous, non-overlapping SSTables that make up the entire keyspace of a repository at that commit. ", "url": "/understand/how/versioning-internals.html#overview", "relUrl": "/understand/how/versioning-internals.html#overview" - },"626": { + },"628": { "doc": "Versioning Internals", "title": "SSTable File Format (“Graveler File”)", "content": "lakeFS metadata is encoded into a format called “Graveler” - a standardized way to encode content-addressable key value pairs. This is what a Graveler file looks like: . Each Key/Value pair (“ValueRecord”) is constructed of a key, identity, and value. A simple identity could be, for example, a sha256 hash of the value’s bytes. It could be any sequence of bytes that uniquely identifies the value. As far as the Graveler is concerned, two ValueRecords are considered identical if their key and identity fields are equal. A Graveler file itself is content-addressable, i.e., similarly to Git, the name of the file is its identity. File identity is calculated based on the identity of the ValueRecords the file contains: . valueRecordID = h(h(valueRecord.key) || h(valueRecord.Identity)) fileID = h(valueRecordID1 + … + valueRecordIDN) . ", "url": "/understand/how/versioning-internals.html#sstable-file-format-graveler-file", "relUrl": "/understand/how/versioning-internals.html#sstable-file-format-graveler-file" - },"627": { + },"629": { "doc": "Versioning Internals", "title": "Constructing a consistent view of the keyspace (i.e., a commit)", "content": "We have two additional requirements for the storage format: . | Be space and time efficient when creating a commit - assuming a commit changes a single object out of a billion, we don’t want to write a full snapshot of the entire repository. Ideally, we’ll be able to reuse some data files that haven’t changed to make the commit operations (in both space and time) proportional to the size of the difference as opposed to the total size of the repository. | Allow an efficient diff between commits which runs in time proportional to the size of their difference and not their absolute sizes. | . To support these requirements, we decided to essentially build a 2-layer Merkle tree composed of a set of leaf nodes (“Range”) addressed by their content address, and a “Meta Range”, which is a special range containing all ranges, thus representing an entire consistent view of the keyspace: . Assuming commit B is derived from commit A, and only changed files in range e-f, it can reuse all ranges except for SSTable #N (the one containing the modified range of keys), which will be recreated with a new hash representing the state as exists after applying commit B’s changes. This will, in turn, also create a new Metarange since its hash is now changed as well (as it is derived from the hash of all contained ranges). Assuming most commits usually change related objects (i.e., that are likely to share some common prefix), the reuse ratio could be very high. We tested this assumption using S3 inventory from 2 design partners - we partitioned the keyspace to an arbitrary number of simulated blocks and measured their change over time. We saw a daily change rate of about 5-20%. Given the size of the repositories, it’s safe to assume that a single day would translate into multiple commits. At a modest 20 commits per day, a commit is expected to reuse >= 99% of the previous commit blocks, so acceptable in terms of write amplification generated on commit. On the object store, ranges are stored in the following hierarchy: . <lakefs root> _lakefs/ <range hash1> <range hash2> <range hashN> ... <metarange hash1> <metarange hash2> <metarange hashN> ... <data object hash1> <data object hash2> <data object hashN> ... Note: This relatively flat structure could be modified in the future. Looking at the diagram above, it imposes no real limitations on the depth of the tree. A tree could easily be made recursive by having Meta Ranges point to other Meta Ranges - and still provide all the same characteristics. For simplicity, we decided to start with a fixed 2-level hierarchy. ", "url": "/understand/how/versioning-internals.html#constructing-a-consistent-view-of-the-keyspace-ie-a-commit", "relUrl": "/understand/how/versioning-internals.html#constructing-a-consistent-view-of-the-keyspace-ie-a-commit" - },"628": { + },"630": { "doc": "Versioning Internals", "title": "Representing references and uncommitted metadata", "content": "lakeFS always stores the object data in the storage namespace in the user’s object store, committed and uncommitted data alike. However, the lakeFS object metadata might be stored in either the object store or a key-value store. Unlike committed metadata which is immutable, uncommitted (or “staged”) metadata experiences frequent random writes and is very mutable in nature. This is also true for “refs” - in particular, branches, which are simply pointers to an underlying commit, are modified frequently: on every commit or merge operation. Both these types of metadata are not only mutable, but also require strong consistency guarantees while also being fault tolerant. If we can’t access the current pointer of the main branch, a big portion of the system is essentially down. Luckily, this is also much smaller set of metadata compared to the committed metadata. References and uncommitted metadata are currently stored on a key-value store (See supported databases) for consistency guarantees. ", "url": "/understand/how/versioning-internals.html#representing-references-and-uncommitted-metadata", "relUrl": "/understand/how/versioning-internals.html#representing-references-and-uncommitted-metadata" - },"629": { + },"631": { "doc": "Vertex AI", "title": "Using Vertex AI with lakeFS", "content": "Vertex AI lets Google Cloud users Build, deploy, and scale machine learning (ML) models faster, with fully managed ML tools for any use case. lakeFS Works with Vertex AI by allowing users to create repositories on GCS Buckets, then use either the Dataset API to create managed Datasets on top of lakeFS version, or by automatically exporting lakeFS object versions in a way readable by Cloud Storage Mounts. ", "url": "/integrations/vertex_ai.html#using-vertex-ai-with-lakefs", "relUrl": "/integrations/vertex_ai.html#using-vertex-ai-with-lakefs" - },"630": { + },"632": { "doc": "Vertex AI", "title": "Table of contents", "content": ". | Using lakeFS with Vertex Managed Datasets | Using lakeFS with Cloud Storage Fuse | . ", "url": "/integrations/vertex_ai.html#table-of-contents", "relUrl": "/integrations/vertex_ai.html#table-of-contents" - },"631": { + },"633": { "doc": "Vertex AI", "title": "Using lakeFS with Vertex Managed Datasets", "content": "Vertex’s ImageDataset and VideoDataset allow creating a dataset by importing a CSV file from gcs (see gcs_source). This CSV file contains GCS addresses of image files and their corresponding labels. Since the lakeFS API supports exporting the underlying GCS address of versioned objects, we can generate such a CSV file when creating the dataset: . #!/usr/bin/env python # Requirements: # google-cloud-aiplatform>=1.31.0 # lakefs-client>=0.107.0 import csv from pathlib import PosixPath from io import StringIO import lakefs_client from lakefs_client.client import LakeFSClient from google.cloud import storage from google.cloud import aiplatform # lakeFS connection details configuration = lakefs_client.Configuration() configuration.username = 'AKIAIOSFODNN7EXAMPLE' configuration.password = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' configuration.host = 'https://lakefs.example.com/' client = LakeFSClient(configuration) # Dataset configuration lakefs_repo = 'my-repository' lakefs_ref = 'main' img_dataset = 'datasets/my-images/' # Vertex configuration import_bucket = 'underlying-gcs-bucket' # produce import file for Vertex's SDK buf = StringIO() csv_writer = csv.writer(buf) has_more = True next_offset = \"\" while has_more: files = client.objects_api.list_objects( lakefs_repo, lakefs_ref, prefix=img_dataset, after=next_offset) for r in files.get('results'): p = PosixPath(r.path) csv_writer.writerow((r.physical_address, p.parent.name)) has_more = files.get('pagination').get('has_more') next_offset = files.get('pagination').get('next_offset') # spit out CSV print('generated path and labels CSV') buf.seek(0) # Write it to storage storage_client = storage.Client() bucket = storage_client.bucket(import_bucket) blob = bucket.blob(f'vertex/imports/{lakefs_repo}/{lakefs_ref}/labels.csv') with blob.open('w') as out: out.write(buf.read()) print(f'wrote csv to gs: gs://{import_bucket}/vertex/imports/{lakefs_repo}/{lakefs_ref}/labels.csv') # import in vertex, as dataset print('Importing dataset...') ds = aiplatform.ImageDataset.create( display_name=f'{lakefs_repo}_{lakefs_ref}_imgs', gcs_source=f'gs://{import_bucket}/vertex/imports/{lakefs_repo}/{lakefs_ref}/labels.csv', import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification, sync=True ) ds.wait() print(f'Done! {ds.display_name} ({ds.resource_name})') . ", "url": "/integrations/vertex_ai.html#using-lakefs-with-vertex-managed-datasets", "relUrl": "/integrations/vertex_ai.html#using-lakefs-with-vertex-managed-datasets" - },"632": { + },"634": { "doc": "Vertex AI", "title": "Using lakeFS with Cloud Storage Fuse", "content": "Vertex allows using Google Cloud Storage mounted as a Fuse Filesystem as custom input for training jobs. Instead of having to copy lakeFS files for each version we want to consume, we can create symlinks by using gcsfuse’s native symlink inodes. This process can be fully automated by using the example gcsfuse_symlink_exporter.lua Lua hook. Here’s what we need to do: . | Upload the example .lua file into our lakeFS repository. For this example, we’ll put it under scripts/gcsfuse_symlink_exporter.lua. | Create a new hook definition file and upload to _lakefs_actions/export_images.yaml: | . --- # Example hook declaration: (_lakefs_actions/export_images.yaml): name: export_images on: post-commit: branches: [\"main\"] post-merge: branches: [\"main\"] post-create-tag: hooks: - id: gcsfuse_export_images type: lua properties: script_path: scripts/export_gcs_fuse.lua # Path to the script we uploaded in the previous step args: prefix: \"datasets/images/\" # Path we want to export every commit destination: \"gs://my-bucket/exports/my-repo/\" # Where should we create the symlinks? mount: from: \"gs://my-bucket/repos/my-repo/\" # Symlinks are to a unix-mounted file to: \"/gcs/my-bucket/repos/my-repo/\" # This will ensure they point to a location that exists. # Should be the contents of a valid credentials.json file # See: https://developers.google.com/workspace/guides/create-credentials # Will be used to write the symlink files gcs_credentials_json_string: | { \"client_id\": \"...\", \"client_secret\": \"...\", \"refresh_token\": \"...\", \"type\": \"...\" } . Done! On the next tag creation or update to the main branch, we’ll automatically export the lakeFS version of datasets/images/ to a mountable location. To consume the symlink-ed files, we can read them normally from the mount: . with open('/gcs/my-bucket/exports/my-repo/branches/main/datasets/images/001.jpg') as f: image_data = f.read() . Previously exported commits are also readable, if we exported them in the past: . commit_id = 'abcdef123deadbeef567' with open(f'/gcs/my-bucket/exports/my-repo/commits/{commit_id}/datasets/images/001.jpg') as f: image_data = f.read() . Considerations when using lakeFS with Cloud Storage Fuse . For lakeFS paths to be readable by gcsfuse, the mount option --implicit-dirs must be specified. ", "url": "/integrations/vertex_ai.html#using-lakefs-with-cloud-storage-fuse", "relUrl": "/integrations/vertex_ai.html#using-lakefs-with-cloud-storage-fuse" - },"633": { + },"635": { "doc": "Vertex AI", "title": "Vertex AI", "content": " ", "url": "/integrations/vertex_ai.html", "relUrl": "/integrations/vertex_ai.html" - },"634": { + },"636": { "doc": "S3 Virtual-host addressing (advanced)", "title": "Configuring lakeFS to use S3 Virtual-Host addressing", "content": " ", "url": "/howto/virtual-host-addressing.html#configuring-lakefs-to-use-s3-virtual-host-addressing", "relUrl": "/howto/virtual-host-addressing.html#configuring-lakefs-to-use-s3-virtual-host-addressing" - },"635": { + },"637": { "doc": "S3 Virtual-host addressing (advanced)", "title": "Understanding virtual-host addressing", "content": "Some systems require S3 endpoints (such as lakeFS’s S3 Gateway) to support virtual-host style addressing. lakeFS supports this, but requires some configuration in order to extract the bucket name (used as the lakeFS repository ID) from the host address. For example: . GET http://foo.example.com/some/location . There are two ways to interpret the URL above: . | as a virtual-host URL where the endpoint URL is example.com, the bucket name is foo, and the path is /some/location | as a path-based URL where the endpoint is foo.example.com, the bucket name is some and the path is location | . By default, lakeFS reads URLs as path-based. To read the URL as a virtual-host request, lakeFS requires additional configuration which includes defining an explicit set of DNS records for the lakeFS S3 gateway. ", "url": "/howto/virtual-host-addressing.html#understanding-virtual-host-addressing", "relUrl": "/howto/virtual-host-addressing.html#understanding-virtual-host-addressing" - },"636": { + },"638": { "doc": "S3 Virtual-host addressing (advanced)", "title": "Adding an explicit S3 domain name to the S3 Gateway configuration", "content": "The first step would be to tell the lakeFS installation which hostnames are used for the S3 Gateway. This should be a different DNS record from the one used for e.g. the UI or API. Typically, if the lakeFS installation is served under lakefs.example.com, a good choice would be s3.lakefs.example.com. This could be done using either an environment variable: . LAKEFS_GATEWAYS_S3_DOMAIN_NAME=\"s3.lakefs.example.com\" . Or by adding the gateways.s3.domain_name setting to the lakeFS config.yaml file: . --- database: connection_string: \"...\" ... # This section defines an explict S3 gateway address that supports virtual-host addressing gateways: s3: domain_name: s3.lakefs.example.com . For more information on how to configure lakeFS, check out the configuration reference . ", "url": "/howto/virtual-host-addressing.html#adding-an-explicit-s3-domain-name-to-the-s3-gateway-configuration", "relUrl": "/howto/virtual-host-addressing.html#adding-an-explicit-s3-domain-name-to-the-s3-gateway-configuration" - },"637": { + },"639": { "doc": "S3 Virtual-host addressing (advanced)", "title": "Setting up the appropriate DNS records", "content": "Once our lakeFS installation is configured with an explicit S3 gateway endpoint address, we need to define 2 DNS records and have them point at our lakeFS installation. This requires 2 CNAME records: . | s3.lakefs.example.com - CNAME to lakefs.example.com. This would be used as the S3 endpoint when configuring clients and will serve as our bare domain. | *.s3.lakefs.example.com - Also a CNAME to lakefs.example.com. This will resolve virtual-host requests such as example-repo.s3.lakefs.example.com that lakeFS would now know how to parse. | . For more information on how to configure these, see the official documentation of your DNS provider. On AWS, This could also be done using ALIAS records for a load balancer. ", "url": "/howto/virtual-host-addressing.html#setting-up-the-appropriate-dns-records", "relUrl": "/howto/virtual-host-addressing.html#setting-up-the-appropriate-dns-records" - },"638": { + },"640": { "doc": "S3 Virtual-host addressing (advanced)", "title": "S3 Virtual-host addressing (advanced)", "content": " ", "url": "/howto/virtual-host-addressing.html", "relUrl": "/howto/virtual-host-addressing.html" - },"639": { + },"641": { "doc": "Webhooks", "title": "Webhooks", "content": " ", "url": "/howto/hooks/webhooks.html", "relUrl": "/howto/hooks/webhooks.html" - },"640": { + },"642": { "doc": "Webhooks", "title": "Table of contents", "content": ". | Action File Webhook properties | Request body schema | . A Webhook is a Hook type that sends an HTTP POST request to the configured URL. Any non 2XX response by the responding endpoint will fail the Hook, cancel the execution of the following Hooks under the same Action. For pre-* hooks, the triggering operation will also be aborted. Warning: You should not use pre-* webhooks for long-running tasks, since they block the performed operation. Moreover, the branch is locked during the execution of pre-* hooks, so the webhook server cannot perform any write operations on the branch (like uploading or commits). ", "url": "/howto/hooks/webhooks.html#table-of-contents", "relUrl": "/howto/hooks/webhooks.html#table-of-contents" - },"641": { + },"643": { "doc": "Webhooks", "title": "Action File Webhook properties", "content": "See the Action configuration for overall configuration schema and details. | Property | Description | Data Type | Required | Default Value | Env Vars Support | . | url | The URL address of the request | String | true |   | no | . | timeout | Time to wait for response before failing the hook | String (golang’s Duration representation) | false | 1 minute | no | . | query_params | List of query params that will be added to the request | Dictionary(String:String or String:List(String) | false |   | yes | . | headers | Headers to add to the request | Dictionary(String:String) | false |   | yes | . Secrets & Environment Variables lakeFS Actions supports secrets by using environment variables. The format {{ ENV.SOME_ENV_VAR }} will be replaced with the value of $SOME_ENV_VAR during the execution of the action. If that environment variable doesn’t exist in the lakeFS server environment, the action run will fail. All environment variables need to begin with “LAKEFSACTIONS_”. Otherwise, they will be blocked. Additionally, the actions.env.enabled configuration parameter can be set to false to block access to all environment variables. Example: ... hooks: - id: prevent_user_columns type: webhook description: Ensure no user_* columns under public/ properties: url: \"http://<host:port>/webhooks/schema\" timeout: 1m30s query_params: disallow: [\"user_\", \"private_\"] prefix: public/ headers: secret_header: \"{{ ENV.MY_SECRET }}\" ... ", "url": "/howto/hooks/webhooks.html#action-file-webhook-properties", "relUrl": "/howto/hooks/webhooks.html#action-file-webhook-properties" - },"642": { + },"644": { "doc": "Webhooks", "title": "Request body schema", "content": "Upon execution, a webhook will send a request containing a JSON object with the following fields: . | Field | Description | Type | . | event_type | Type of the event that triggered the Action | string | . | event_time | Time of the event that triggered the Action (RFC3339 formatted) | string | . | action_name | Containing Hook Action’s Name | string | . | hook_id | ID of the Hook | string | . | repository_id | ID of the Repository | string | . | branch_id1 | ID of the Branch | string | . | source_ref | Reference to the source on which the event was triggered | string | . | commit_message2 | The message for the commit (or merge) that is taking place | string | . | committer2 | Name of the committer | string | . | commit_metadata2 | The metadata for the commit that is taking place | string | . | commit_id[^2,^4] | The ID of the commit that is being created | string | . | tag_id3 | The ID of the created/deleted tag | string | . Example: . { \"event_type\": \"pre-merge\", \"event_time\": \"2021-02-28T14:03:31Z\", \"action_name\": \"test action\", \"hook_id\": \"prevent_user_columns\", \"repository_id\": \"repo1\", \"branch_id\": \"feature-1\", \"source_ref\": \"feature-1\", \"commit_message\": \"merge commit message\", \"commit_id\": \"5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03\", \"committer\": \"committer\", \"commit_metadata\": { \"key\": \"value\" } } . | N\\A for Tag events ↩ . | N\\A for Tag and Create/Delete Branch events ↩ ↩2 ↩3 . | Applicable only for Tag events ↩ . | . ", "url": "/howto/hooks/webhooks.html#request-body-schema", "relUrl": "/howto/hooks/webhooks.html#request-body-schema" - },"643": { + },"645": { "doc": "7️⃣ Work with lakeFS data locally", "title": "Work with lakeFS Data Locally", "content": "When working with lakeFS, there are scenarios where we need to access and manipulate data locally. An example use case for working locally is machine learning model development. Machine learning model development is dynamic and iterative. To optimize this process, experiments need to be conducted with speed, tracking ease, and reproducibility. Localizing model data during development accelerates the process by enabling interactive and offline development and reducing data access latency. lakeFS provides 2 ways to expose versioned data locally . ", "url": "/quickstart/work-with-data-locally.html#work-with-lakefs-data-locally", "relUrl": "/quickstart/work-with-data-locally.html#work-with-lakefs-data-locally" - },"644": { + },"646": { "doc": "7️⃣ Work with lakeFS data locally", "title": "Table of contents", "content": ". | lakeFS Mount | lakectl local | . ", "url": "/quickstart/work-with-data-locally.html#table-of-contents", "relUrl": "/quickstart/work-with-data-locally.html#table-of-contents" - },"645": { + },"647": { "doc": "7️⃣ Work with lakeFS data locally", "title": "lakeFS Mount", "content": "lakeFS Mount is available (in preview) for lakeFS Enterprise and lakeFS Cloud customers. You can try it out by signing up for the preview . Getting started with lakeFS Mount . Prerequisites: . | A working lakeFS Server running either lakeFS Enterprise or lakeFS Cloud | You’ve installed the lakectl command line utility: this is the official lakeFS command line interface, on top of which lakeFS Mount is built. | lakectl is configured properly to access your lakeFS server as detailed in the configuration instructions | . Mounting a path to a local directory: . | In lakeFS create a new branch called my-experiment. You can do this through the UI or with lakectl: . docker exec lakefs \\ lakectl branch create \\ lakefs://quickstart/my-experiment \\ --source lakefs://quickstart/main . | Mount images from your quickstart repository into a local directory named my_local_dir . everest mount lakefs://quickstart/my-experiment/images my_local_dir . | . Once complete, my_local_dir should be mounted with the specified path. | Verify that my_local_dir is linked to the correct path in your lakeFS remote: | . ls -l my_local_dir . | To unmount the directory, simply run: | . everest umount ./my_local_dir . Which will unmount the path and terminate the local mount-server. ", "url": "/quickstart/work-with-data-locally.html#lakefs-mount", "relUrl": "/quickstart/work-with-data-locally.html#lakefs-mount" - },"646": { + },"648": { "doc": "7️⃣ Work with lakeFS data locally", "title": "lakectl local", "content": "Alternatively, we can use lakectl local to bring a subset of our lakeFS data to a local directory within the lakeFS container and edit an image dataset used for ML model development. Unlike lakeFS Mount, using lakectl local requires copying data to/from lakeFS and your local machine. Reference Guide: lakeFS lakectl local for machine learning . Cloning a Subset of lakeFS Data into a Local Directory . | In lakeFS create a new branch called my-experiment. You can do this through the UI or with lakectl: . docker exec lakefs lakectl branch create lakefs://quickstart/my-experiment --source lakefs://quickstart/main . | Clone images from your quickstart repository into a local directory named my_local_dir within your container: . docker exec lakefs lakectl local clone lakefs://quickstart/my-experiment/images my_local_dir . | Verify that my_local_dir is linked to the correct path in your lakeFS remote: . docker exec lakefs lakectl local list . You should see confirmation that my_local_dir is tracking the desired lakeFS path.: . my_local_dir lakefs://quickstart/my-experiment/images/8614575b5488b47a094163bd17a12ed0b82e0bcbfd22ed1856151c671f1faa53 . | Verify that your local environment is up-to-date with its remote path: . docker exec lakefs lakectl local status my_local_dir . You should get a confirmation message like this showing that there is no difference between your local environment and the lakeFS remote: . diff 'local:///home/lakefs/my_local_dir' <--> 'lakefs://quickstart/8614575b5488b47a094163bd17a12ed0b82e0bcbfd22ed1856151c671f1faa53/images/'... diff 'lakefs://quickstart/8614575b5488b47a094163bd17a12ed0b82e0bcbfd22ed1856151c671f1faa53/images/' <--> 'lakefs://quickstart/my-experiment/images/'... No diff found. | . Making Changes to Data Locally . | Download a new image of an Axolotl and add it to the dataset cloned into my_local_dir: . curl -L https://go.lakefs.io/43ENDyS > axolotl.png docker cp axolotl.png lakefs:/home/lakefs/my_local_dir . | Clean the dataset by removing images larger than 225 KB: docker exec lakefs find my_local_dir -type f -size +225k -delete . | Check the status of your local changes compared to the lakeFS remote path: docker exec lakefs lakectl local status my_local_dir . You should get a confirmation message like this, showing the modifications you made locally: . diff 'local:///home/lakefs/my_local_dir' <--> 'lakefs://quickstart/8614575b5488b47a094163bd17a12ed0b82e0bcbfd22ed1856151c671f1faa53/images/'... diff 'lakefs://quickstart/8614575b5488b47a094163bd17a12ed0b82e0bcbfd22ed1856151c671f1faa53/images/' <--> 'lakefs://quickstart/my-experiment/images/'... ╔════════╦══════════╦═════════════════════╗ ║ SOURCE ║ CHANGE ║ PATH ║ ╠════════╬══════════╬═════════════════════╣ ║ local ║ modified ║ axolotl.png ║ ║ local ║ removed ║ duckdb-main-02.png ║ ║ local ║ removed ║ empty-repo-list.png ║ ║ local ║ removed ║ repo-contents.png ║ ╚════════╩══════════╩═════════════════════╝ . | . Pushing Local Changes to lakeFS . Once we are done with editing the image dataset in our local environment, we will push our changes to the lakeFS remote so that the improved dataset is shared and versioned. | Commit your local changes to lakeFS: . docker exec lakefs lakectl local commit -m 'Deleted images larger than 225KB in size and changed the Axolotl image' my_local_dir . In your branch, you should see the commit including your local changes: . | Compare my-experiment branch to the main branch to visualize your changes: . | . ", "url": "/quickstart/work-with-data-locally.html#lakectl-local", "relUrl": "/quickstart/work-with-data-locally.html#lakectl-local" - },"647": { + },"649": { "doc": "7️⃣ Work with lakeFS data locally", "title": "Bonus Challenge", "content": "And so with that, this quickstart for lakeFS draws to a close. If you’re simply having too much fun to stop then here’s an exercise for you. Implement the requirement from the beginning of this quickstart correctly, such that you write denmark-lakes.parquet in the respective branch and successfully merge it back into main. Look up how to list the contents of the main branch and verify that it looks like this: . object 2023-03-21 17:33:51 +0000 UTC 20.9 kB denmark-lakes.parquet object 2023-03-21 14:45:38 +0000 UTC 916.4 kB lakes.parquet . ", "url": "/quickstart/work-with-data-locally.html#bonus-challenge", "relUrl": "/quickstart/work-with-data-locally.html#bonus-challenge" - },"648": { + },"650": { "doc": "7️⃣ Work with lakeFS data locally", "title": "Finishing Up", "content": "Once you’ve finished the quickstart, shut down your local environment with the following command: . docker stop lakefs . ", "url": "/quickstart/work-with-data-locally.html#finishing-up", "relUrl": "/quickstart/work-with-data-locally.html#finishing-up" - },"649": { + },"651": { "doc": "7️⃣ Work with lakeFS data locally", "title": "7️⃣ Work with lakeFS data locally", "content": " ", diff --git a/cloud/index.html b/cloud/index.html index ab7f334ea..6f98d29ec 100644 --- a/cloud/index.html +++ b/cloud/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/architecture.html b/enterprise/architecture.html index 36859ddfb..7116c1c8f 100644 --- a/enterprise/architecture.html +++ b/enterprise/architecture.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/configuration.html b/enterprise/configuration.html index 96ab8af9f..4c42b7531 100644 --- a/enterprise/configuration.html +++ b/enterprise/configuration.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/getstarted/index.html b/enterprise/getstarted/index.html index 8c55053a3..12155f581 100644 --- a/enterprise/getstarted/index.html +++ b/enterprise/getstarted/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/getstarted/install.html b/enterprise/getstarted/install.html index fa3c75d94..c523a31f1 100644 --- a/enterprise/getstarted/install.html +++ b/enterprise/getstarted/install.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/getstarted/migrate-from-oss.html b/enterprise/getstarted/migrate-from-oss.html index 1f995788a..873ee2fc3 100644 --- a/enterprise/getstarted/migrate-from-oss.html +++ b/enterprise/getstarted/migrate-from-oss.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/getstarted/quickstart.html b/enterprise/getstarted/quickstart.html index b022329f0..59ab1bca6 100644 --- a/enterprise/getstarted/quickstart.html +++ b/enterprise/getstarted/quickstart.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/index.html b/enterprise/index.html index 3e17a824e..400b41a7b 100644 --- a/enterprise/index.html +++ b/enterprise/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/troubleshooting.html b/enterprise/troubleshooting.html index 523d4afef..4c09152a3 100644 --- a/enterprise/troubleshooting.html +++ b/enterprise/troubleshooting.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/enterprise/upgrade.html b/enterprise/upgrade.html index 4a3956fdc..2e098c8f0 100644 --- a/enterprise/upgrade.html +++ b/enterprise/upgrade.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/experimental/lakectl_local_posix.html b/experimental/lakectl_local_posix.html index 38334f7ec..410d55433 100644 --- a/experimental/lakectl_local_posix.html +++ b/experimental/lakectl_local_posix.html @@ -302,7 +302,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/backup-and-restore.html b/howto/backup-and-restore.html index dcb3388d7..2f396eb7e 100644 --- a/howto/backup-and-restore.html +++ b/howto/backup-and-restore.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/catalog_exports.html b/howto/catalog_exports.html index 1e024a5d0..c18d706b7 100644 --- a/howto/catalog_exports.html +++ b/howto/catalog_exports.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/copying.html b/howto/copying.html index 35d6faec6..74d55994d 100644 --- a/howto/copying.html +++ b/howto/copying.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/aws.html b/howto/deploy/aws.html index 10653558c..9cce4857c 100644 --- a/howto/deploy/aws.html +++ b/howto/deploy/aws.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/azure.html b/howto/deploy/azure.html index 162e1f905..ddcd46e08 100644 --- a/howto/deploy/azure.html +++ b/howto/deploy/azure.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/gcp.html b/howto/deploy/gcp.html index e565d73f9..b9d292966 100644 --- a/howto/deploy/gcp.html +++ b/howto/deploy/gcp.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/index.html b/howto/deploy/index.html index 6dfc3912e..e27494523 100644 --- a/howto/deploy/index.html +++ b/howto/deploy/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/onprem.html b/howto/deploy/onprem.html index f1f6beb50..4147b89cc 100644 --- a/howto/deploy/onprem.html +++ b/howto/deploy/onprem.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/deploy/upgrade.html b/howto/deploy/upgrade.html index 3ff84eeba..82a4973a6 100644 --- a/howto/deploy/upgrade.html +++ b/howto/deploy/upgrade.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/export.html b/howto/export.html index f4ce6a549..33ce54527 100644 --- a/howto/export.html +++ b/howto/export.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/garbage-collection/gc.html b/howto/garbage-collection/gc.html index a21b46882..38e1d3e6b 100644 --- a/howto/garbage-collection/gc.html +++ b/howto/garbage-collection/gc.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/garbage-collection/managed-gc.html b/howto/garbage-collection/managed-gc.html index 7e2cab54a..5081f1bfd 100644 --- a/howto/garbage-collection/managed-gc.html +++ b/howto/garbage-collection/managed-gc.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/garbage-collection/standalone-gc.html b/howto/garbage-collection/standalone-gc.html index 2775ea01d..12684611c 100644 --- a/howto/garbage-collection/standalone-gc.html +++ b/howto/garbage-collection/standalone-gc.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/hooks/airflow.html b/howto/hooks/airflow.html index 3bbc2bb0d..7594faf71 100644 --- a/howto/hooks/airflow.html +++ b/howto/hooks/airflow.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/hooks/index.html b/howto/hooks/index.html index a483c1a53..637796428 100644 --- a/howto/hooks/index.html +++ b/howto/hooks/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/hooks/lua.html b/howto/hooks/lua.html index 598994b56..28bd4dd1f 100644 --- a/howto/hooks/lua.html +++ b/howto/hooks/lua.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/hooks/webhooks.html b/howto/hooks/webhooks.html index 937951929..407d10a97 100644 --- a/howto/hooks/webhooks.html +++ b/howto/hooks/webhooks.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/import.html b/howto/import.html index b49e8abb8..d47e62d96 100644 --- a/howto/import.html +++ b/howto/import.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/index.html b/howto/index.html index 81b1277be..d1468d0ed 100644 --- a/howto/index.html +++ b/howto/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/local-checkouts.html b/howto/local-checkouts.html index d7295801d..3414e2022 100644 --- a/howto/local-checkouts.html +++ b/howto/local-checkouts.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/migrate-away.html b/howto/migrate-away.html index 129855cef..1138261b2 100644 --- a/howto/migrate-away.html +++ b/howto/migrate-away.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/mirroring.html b/howto/mirroring.html index aded4b56f..be2d75f6d 100644 --- a/howto/mirroring.html +++ b/howto/mirroring.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/private-link.html b/howto/private-link.html index 5219bc53a..34419d9fb 100644 --- a/howto/private-link.html +++ b/howto/private-link.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/protect-branches.html b/howto/protect-branches.html index 21c46c4a9..3bb716e85 100644 --- a/howto/protect-branches.html +++ b/howto/protect-branches.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/pull-requests.html b/howto/pull-requests.html index c798d7d01..3137c9037 100644 --- a/howto/pull-requests.html +++ b/howto/pull-requests.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/scim.html b/howto/scim.html index ad87050fa..e60637da2 100644 --- a/howto/scim.html +++ b/howto/scim.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/sizing-guide.html b/howto/sizing-guide.html index 4fff42957..688e4f19b 100644 --- a/howto/sizing-guide.html +++ b/howto/sizing-guide.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/unity-delta-sharing.html b/howto/unity-delta-sharing.html index 312fffef2..e6cbab215 100644 --- a/howto/unity-delta-sharing.html +++ b/howto/unity-delta-sharing.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/howto/virtual-host-addressing.html b/howto/virtual-host-addressing.html index ebb6b8e9d..231c3962f 100644 --- a/howto/virtual-host-addressing.html +++ b/howto/virtual-host-addressing.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/index.html b/index.html index ce4955b52..a60fd749d 100644 --- a/index.html +++ b/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/airbyte.html b/integrations/airbyte.html index 8171773e1..45bb6c233 100644 --- a/integrations/airbyte.html +++ b/integrations/airbyte.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/airflow.html b/integrations/airflow.html index 303b6c1f5..0ea84512b 100644 --- a/integrations/airflow.html +++ b/integrations/airflow.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/athena.html b/integrations/athena.html index b0b8e28c0..553c2a7bd 100644 --- a/integrations/athena.html +++ b/integrations/athena.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/aws_cli.html b/integrations/aws_cli.html index e37e9ad35..aaa0b9ea5 100644 --- a/integrations/aws_cli.html +++ b/integrations/aws_cli.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/cloudera.html b/integrations/cloudera.html index 765baf29c..4f2378747 100644 --- a/integrations/cloudera.html +++ b/integrations/cloudera.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/databricks.html b/integrations/databricks.html index 7752d2336..dfed4b6b8 100644 --- a/integrations/databricks.html +++ b/integrations/databricks.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/delta.html b/integrations/delta.html index f33ec296c..5aa22ef4e 100644 --- a/integrations/delta.html +++ b/integrations/delta.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/dremio.html b/integrations/dremio.html index 8cdaec8ba..66e9cca22 100644 --- a/integrations/dremio.html +++ b/integrations/dremio.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/duckdb.html b/integrations/duckdb.html index da6786ac7..2c5592b40 100644 --- a/integrations/duckdb.html +++ b/integrations/duckdb.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/git.html b/integrations/git.html index 65a381268..ff162eeae 100644 --- a/integrations/git.html +++ b/integrations/git.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/glue_hive_metastore.html b/integrations/glue_hive_metastore.html index 1b113c80d..6ca79877b 100644 --- a/integrations/glue_hive_metastore.html +++ b/integrations/glue_hive_metastore.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/glue_metastore.html b/integrations/glue_metastore.html index 2de784f66..7ae4683b4 100644 --- a/integrations/glue_metastore.html +++ b/integrations/glue_metastore.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/hive.html b/integrations/hive.html index 61ad48e1d..2fb315bf1 100644 --- a/integrations/hive.html +++ b/integrations/hive.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/huggingface_datasets.html b/integrations/huggingface_datasets.html index 0c5aa7f25..c4c9e8ae9 100644 --- a/integrations/huggingface_datasets.html +++ b/integrations/huggingface_datasets.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/iceberg.html b/integrations/iceberg.html index 18979f1fa..3a57f65e3 100644 --- a/integrations/iceberg.html +++ b/integrations/iceberg.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/index.html b/integrations/index.html index 4c7e1bd3f..39ebcec61 100644 --- a/integrations/index.html +++ b/integrations/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • @@ -570,25 +571,25 @@

    cli logo
    AWS CLI     cloudera logo
    Cloudera     + Databricks Logo
    Databricks     delta lake logo
    Delta Lake     - git logo
    Git     dremio logo
    Dremio     duckdb logo
    DuckDB     + git logo
    Git     glue logo
    Glue / Hive metastore     - kubeflow logo
    Kubeflow     + Hugging Face Logo
    HuggingFace Datasets     + kubeflow logo
    Kubeflow     presto and trino logos
    Presto / Trino     python logo
    Python     - r logo
    R     - Vertex AI Logo
    Vertex AI     - Hugging Face Logo
    HuggingFace Datasets     - Databricks Logo
    Databricks     - + r logo
    R     + Red Hat OpenShift AI Logo
    Red Hat OpenShift AI     + Vertex AI Logo
    Vertex AI     diff --git a/integrations/kafka.html b/integrations/kafka.html index e6c6cb8c7..ed447e803 100644 --- a/integrations/kafka.html +++ b/integrations/kafka.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow

  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/kubeflow.html b/integrations/kubeflow.html index 24d53f78d..af85c3142 100644 --- a/integrations/kubeflow.html +++ b/integrations/kubeflow.html @@ -311,7 +311,8 @@ class="nav-list-link active">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/presto_trino.html b/integrations/presto_trino.html index dde5a4295..216af7c36 100644 --- a/integrations/presto_trino.html +++ b/integrations/presto_trino.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/python.html b/integrations/python.html index 16dfe9946..6f6e4c574 100644 --- a/integrations/python.html +++ b/integrations/python.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/r.html b/integrations/r.html index 536fb35b5..d653dd705 100644 --- a/integrations/r.html +++ b/integrations/r.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/red_hat_openshift_ai.html b/integrations/red_hat_openshift_ai.html new file mode 100644 index 000000000..da1aa00d8 --- /dev/null +++ b/integrations/red_hat_openshift_ai.html @@ -0,0 +1,736 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Red Hat OpenShift AI | lakeFS Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Link + + + + + + + Search + + + + + + + Menu + + + + + + + + + + + + Expand + + + + + + Document + + + + + + +
    +
    +
    + + + + + + + + + +
    + + +
    +
    + +
    + + + +
    +
    + + + + + +
    + +

    + + + Using lakeFS with Red Hat OpenShift AI + + +

    + +

    Red Hat® OpenShift® is an enterprise-ready Kubernetes container platform with full-stack automated operations to manage hybrid cloud, multi-cloud, and edge deployments. OpenShift includes an enterprise-grade Linux operating system, container runtime, networking, monitoring, registry, and authentication and authorization solutions.

    + +

    Red Hat® OpenShift® AI is a flexible, scalable artificial intelligence (AI) and machine learning (ML) platform that enables enterprises to create and deliver AI-enabled applications at scale across hybrid cloud environments. Built using open source technologies, OpenShift AI provides trusted, operationally consistent capabilities for teams to experiment, serve models, and deliver innovative apps.

    + +

    OpenShift AI and lakeFS can be deployed in OpenShift cluster in 3 different architectures:

    +
      +
    1. OpenShift AI, lakeFS and object storage are delpoyed in OpenShift cluster
      2. +
    2. OpenShift AI and lakeFS are deployed in OpenShift cluster while object storage is external
      3. +
    3. OpenShift AI is deployed in OpenShift cluster while lakeFS and object storage are external
      4. +
    + +

    OpenShift AI and lakeFS Deployment Architecture

    + +

    Refer to an example in lakeFS-samples to deploy lakeFS, MinIO and OpenShift AI tutorial (Fraud Detection demo) in OpenShift cluster. Fraud detection demo is a step-by-step guide for using OpenShift AI to train an example model in JupyterLab, deploy the model, and refine the model by using automated pipelines.

    + +

    In this example, OpenShift AI is configured to connect over S3 interace to lakeFS, which will version the data in a backend MinIO instance. This is the architecture to run Fraud Detection demo with or without lakeFS: +OpenShift AI and lakeFS Deployment Architecture for the demo

    + +

    lakeFS-samples also includes multiple Helm chart examples to deploy lakeFS and MinIO in different scenarios:

    +
      +
    1. lakefs-local.yaml: Bring up lakeFS using local object storage. This would be useful for a quick demo where MinIO is not included.
      2. +
    2. lakefs-minio.yaml: Bring up lakeFS configured to use MinIO as backend object storage. This will be used in the lakeFS demo.
      3. +
    3. minio-direct.yaml: This file would only be used if lakeFS is not in the picture and OpenShift AI will communicate directly with MinIO. It will bring up MinIO as it is in the default Fraud Detection demo, complete with configuring MinIO storage buckets and the OpenShift AI data connections. It may serve useful in debugging an issue.
      4. +
    4. minio-via-lakefs.yaml: Bring up MinIO for the modified Fraud Detection demo that includes lakeFS, complete with configuring MinIO storage buckets, but do NOT configure the OpenShift AI data connections. This will be used in the lakeFS demo.
      5. +
    + + + + + + + + +
    +
    + + + + +
    + +
    +
    + + + + + + + + diff --git a/integrations/sagemaker.html b/integrations/sagemaker.html index 7175de060..ca306fd96 100644 --- a/integrations/sagemaker.html +++ b/integrations/sagemaker.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/spark.html b/integrations/spark.html index 090e09b1c..1e97395a4 100644 --- a/integrations/spark.html +++ b/integrations/spark.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/unity-catalog.html b/integrations/unity-catalog.html index 7b3061e0a..cb11f98f8 100644 --- a/integrations/unity-catalog.html +++ b/integrations/unity-catalog.html @@ -10,10 +10,10 @@
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/integrations/vertex_ai.html b/integrations/vertex_ai.html index 3a46d5424..c751238a1 100644 --- a/integrations/vertex_ai.html +++ b/integrations/vertex_ai.html @@ -10,10 +10,10 @@
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/posts/deprecate-py-legacy.html b/posts/deprecate-py-legacy.html index e32dffa0f..3d58d7421 100644 --- a/posts/deprecate-py-legacy.html +++ b/posts/deprecate-py-legacy.html @@ -303,7 +303,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/posts/index.html b/posts/index.html index de8986ca8..84436b641 100644 --- a/posts/index.html +++ b/posts/index.html @@ -302,7 +302,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • @@ -1516,6 +1517,10 @@

    Table of contents

    R
    • +
  • + Red Hat OpenShift AI +
    • +
  • Remote Authenticator
    • diff --git a/posts/security_update.html b/posts/security_update.html index 27f91cd00..50daf3f70 100644 --- a/posts/security_update.html +++ b/posts/security_update.html @@ -303,7 +303,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/project/code-migrate-1.0-sdk.html b/project/code-migrate-1.0-sdk.html index 3f0314188..f9fe4d8c3 100644 --- a/project/code-migrate-1.0-sdk.html +++ b/project/code-migrate-1.0-sdk.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/project/contributing.html b/project/contributing.html index ca02de34f..098206850 100644 --- a/project/contributing.html +++ b/project/contributing.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/project/docs/callouts.html b/project/docs/callouts.html index d1723f893..b78422609 100644 --- a/project/docs/callouts.html +++ b/project/docs/callouts.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/project/docs/index.html b/project/docs/index.html index ba6cfb593..c77ac4595 100644 --- a/project/docs/index.html +++ b/project/docs/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/project/index.html b/project/index.html index 0190dcad0..82df65d4c 100644 --- a/project/index.html +++ b/project/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/actions-and-hooks.html b/quickstart/actions-and-hooks.html index 6033b683e..04fd7e0c6 100644 --- a/quickstart/actions-and-hooks.html +++ b/quickstart/actions-and-hooks.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/branch.html b/quickstart/branch.html index 592fba280..52f10f17d 100644 --- a/quickstart/branch.html +++ b/quickstart/branch.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/commit-and-merge.html b/quickstart/commit-and-merge.html index e530ecea0..29d4da161 100644 --- a/quickstart/commit-and-merge.html +++ b/quickstart/commit-and-merge.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/index.html b/quickstart/index.html index df97a01e6..020f285df 100644 --- a/quickstart/index.html +++ b/quickstart/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/launch.html b/quickstart/launch.html index 2ec2fdb44..e9162ff5a 100644 --- a/quickstart/launch.html +++ b/quickstart/launch.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/learning-more-lakefs.html b/quickstart/learning-more-lakefs.html index ee596bc2a..40187951f 100644 --- a/quickstart/learning-more-lakefs.html +++ b/quickstart/learning-more-lakefs.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/query.html b/quickstart/query.html index 7d118ee99..8abbb1700 100644 --- a/quickstart/query.html +++ b/quickstart/query.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/rollback.html b/quickstart/rollback.html index 5b190fc99..6c0b4aed8 100644 --- a/quickstart/rollback.html +++ b/quickstart/rollback.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/quickstart/work-with-data-locally.html b/quickstart/work-with-data-locally.html index a451192f1..97a7fe3bb 100644 --- a/quickstart/work-with-data-locally.html +++ b/quickstart/work-with-data-locally.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/api.html b/reference/api.html index cfb198f47..bda1a7510 100644 --- a/reference/api.html +++ b/reference/api.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/auditing.html b/reference/auditing.html index 104325c7e..798159931 100644 --- a/reference/auditing.html +++ b/reference/auditing.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/cli.html b/reference/cli.html index 9e031c44a..eb8ae1f7b 100644 --- a/reference/cli.html +++ b/reference/cli.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/configuration.html b/reference/configuration.html index f228c2bb5..d480c1b2a 100644 --- a/reference/configuration.html +++ b/reference/configuration.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/index.html b/reference/index.html index 427fee4f2..f32e558f4 100644 --- a/reference/index.html +++ b/reference/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/monitor.html b/reference/monitor.html index 08ab96688..cd24bae6e 100644 --- a/reference/monitor.html +++ b/reference/monitor.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/mount-csi-driver.html b/reference/mount-csi-driver.html index 51196dc2f..1da2a3c32 100644 --- a/reference/mount-csi-driver.html +++ b/reference/mount-csi-driver.html @@ -302,7 +302,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/mount.html b/reference/mount.html index 968c40da9..77910d6ca 100644 --- a/reference/mount.html +++ b/reference/mount.html @@ -302,7 +302,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/s3.html b/reference/s3.html index 62dbbe490..f3f2aca4e 100644 --- a/reference/s3.html +++ b/reference/s3.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/reference/spark-client.html b/reference/spark-client.html index c9c63a936..f3797d1f1 100644 --- a/reference/spark-client.html +++ b/reference/spark-client.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/access-control-lists.html b/security/access-control-lists.html index 085d04f8a..547353bb4 100644 --- a/security/access-control-lists.html +++ b/security/access-control-lists.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/authentication.html b/security/authentication.html index b7869515c..bbc91cd62 100644 --- a/security/authentication.html +++ b/security/authentication.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/external-principals-aws.html b/security/external-principals-aws.html index 02f311ab7..9ac443415 100644 --- a/security/external-principals-aws.html +++ b/security/external-principals-aws.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/index.html b/security/index.html index 314c0e0e1..ec4f9a916 100644 --- a/security/index.html +++ b/security/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/presigned-url.html b/security/presigned-url.html index 44d7bc6ed..9d7106fb8 100644 --- a/security/presigned-url.html +++ b/security/presigned-url.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/rbac.html b/security/rbac.html index c6f2e6344..09ccf95f4 100644 --- a/security/rbac.html +++ b/security/rbac.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/remote-authenticator.html b/security/remote-authenticator.html index 8cb1ac348..0edae66f8 100644 --- a/security/remote-authenticator.html +++ b/security/remote-authenticator.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/sso.html b/security/sso.html index e72c8dbc2..863fc0b0b 100644 --- a/security/sso.html +++ b/security/sso.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/security/sts-login.html b/security/sts-login.html index 192e92fa9..150856310 100644 --- a/security/sts-login.html +++ b/security/sts-login.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/sitemap.xml b/sitemap.xml index 504e00785..cd932185c 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -298,6 +298,9 @@ https://docs.lakefs.io/security/rbac.html +https://docs.lakefs.io/integrations/red_hat_openshift_ai.html + + https://docs.lakefs.io/security/remote-authenticator.html diff --git a/understand/architecture.html b/understand/architecture.html index 976d3d778..1d8b77be5 100644 --- a/understand/architecture.html +++ b/understand/architecture.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/data-structure.html b/understand/data-structure.html index 67a49f644..79829057c 100644 --- a/understand/data-structure.html +++ b/understand/data-structure.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/data_lifecycle_management/ci.html b/understand/data_lifecycle_management/ci.html index 922a1e2ca..f5b586f0b 100644 --- a/understand/data_lifecycle_management/ci.html +++ b/understand/data_lifecycle_management/ci.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/data_lifecycle_management/data-devenv.html b/understand/data_lifecycle_management/data-devenv.html index 907995221..53bf4e15b 100644 --- a/understand/data_lifecycle_management/data-devenv.html +++ b/understand/data_lifecycle_management/data-devenv.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/data_lifecycle_management/index.html b/understand/data_lifecycle_management/index.html index a3ac52b3d..c3db8a551 100644 --- a/understand/data_lifecycle_management/index.html +++ b/understand/data_lifecycle_management/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/data_lifecycle_management/production.html b/understand/data_lifecycle_management/production.html index 7705dff91..d60d51e80 100644 --- a/understand/data_lifecycle_management/production.html +++ b/understand/data_lifecycle_management/production.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/faq.html b/understand/faq.html index 88d1c26fe..95051382d 100644 --- a/understand/faq.html +++ b/understand/faq.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/glossary.html b/understand/glossary.html index 1a362ce9b..c7e511320 100644 --- a/understand/glossary.html +++ b/understand/glossary.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/how/index.html b/understand/how/index.html index e70100560..040b312c0 100644 --- a/understand/how/index.html +++ b/understand/how/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/how/kv.html b/understand/how/kv.html index c7f810b2d..bcbc34996 100644 --- a/understand/how/kv.html +++ b/understand/how/kv.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/how/merge.html b/understand/how/merge.html index 607bd604e..5d0de9f7c 100644 --- a/understand/how/merge.html +++ b/understand/how/merge.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/how/versioning-internals.html b/understand/how/versioning-internals.html index 51e31c518..1e3166e6c 100644 --- a/understand/how/versioning-internals.html +++ b/understand/how/versioning-internals.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/index.html b/understand/index.html index 04620b62b..f1f14d990 100644 --- a/understand/index.html +++ b/understand/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/model.html b/understand/model.html index 3b4344934..c5d4ddec3 100644 --- a/understand/model.html +++ b/understand/model.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/performance-best-practices.html b/understand/performance-best-practices.html index bf5b22a11..7b7040346 100644 --- a/understand/performance-best-practices.html +++ b/understand/performance-best-practices.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/use_cases/cicd_for_data.html b/understand/use_cases/cicd_for_data.html index 0d4bab4e6..0ab1f9ba8 100644 --- a/understand/use_cases/cicd_for_data.html +++ b/understand/use_cases/cicd_for_data.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/use_cases/etl_testing.html b/understand/use_cases/etl_testing.html index c308f21df..a5ce41222 100644 --- a/understand/use_cases/etl_testing.html +++ b/understand/use_cases/etl_testing.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/use_cases/index.html b/understand/use_cases/index.html index 897fa72f5..63efda19f 100644 --- a/understand/use_cases/index.html +++ b/understand/use_cases/index.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/use_cases/reproducibility.html b/understand/use_cases/reproducibility.html index df6e51943..bf47bb0a5 100644 --- a/understand/use_cases/reproducibility.html +++ b/understand/use_cases/reproducibility.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI
  • diff --git a/understand/use_cases/rollback.html b/understand/use_cases/rollback.html index 0a2a49abd..569295197 100644 --- a/understand/use_cases/rollback.html +++ b/understand/use_cases/rollback.html @@ -311,7 +311,8 @@ class="nav-list-link">Kubeflow
  • Presto / Trino
  • Python
  • R
  • R
  • Red Hat OpenShift AI
  • Unity Catalog
  • Vertex AI