Update from treeverse/lakeFS@9d179c7

assets/js/search-data.json

@@ -1912,7 +1912,7 @@
   },"273": {
     "doc": "Import Data",
     "title": "Fine-grained permissions",
-    "content": "lakeFS Cloud . lakeFS Enterprise . With RBAC support, The lakeFS user running the import command should have the following permissions in lakeFS: fs:WriteObject, fs:CreateMetaRange, fs:CreateCommit, fs:ImportFromStorage and fs:ImportCancel. As mentioned above, all of these permissions are available by default to the Supers (open-source) group or the SuperUsers (Cloud/Enterprise). ",
+    "content": "lakeFS Cloud . lakeFS Enterprise . With RBAC support, The lakeFS user running the import command should have the following permissions in lakeFS: fs:WriteObject, fs:CreateCommit, fs:ImportFromStorage and fs:ImportCancel. As mentioned above, all of these permissions are available by default to the Supers (open-source) group or the SuperUsers (Cloud/Enterprise). ",
     "url": "/howto/import.html#fine-grained-permissions",
 
     "relUrl": "/howto/import.html#fine-grained-permissions"
@@ -2976,7 +2976,7 @@
   },"425": {
     "doc": "Mirroring",
     "title": "Setting up mirroring",
-    "content": "Configuring bucket replication on S3 . The objects within the repository are copied using your cloud provider’s object store replication mechanism. For AWS S3, please refer to the AWS S3 replication documentation to make sure your lakeFS repository’s storage namespace (source) is replicated to the region you’d like your mirror to be located on (target). After setting the replication rule, new objects will be replicated to the destination bucket. In order to replicate the existing objects, we’d need to manually copy them - however, we can use S3 batch jobs to do this. Creating a lakeFS user with a “replicator” policy . On our source lakeFS installation, under Administration create a new user that will be used by the replication subsystem. The user should have the following RBAC policy attached: . { \"id\": \"ReplicationPolicy\", \"statement\": [ { \"action\": [ \"fs:ReadRepository\", \"fs:CreateRepository\", \"fs:UpdateRepository\", \"fs:DeleteRepository\", \"fs:ListRepositories\", \"fs:AttachStorageNamespace\", \"fs:ReadObject\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:ListObjects\", \"fs:CreateCommit\", \"fs:CreateMetaRange\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:CreateBranch\", \"fs:DeleteBranch\", \"fs:RevertBranch\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . Alternatively, we can create a policy with a narrower scope, only for a specific repository and/or mirror: . { \"id\": \"ReplicationPolicy\", \"statement\": [ { \"action\": [ \"fs:ListRepositories\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"fs:ReadRepository\", \"fs:ReadObject\", \"fs:ListObjects\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::repository/{sourceRepositoryId}\" }, { \"action\": [ \"fs:ReadRepository\", \"fs:CreateRepository\", \"fs:UpdateRepository\", \"fs:DeleteRepository\", \"fs:AttachStorageNamespace\", \"fs:ReadObject\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:ListObjects\", \"fs:CreateCommit\", \"fs:CreateMetaRange\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:CreateBranch\", \"fs:DeleteBranch\", \"fs:RevertBranch\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::repository/{mirrorId}\" }, { \"action\": [ \"fs:AttachStorageNamespace\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::namespace/{DestinationStorageNamespace}\" } ] } . Once a user has been created and the replication policy attached to it, create an access key and secret to be used by the mirroring process. Authorizing the lakeFS Mirror process to use the replication user . Please contact Treeverse customer success to connect the newly created user with the mirroring process . Configuring repository replication . Replication has a stand-alone HTTP API. In this example, we’ll use cURL, but feel free to use any HTTP client or library: . curl --location 'https://<ORGANIZATION_ID>.<SOURCE_REGION>.lakefscloud.io/service/replication/v1/repositories/<SOURCE_REPO>/mirrors' \\ --header 'Content-Type: application/json' \\ -u <ACCESS_KEY_ID>:<SECRET_ACCESS_KEY> \\ -X POST \\ --data '{ \"name\": \"<MIRROR_NAME>\", \"region\": \"<MIRROR_REGION>\", \"storage_namespace\": \"<MIRROR_STORAGE_NAMESPACE>\" }' . Using the following parameters: . | ORGANIZATION_ID - The ID as it appears in the URL of your lakeFS installation (e.g. https://my-org.us-east-1.lakefscloud.io/) | SOURCE_REGION - The region where our source repository is hosted | SOURCE_REPO - Name of the repository acting as our replication source. It should exist | ACCESS_KEY_ID & SECRET_ACCESS_KEY - Credentials for your lakeFS user (make sure you have the necessary RBAC permissions as listed below) | MIRROR_NAME - Name used for the read-only mirror to be created on the destination region | MIRROR_STORAGE_NAMESPACE - Location acting as the replication target for the storage namespace of our source repository | . Mirroring and Garbage Collection . Garbage collection won’t run on mirrored repositories. Deletions from garbage collection should be replicated from the source: . | Enable DELETED marker replication on the source bucket. | Create a lifecycle policy on the destination bucket to delete the objects with the DELETED marker. | . ",
+    "content": "Configuring bucket replication on S3 . The objects within the repository are copied using your cloud provider’s object store replication mechanism. For AWS S3, please refer to the AWS S3 replication documentation to make sure your lakeFS repository’s storage namespace (source) is replicated to the region you’d like your mirror to be located on (target). After setting the replication rule, new objects will be replicated to the destination bucket. In order to replicate the existing objects, we’d need to manually copy them - however, we can use S3 batch jobs to do this. Creating a lakeFS user with a “replicator” policy . On our source lakeFS installation, under Administration create a new user that will be used by the replication subsystem. The user should have the following RBAC policy attached: . { \"id\": \"ReplicationPolicy\", \"statement\": [ { \"action\": [ \"fs:ReadRepository\", \"fs:CreateRepository\", \"fs:UpdateRepository\", \"fs:DeleteRepository\", \"fs:ListRepositories\", \"fs:AttachStorageNamespace\", \"fs:ReadObject\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:ListObjects\", \"fs:CreateCommit\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:CreateBranch\", \"fs:DeleteBranch\", \"fs:RevertBranch\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . Alternatively, we can create a policy with a narrower scope, only for a specific repository and/or mirror: . { \"id\": \"ReplicationPolicy\", \"statement\": [ { \"action\": [ \"fs:ListRepositories\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"fs:ReadRepository\", \"fs:ReadObject\", \"fs:ListObjects\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::repository/{sourceRepositoryId}\" }, { \"action\": [ \"fs:ReadRepository\", \"fs:CreateRepository\", \"fs:UpdateRepository\", \"fs:DeleteRepository\", \"fs:AttachStorageNamespace\", \"fs:ReadObject\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:ListObjects\", \"fs:CreateCommit\", \"fs:ReadCommit\", \"fs:ListCommits\", \"fs:CreateBranch\", \"fs:DeleteBranch\", \"fs:RevertBranch\", \"fs:ReadBranch\", \"fs:ListBranches\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::repository/{mirrorId}\" }, { \"action\": [ \"fs:AttachStorageNamespace\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:fs:::namespace/{DestinationStorageNamespace}\" } ] } . Once a user has been created and the replication policy attached to it, create an access key and secret to be used by the mirroring process. Authorizing the lakeFS Mirror process to use the replication user . Please contact Treeverse customer success to connect the newly created user with the mirroring process . Configuring repository replication . Replication has a stand-alone HTTP API. In this example, we’ll use cURL, but feel free to use any HTTP client or library: . curl --location 'https://<ORGANIZATION_ID>.<SOURCE_REGION>.lakefscloud.io/service/replication/v1/repositories/<SOURCE_REPO>/mirrors' \\ --header 'Content-Type: application/json' \\ -u <ACCESS_KEY_ID>:<SECRET_ACCESS_KEY> \\ -X POST \\ --data '{ \"name\": \"<MIRROR_NAME>\", \"region\": \"<MIRROR_REGION>\", \"storage_namespace\": \"<MIRROR_STORAGE_NAMESPACE>\" }' . Using the following parameters: . | ORGANIZATION_ID - The ID as it appears in the URL of your lakeFS installation (e.g. https://my-org.us-east-1.lakefscloud.io/) | SOURCE_REGION - The region where our source repository is hosted | SOURCE_REPO - Name of the repository acting as our replication source. It should exist | ACCESS_KEY_ID & SECRET_ACCESS_KEY - Credentials for your lakeFS user (make sure you have the necessary RBAC permissions as listed below) | MIRROR_NAME - Name used for the read-only mirror to be created on the destination region | MIRROR_STORAGE_NAMESPACE - Location acting as the replication target for the storage namespace of our source repository | . Mirroring and Garbage Collection . Garbage collection won’t run on mirrored repositories. Deletions from garbage collection should be replicated from the source: . | Enable DELETED marker replication on the source bucket. | Create a lifecycle policy on the destination bucket to delete the objects with the DELETED marker. | . ",
     "url": "/howto/mirroring.html#setting-up-mirroring",
 
     "relUrl": "/howto/mirroring.html#setting-up-mirroring"
@@ -3732,7 +3732,7 @@
   },"533": {
     "doc": "Role-Based Access Control (RBAC)",
     "title": "Preconfigured Policies",
-    "content": "The following Policies are created during initial setup: . FSFullAccess . { \"statement\": [ { \"action\": [ \"fs:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . FSReadAll . { \"statement\": [ { \"action\": [ \"fs:List*\", \"fs:Read*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . FSReadWriteAll . { \"statement\": [ { \"action\": [ \"fs:Read*\", \"fs:List*\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:RevertBranch\", \"fs:CreateBranch\", \"fs:CreateTag\", \"fs:DeleteBranch\", \"fs:DeleteTag\", \"fs:CreateCommit\", \"fs:CreateMetaRange\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . AuthFullAccess . { \"statement\": [ { \"action\": [ \"auth:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . AuthManageOwnCredentials . { \"statement\": [ { \"action\": [ \"auth:CreateCredentials\", \"auth:DeleteCredentials\", \"auth:ListCredentials\", \"auth:ReadCredentials\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:auth:::user/${user}\" } ] } . RepoManagementFullAccess . { \"statement\": [ { \"action\": [ \"ci:*\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"retention:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . RepoManagementReadAll . { \"statement\": [ { \"action\": [ \"ci:Read*\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"retention:Get*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . ",
+    "content": "The following Policies are created during initial setup: . FSFullAccess . { \"statement\": [ { \"action\": [ \"fs:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . FSReadAll . { \"statement\": [ { \"action\": [ \"fs:List*\", \"fs:Read*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . FSReadWriteAll . { \"statement\": [ { \"action\": [ \"fs:Read*\", \"fs:List*\", \"fs:WriteObject\", \"fs:DeleteObject\", \"fs:RevertBranch\", \"fs:CreateBranch\", \"fs:CreateTag\", \"fs:DeleteBranch\", \"fs:DeleteTag\", \"fs:CreateCommit\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . AuthFullAccess . { \"statement\": [ { \"action\": [ \"auth:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . AuthManageOwnCredentials . { \"statement\": [ { \"action\": [ \"auth:CreateCredentials\", \"auth:DeleteCredentials\", \"auth:ListCredentials\", \"auth:ReadCredentials\" ], \"effect\": \"allow\", \"resource\": \"arn:lakefs:auth:::user/${user}\" } ] } . RepoManagementFullAccess . { \"statement\": [ { \"action\": [ \"ci:*\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"retention:*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . RepoManagementReadAll . { \"statement\": [ { \"action\": [ \"ci:Read*\" ], \"effect\": \"allow\", \"resource\": \"*\" }, { \"action\": [ \"retention:Get*\" ], \"effect\": \"allow\", \"resource\": \"*\" } ] } . ",
     "url": "/security/rbac.html#preconfigured-policies",
 
     "relUrl": "/security/rbac.html#preconfigured-policies"

howto/import.html

@@ -691,7 +691,7 @@ <h2 class="no_toc d-inline-block" id="fine-grained-permissions">
 <p class="label label-purple">lakeFS Enterprise</p>
 
 <p>With RBAC support, The lakeFS user running the import command should have the following permissions in lakeFS:
-<code class="language-plaintext highlighter-rouge">fs:WriteObject</code>, <code class="language-plaintext highlighter-rouge">fs:CreateMetaRange</code>, <code class="language-plaintext highlighter-rouge">fs:CreateCommit</code>, <code class="language-plaintext highlighter-rouge">fs:ImportFromStorage</code> and <code class="language-plaintext highlighter-rouge">fs:ImportCancel</code>.</p>
+<code class="language-plaintext highlighter-rouge">fs:WriteObject</code>, <code class="language-plaintext highlighter-rouge">fs:CreateCommit</code>, <code class="language-plaintext highlighter-rouge">fs:ImportFromStorage</code> and <code class="language-plaintext highlighter-rouge">fs:ImportCancel</code>.</p>
 
 <p>As mentioned above, all of these permissions are available by default to the Supers (open-source) group or the SuperUsers (Cloud/Enterprise).</p>
 <h2 class="no_toc" id="provider-specific-permissions">

howto/mirroring.html

@@ -703,7 +703,6 @@ <h3 id="creating-a-lakefs-user-with-a-replicator-policy">
             </span><span class="s2">"fs:DeleteObject"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ListObjects"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:CreateCommit"</span><span class="p">,</span><span class="w">
-            </span><span class="s2">"fs:CreateMetaRange"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ReadCommit"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ListCommits"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:CreateBranch"</span><span class="p">,</span><span class="w">
@@ -756,7 +755,6 @@ <h3 id="creating-a-lakefs-user-with-a-replicator-policy">
             </span><span class="s2">"fs:DeleteObject"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ListObjects"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:CreateCommit"</span><span class="p">,</span><span class="w">
-            </span><span class="s2">"fs:CreateMetaRange"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ReadCommit"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:ListCommits"</span><span class="p">,</span><span class="w">
             </span><span class="s2">"fs:CreateBranch"</span><span class="p">,</span><span class="w">

security/rbac.html

@@ -1301,8 +1301,7 @@ <h3 id="fsreadwriteall">
                 </span><span class="s2">"fs:CreateTag"</span><span class="p">,</span><span class="w">
                 </span><span class="s2">"fs:DeleteBranch"</span><span class="p">,</span><span class="w">
                 </span><span class="s2">"fs:DeleteTag"</span><span class="p">,</span><span class="w">
-                </span><span class="s2">"fs:CreateCommit"</span><span class="p">,</span><span class="w">
-                </span><span class="s2">"fs:CreateMetaRange"</span><span class="w">
+                </span><span class="s2">"fs:CreateCommit"</span><span class="w">
             </span><span class="p">],</span><span class="w">
             </span><span class="nl">"effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"allow"</span><span class="p">,</span><span class="w">
             </span><span class="nl">"resource"</span><span class="p">:</span><span class="w"> </span><span class="s2">"*"</span><span class="w">