feat: auto-enable locking if table exists (#116)

* docs: add "Why?" section to readme

* test: add pyinstrument profiler

* feat: auto-enable locking if table exists

* feat: add `s3pypi force-unlock` command

* feat: use `user@hostname` for lock owner (faster than sts caller identity)

* fix: parse distribution filenames consistently in `upload` and `delete`

* docs: add example iam policy

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 ### Added
 
 - `s3pypi delete` command to delete packages from S3.
+- `s3pypi force-unlock` command to release a stuck lock in DynamoDB.
+- `--locks-table` option to customise the DynamoDB table name used for locking.
 
 ### Changed
 
@@ -20,6 +22,7 @@ and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 ### Removed
 
 - `--acl` option. Use `--s3-put-args='ACL=...'` instead. [The use of ACLs is discouraged].
+- `--lock-indexes` option. Locking is enabled automatically if a DynamoDB table exists.
 
 [The use of ACLs is discouraged]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html
 

Makefile

@@ -11,6 +11,9 @@ format:
 	poetry run isort --apply
 	poetry run black .
 
+profile:
+	poetry run pyinstrument -r html -m pytest tests/integration/test_main.py
+
 clean:
 	rm -rf .coverage .eggs/ .pytest_cache/ .tox/ \
 		build/ coverage/ dist/ pip-wheel-metadata/

README.md

@@ -3,6 +3,28 @@
 S3PyPI is a CLI for creating a Python Package Repository in an S3 bucket.
 
 
+## Why?
+
+The official [Python Package Index (PyPI)](https://pypi.org) is a public
+repository of Python software. It's used by `pip` to download packages.
+
+If you work at a company, you may wish to publish your packages somewhere
+private instead, and still have them be accessible via `pip install`. This
+requires hosting your own repository.
+
+S3PyPI enables hosting a private repository at a low cost. It requires only an
+[S3 bucket] for storage, and some way to serve files over HTTPS (e.g. [Amazon
+CloudFront]).
+
+Publishing packages and index pages to S3 is done using the `s3pypi` CLI.
+Creating the S3 bucket and CloudFront distribution is done using a provided
+[Terraform] configuration, which you can tailor to your own needs.
+
+[S3 bucket]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
+[Amazon CloudFront]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html
+[Terraform]: https://www.terraform.io/
+
+
 ## Alternatives
 
 - [AWS CodeArtifact](https://aws.amazon.com/codeartifact/) is a fully managed
@@ -24,8 +46,8 @@ $ pip install s3pypi
 
 Before you can start using `s3pypi`, you must set up an S3 bucket for storing
 packages, and a CloudFront distribution for serving files over HTTPS. Both of
-these can be created using the [Terraform](https://www.terraform.io/)
-configuration provided in the `terraform/` directory:
+these can be created using the [Terraform] configuration provided in the
+`terraform/` directory:
 
 ```console
 $ git clone https://github.com/gorilla-co/s3pypi.git
@@ -148,6 +170,45 @@ $ s3pypi upload ... --index.html --s3-put-args='ACL=public-read'
 [Origin Access Identity (OAI)]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html
 
 
+### Example IAM policy
+
+The `s3pypi` CLI requires the following IAM permissions to access S3 and
+(optionally) DynamoDB. Replace `example-bucket` by your S3 bucket name.
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:DeleteObject"
+      ],
+      "Resource": "arn:aws:s3:::example-bucket/*"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:ListBucket"
+      ],
+      "Resource": "arn:aws:s3:::example-bucket"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "dynamodb:GetItem",
+        "dynamodb:PutItem",
+        "dynamodb:DeleteItem"
+      ],
+      "Resource": "arn:aws:dynamodb:*:*:table/example-bucket-locks"
+    }
+  ]
+}
+```
+
+
 ## Usage
 
 ### Distributing packages

pyproject.toml

@@ -12,7 +12,7 @@ s3pypi = "s3pypi.__main__:main"
 
 [tool.poetry.dependencies]
 boto3 = "^1.34.11"
-boto3-stubs = {extras = ["s3"], version = "^1.34.11"}
+boto3-stubs = {extras = ["dynamodb", "s3"], version = "^1.34.11"}
 python = "^3.8"
 
 [tool.poetry.group.dev.dependencies]
@@ -22,6 +22,7 @@ flake8 = "^5.0.4"
 isort = "^5.13.2"
 moto = "^4.2.12"
 mypy = "^1.8.0"
+pyinstrument = "^4.6.1"
 pytest = "^7.4.3"
 pytest-cov = "^4.1.0"
 

s3pypi/__main__.py

@@ -59,15 +59,24 @@ def add_command(
     d.add_argument("version", help="Package version.")
     build_s3_args(d)
 
+    ul = add_command(force_unlock, help="Release a stuck lock in DynamoDB.")
+    ul.add_argument("table", help="DynamoDB table.")
+    ul.add_argument("lock_id", help="ID of the lock to release.")
+    build_aws_args(ul)
+
     return p
 
 
+def build_aws_args(p: ArgumentParser) -> None:
+    p.add_argument("--profile", help="Optional AWS profile to use.")
+    p.add_argument("--region", help="Optional AWS region to target.")
+
+
 def build_s3_args(p: ArgumentParser) -> None:
     p.add_argument("-b", "--bucket", required=True, help="The S3 bucket to upload to.")
     p.add_argument("--prefix", help="Optional prefix to use for S3 object names.")
 
-    p.add_argument("--profile", help="Optional AWS profile to use.")
-    p.add_argument("--region", help="Optional AWS region to target.")
+    build_aws_args(p)
     p.add_argument(
         "--no-sign-request",
         action="store_true",
@@ -96,12 +105,9 @@ def build_s3_args(p: ArgumentParser) -> None:
         ),
     )
     p.add_argument(
-        "--lock-indexes",
-        action="store_true",
-        help=(
-            "Lock index objects in S3 using a DynamoDB table named `<bucket>-locks`. "
-            "This ensures that concurrent invocations of s3pypi do not overwrite each other's changes."
-        ),
+        "--locks-table",
+        metavar="TABLE",
+        help="DynamoDB table to use for locking (default: `<bucket>-locks`).",
     )
 
 
@@ -119,6 +125,10 @@ def delete(cfg: core.Config, args: Namespace) -> None:
     core.delete_package(cfg, name=args.name, version=args.version)
 
 
+def force_unlock(cfg: core.Config, args: Namespace) -> None:
+    core.force_unlock(cfg, args.table, args.lock_id)
+
+
 def main(*raw_args: str) -> None:
     args = build_arg_parser().parse_args(raw_args or sys.argv[1:])
     log.setLevel(logging.DEBUG if args.verbose else logging.INFO)
@@ -133,7 +143,13 @@ def main(*raw_args: str) -> None:
             endpoint_url=args.s3_endpoint_url,
             put_kwargs=args.s3_put_args,
             index_html=args.index_html,
-            lock_indexes=args.lock_indexes,
+            locks_table=args.locks_table,
+        )
+        if hasattr(args, "bucket")
+        else core.S3Config(
+            bucket="",
+            profile=args.profile,
+            region=args.region,
         ),
     )