I am trying to use S3 of LocalStack with Nextflow but I got runtime error. #5181

spksoft · 2024-07-28T18:27:38Z

spksoft
Jul 28, 2024

Bug report

I trying to use S3 from LocalStack by setting nextflow.config but when I run a workflow it got error

Expected behavior and actual behavior

It should display S3 files.

Steps to reproduce the problem

I have folder structure like this

.
├── Makefile
├── data
├── docker-compose.yml
├── localstack
├── nextflow
│   ├── data
│   ├── main.nf
│   ├── nextflow.config
│   └── work
└── scripts
    └── encode_base64.py

docker-compose.yaml

version: '3.8'

services:
  localstack:
    image: localstack/localstack:latest
    container_name: localstack
    environment:
      - SERVICES=s3
      - DEBUG=1
    healthcheck:
      test:
        - CMD
        - bash
        - -c
        - awslocal s3 ls
      interval: 5s
      timeout: 10s
      start_period: 10s
    ports:
      - "4566:4566"
      - "4571:4571"
    volumes:
      - ./localstack:/var/lib/localstack

  init:
    image: amazon/aws-cli:latest
    container_name: init
    entrypoint: ["sh", "-c", "aws --endpoint-url=http://localstack:4566 s3api create-bucket --bucket test-bucket && aws --endpoint-url=http://localstack:4566 s3api put-object --bucket test-bucket --key work/"]
    environment:
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - /tmp:/tmp
    depends_on:
      localstack:
        condition: service_healthy

  nextflow:
    image: nextflow/nextflow:latest
    container_name: nextflow
    platform: linux/amd64
    volumes:
      - ./nextflow:/workspace/nextflow
      - ./scripts:/workspace/scripts
    working_dir: /workspace/nextflow
    environment:
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_S3_ENDPOINT=http://localstack:4566
    command: bash -c "while true; do sleep 1; done"

  python:
    image: python:3.9-slim
    container_name: python
    volumes:
      - ./scripts:/workspace/scripts
      - ./data:/workspace/data
    working_dir: /workspace/scripts
    command: bash -c "while true; do sleep 1; done"

volumes:
  nextflow_data:
  localstack_data:

nextflow.config

aws {
    accessKey = 'test'
    secretKey = 'test'
    region = 'us-east-1'
    client {
        endpoint = 'http://localstack:4566'
        s3PathStyleAccess = true
    }
}

main.nf

println "AWS_ACCESS_KEY_ID: ${System.getenv('AWS_ACCESS_KEY_ID')}"
println "AWS_SECRET_ACCESS_KEY: ${System.getenv('AWS_SECRET_ACCESS_KEY')}"
println "AWS_DEFAULT_REGION: ${System.getenv('AWS_DEFAULT_REGION')}"
println "AWS_S3_ENDPOINT Endpoint: ${System.getenv('AWS_S3_ENDPOINT')}"

file('s3://test-bucket/work/').list().each { println it }

and then I execute workflow by

$ docker compose up --build
$ docker exec -it nextflow nextflow run /workspace/nextflow/main.nf

Program output

Output:

 N E X T F L O W   ~  version 24.04.3

Launching `main.nf` [angry_poisson] DSL2 - revision: 3a7c65154b

AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_DEFAULT_REGION: us-east-1
S3 Endpoint: http://localstack:4566
ERROR ~ The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)

 -- Check script 'main.nf' at line: 6 or see '.nextflow.log' file for more details

.nextflow.log

Jul-28 18:23:53.891 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Jul-28 18:23:54.160 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.3
Jul-28 18:23:54.226 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jul-28 18:23:54.284 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-28 18:23:54.286 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-28 18:23:54.295 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jul-28 18:23:54.334 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jul-28 18:23:54.401 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /workspace/nextflow/nextflow.config
Jul-28 18:23:54.405 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /workspace/nextflow/nextflow.config
Jul-28 18:23:54.474 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /.nextflow/secrets/store.json
Jul-28 18:23:54.481 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@267bbe1a] - activable => nextflow.secret.LocalSecretsProvider@267bbe1a
Jul-28 18:23:54.511 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jul-28 18:23:55.580 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-28 18:23:55.599 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [angry_poisson] DSL2 - revision: 3a7c65154b
Jul-28 18:23:55.603 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-28 18:23:55.604 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-28 18:23:55.699 [main] DEBUG nextflow.Session - Session UUID: 15cf5789-31da-45a2-8cf8-f631afb484ab
Jul-28 18:23:55.700 [main] DEBUG nextflow.Session - Run name: angry_poisson
Jul-28 18:23:55.702 [main] DEBUG nextflow.Session - Executor pool size: 10
Jul-28 18:23:55.716 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-28 18:23:55.727 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=30; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-28 18:23:55.766 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 24.04.3 build 5916
  Created: 09-07-2024 19:35 UTC 
  System: Linux 6.6.32-linuxkit
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 17.0.10+7-LTS
  Encoding: UTF-8 (UTF-8)
  Process: 897@347678c70a13 [172.21.0.3]
  CPUs: 10 - Mem: 7.7 GB (4.7 GB) - Swap: 1024 MB (1024 MB)
Jul-28 18:23:55.834 [main] DEBUG nextflow.Session - Work-dir: /workspace/nextflow/work [UNKNOWN (0x6a656a63)]
Jul-28 18:23:55.836 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /workspace/nextflow/bin
Jul-28 18:23:55.847 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-28 18:23:55.862 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-28 18:23:55.960 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-28 18:23:56.029 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 11; maxThreads: 1000
Jul-28 18:23:56.175 [main] DEBUG nextflow.Session - Session start
Jul-28 18:23:56.557 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-28 18:23:56.593 [main] INFO  nextflow.script.BaseScript - AWS_ACCESS_KEY_ID: test
Jul-28 18:23:56.594 [main] INFO  nextflow.script.BaseScript - AWS_SECRET_ACCESS_KEY: test
Jul-28 18:23:56.595 [main] INFO  nextflow.script.BaseScript - AWS_DEFAULT_REGION: us-east-1
Jul-28 18:23:56.596 [main] INFO  nextflow.script.BaseScript - S3 Endpoint: http://localstack:4566
Jul-28 18:23:56.609 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.5.2
Jul-28 18:23:56.651 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Jul-28 18:23:56.652 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Jul-28 18:23:56.686 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Jul-28 18:23:56.710 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jul-28 18:23:56.712 [main] DEBUG nextflow.file.FileHelper - Started plugin 'nf-amazon' required to handle file: s3://test-bucket/work/
Jul-28 18:23:56.727 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jul-28 18:23:56.751 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {endpoint=http://localstack:4566, s_3_path_style_access=true, max_error_retry=5}
Jul-28 18:23:58.667 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_212421a51cfab87f: /workspace/nextflow/main.nf
Jul-28 18:23:58.669 [main] DEBUG nextflow.Session - Session aborted -- Cause: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)
Jul-28 18:23:58.703 [main] ERROR nextflow.cli.Launcher - @unknown
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:949)
	at nextflow.cloud.aws.nio.S3Client.listObjects(S3Client.java:137)
	at nextflow.cloud.aws.nio.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:89)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttr0(S3FileSystemProvider.java:702)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttributes(S3FileSystemProvider.java:686)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
	at java.base/java.nio.file.Files.isDirectory(Files.java:2322)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.isDirectory(FilesEx.groovy:601)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:59)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:57)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.listFiles(FilesEx.groovy:640)
	at nextflow.extension.FilesEx.listFiles(FilesEx.groovy)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.list(FilesEx.groovy:628)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:59)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:57)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at Script_212421a51cfab87f.runScript(Script_212421a51cfab87f:6)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.script.BaseScript.run(BaseScript.groovy:209)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
	at nextflow.cli.Launcher.run(Launcher.groovy:503)
	at nextflow.cli.Launcher.main(Launcher.groovy:657)

Environment

Nextflow version: 24.04.3.5916
Java version:

openjdk 17.0.10 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

Operating system: docker of nextflow/nextflow:latest
Bash version:

GNU bash, version 5.2.15(1)-release (x86_64-amazon-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I am trying to use S3 of LocalStack with Nextflow but I got runtime error. #5181

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

I am trying to use S3 of LocalStack with Nextflow but I got runtime error. #5181

spksoft Jul 28, 2024

Bug report

Expected behavior and actual behavior

Steps to reproduce the problem

Program output

Environment

Replies: 0 comments

spksoft
Jul 28, 2024