Issues generating synthetic data

[v-anne]

After @mlissner's helpful advice for the container issue I had, I was able to proceed to running the website locally. However, I'm encountering an unfamiliar error when trying to generate synthetic data using the make_dev_data.py script: botocore.exceptions.NoCredentialsError: Unable to locate credentials. Google tells me that is likely an issue with AWS keys, but I haven't touched those parameters in the .env.dev file I made. This wasn't an issue in the past, but is popping up now. To be clear, I don't think I need access to AWS since it should definitely be possible to generate data locally.

[mlissner]

Thanks @v-anne. Looks like we should update this in our readme. @quevon24, can you help? I bet if you just put dummy values in there, it'd work, but I guess it'd be nice if the script didn't need values.

[v-anne]

Yeah, that's why I mentioned it. It seemed odd to me. Entire stack trace is below.

Click to expand stack trace

Traceback (most recent call last): File "/opt/courtlistener/manage.py", line 15, in main() File "/opt/courtlistener/manage.py", line 11, in main execute_from_command_line(sys.argv) File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute output = self.handle(*args, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/courtlistener/cl/lib/management/commands/make_dev_data.py", line 123, in handle Factory.create_batch(count) File "/usr/local/lib/python3.12/site-packages/factory/base.py", line 540, in create_batch return [cls.create(**kwargs) for _ in range(size)] ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/base.py", line 528, in create return cls._generate(enums.CREATE_STRATEGY, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/django.py", line 121, in _generate return super()._generate(strategy, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/base.py", line 465, in _generate return step.build() ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/builder.py", line 274, in build instance = self.factory_meta.instantiate( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/base.py", line 317, in instantiate return self.factory._create(model, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/factory/django.py", line 174, in _create return manager.create(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/query.py", line 679, in create obj.save(force_insert=True, using=self.db) File "/opt/courtlistener/cl/search/models.py", line 811, in save super().save(update_fields=update_fields, *args, **kwargs) File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 822, in save self.save_base( File "/usr/local/lib/python3.12/site-packages/model_utils/tracker.py", line 343, in inner return original(instance, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/model_utils/tracker.py", line 343, in inner return original(instance, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/model_utils/tracker.py", line 343, in inner return original(instance, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 1 more time] File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 909, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1071, in _save_table results = self._do_insert( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1112, in _do_insert return manager._insert( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/query.py", line 1847, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql for sql, params in self.as_sql(): ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1747, in as_sql self.prepare_value(field, self.pre_save_val(field, obj)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val return field.pre_save(obj, add=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/models/fields/files.py", line 324, in pre_save file.save(file.name, file.file, save=False) File "/usr/local/lib/python3.12/site-packages/django/db/models/fields/files.py", line 94, in save self.name = self.storage.save(name, content, max_length=self.field.max_length) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/core/files/storage/base.py", line 44, in save name = self.get_available_name(name, max_length=max_length) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/courtlistener/cl/lib/storage.py", line 82, in get_available_name return get_name_by_incrementing(self, name, max_length) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/courtlistener/cl/lib/storage.py", line 51, in get_name_by_incrementing while instance.exists(name): ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/storages/backends/s3.py", line 514, in exists self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name) File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 1001, in _make_api_call http, parsed_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 1027, in _make_request return self._endpoint.make_request(operation_model, request_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/endpoint.py", line 119, in make_request return self._send_request(request_dict, operation_model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/endpoint.py", line 198, in _send_request request = self.create_request(request_dict, operation_model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/endpoint.py", line 134, in create_request self._event_emitter.emit( File "/usr/local/lib/python3.12/site-packages/botocore/hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/hooks.py", line 256, in emit return self._emit(event_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/hooks.py", line 239, in _emit response = handler(**kwargs) ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/signers.py", line 105, in handler return self.sign(operation_name, request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/botocore/signers.py", line 199, in sign auth.add_auth(request) File "/usr/local/lib/python3.12/site-packages/botocore/auth.py", line 418, in add_auth raise NoCredentialsError() botocore.exceptions.NoCredentialsError: Unable to locate credentials

[mlissner]

And if you put them into your env file?

[v-anne]

I did this in my .env.dev file. I wasn't sure if they have to fit the actual API key format, or if any dummy data is fine.

python3 AWS_DEV_ACCESS_KEY_ID="a" AWS_DEV_SECRET_ACCESS_KEY="a" AWS_ACCESS_KEY_ID="a" AWS_SECRET_ACCESS_KEY="a" AWS_LAMBDA_PROXY_URL="a"
Apologies for the poor formatting.

[v-anne]

Bottom line--I'm not sure what's up here, because the only change I made to the make_dev_data.py script was to include two more model factories. I didn't have the AWS error a few weeks ago, but it is popping up now.

[mlissner]

I suspect that your factories are on models that use S3 for storage. Maaaaybe that's triggering it, but I think I need some help here. @ERosendo, can you help me out and see what it takes to get @v-anne on track? Seems like we'll want some tweaks to our dev guide too, if there's a missing step in there (which seems likely).