All notable changes to this project will be documented in this file.
Please find documentation here: https://tapis.readthedocs.io/en/latest/technical/files.html
You may also reference live-docs based on the openapi specification here: https://tapis-project.github.io/live-docs
Incremental improvements and new features.
Enhanced handling of transfers so that transfers are fair between tenants.
- None
-
The files postgres data must be migrated to the new postgres 16 container. The new version will have a burnup script for a files-postgres-16 container, and it uses a new (empty) pvc. Before bringing all files services up, you must bring up the new files-postgres-16 container, and migrate the postgres data from the files-postgres container to the files-postgres-16 container. The best place to get information about this is the postgres docs. This could be done manually (using dbdump/dbdumpall). One thing to note is that passwords for postgres users will not be migrated and must be reset due to changes in the way that postgres hashes the passwords, so be sure to find the db user passwords before you get started. This could be scripted and an example of how that might be done is included. Here is the information on the examples:
- Backup your current database in case something goes wrong.
- After deployment, burn down all files pods EXCEPT files-postgres.
- Verify that the scripts look correct for your environment. Check the variables are defined in files-migrate-pg-11-to-16-vars-configmap.yml
- In postgres_16 directory, cd into migrate, and run migrate.sh.
- Check table counts after the job completes (these will be located in stdout for the script). Check for errors, etc.
- cd into the files/postgres directory (postgres - not postgres_16), and burndown the old postgres.
- Once the data is successfully migrated and verified, burndown any files services that are currently running and restart. To do this, cd into the files directory and run the burndown and burnup scripts.
Bug fix for setting destination shared context for transfers.
- None
- Fix issue with passing in correct destination shared context during file transfers.
- None
Incremental improvements and new features.
- None
- Fixed minor issue with file permissions of executable files during transfers
- Fixed issue with "postit" expiration
- None
Incremental improvements and new features.
- None
- Fixed minor issue with ssh session caching / session closing
- Fix for irods client not closing session in certain cases
- Fix for transfers not properly handling sharedCtxGrantor
- Fix for very large globus file listings
- None
Incremental improvements and new features.
- None
- Performance improvements in transfers with many files.
- None
Incremental improvements and new features.
Enhancements to concurrency in the way that SSH/SFTP Sessions are used improving performance.
- Improved handling of pool ssh/sftp connections allowing for better performance under higher loads.
- Added proxy user for irods.
- Fixed issue concerning making directories starting with a '.'.
- Fixed issue where directories with more than 1000 entries were not handled properly for most operations.
- Fixed issue where making a directory during a transfer could fail.
- None
Incremental improvements and new features. The Dockerfiles that build the files api and worker containers now reference an environment variable called ENV_JAVA_CMD_OPTS which can be overridden by the deployment if needed. By default it sets the following: -Xdebug -Xmx3g -agentlib:jdwp=transport=dt_socket,server=y,address=*:8000,suspend=n
Regular expression matching for file listings on linux systems.
- Fix issues with multiple slashes in a relative path. Issue number 151. (Fixed in tapis-shared-java repository)
- Fix issues with Tapis permission checking when a system is shared. For delete and some Linux operations. Issue 150.
- None
Incremental improvements and new features.
- Update and rebuild with latest shared code to incorporate library updates
- Fix broken http/s transfer support.
- None
Incremental improvements and new features.
- Rebuild with latest shared code to fix JWT validation issue.
- None
Incremental improvements and new features.
- Fixed an issue where ssh commands issued by the files service could fail due to a race condition causing the connection to be closed
- Fixed an NPE in mkdir in certain situations
- None
Incremental improvements and new features.
- Use logback.xml outside the tapis files jar, and added additional logging for ssh session pool.
- Added additional logging for linux methods (/v3/files/utils/linux/*).
- Fixed an issue where upload and getStatInfo were not working properly for publicly shared systems.
- Fixed an issue where upload was incorrectly interpreting symlinked directories as files.
- Fixed a race condition in mkdir
- None
Incremental improvements and new features.
- Fixed an issue in file listings / file transfers that could result in duplication of results.
- Fixed an issue in sharing support - Site admin tenant not set in worker service.
- Fixed an issue where transfers continue to retry when effectiveUserId does not have permission to write to a directory.
- None
Incremental improvements and new features.
- Update file path authorization checks to include system sharing. Facilitates sharing an application.
- Fix issue with handling path /~/ for GLOBUS type systems
- Fix issue where no ssh sessions are available
- Fix issue where listing contents of directories containing many subdirectories could result in incomplete results
- Fix issue where transfers containing source and destination paths that are identical could corrupt the file.
- None
Incremental improvements and new features.
- None
- Improvements in caching of system credentials to minimize lag when updating credentials.
- Improved HTTP return statuses in files API.
- None
Incremental improvements and new features.
- Support file operations and transfers for systems of type GLOBUS.
- Improve support of symbolic links on linux systems.
- Corrected an issue where irods username/passords were not retrieved properly.
- IRODS previously used accessKey and accessSecret from system credentials, and did not pay any attention to a system's default authn method. Now IRODS systems must have the default authn method set to "PASSWORD", and credentials for that system must use the password field to set the irods password, and loginUser if the irods user is different than the tapis user.
Shorten system cache timeout from 5 minutes to 10 seconds.
Incremental improvements and bug fixes.
- None
- Fixed a thread leak, and case where we were not cleaning up temporary control queues for file transfers
Incremental improvements and bug fixes.
- None
- Update sharedAppCtx to represent app share grantor. Fix for privilege escalation.
Incremental improvements and bug fixes.
- None
- Fixed issues connecting to rabbitmq on worker startup
- Allow db connection pool to be configurable, and fix issue causing long running transactions.
- Fixed issue where when deleting a file, an error could be returned even though the file was successfully deleted.
- Fixed issue where zip entry for single file did not contain the name of file when downloading a single file.
- When downloading the system root directory the file will now be called systemRoot.zip rather than zip
- When downloading a file, skip the authorization check when retrieving the system.
Incremental improvements and bug fixes.
- None
- Fixed issue where linux directories could not be copied.
- Fixed issue where path permissions cache was not being invalidated when permissions where changed
Incremental improvements.
- Added PostIts feature. This feature allows users to create a link that can be shared allowing files or directories to be downloaded without authentication (limited by time and/or usage count)
- Added endpoint to get/set file acls on Linux systems
- None
Incremental improvements. Improved logging.
- None
- Fix bug in expiring of system cache entry when ssh connection fails.
Incremental improvements and bug fixes.
- Invalidate SSH cache entry when there is an error. Error may get resolved. For example, error could be due to invalid credentials or no route to host.
- Support zip download for system rootDir as long as not all files on host would be included.
- For certain endpoints that take a path make the trailing slash optional when dealing with top level directory: getContents, getListing, sharing.
- When there are multiple transfer elements in a request and one finishes quickly it is possible for some to be missed.
- Fix problems with zip downloads. Require zip when path is directory. Disallow zip if request would include all files on host.
- Fix bugs in updating start times for top level task and parent tasks.
- Fix bug: Listing and download not authorized when path is shared.
Incremental improvements and bug fix.
- Review, refactor and strengthen transfer code. Additional integration tests.
- Fix issue with S3 to LINUX transfer. Destination paths were being constructed incorrectly.
- Fix issue with S3 path contained in FileInfo. Should be relative path and not absolute path.
- Fix issues with populating path and url in FileInfo objects for all system types.
Incremental improvements and preview of new feature.
- Support sharedAppCtx for getContents operation.
- Remove hard coded default value for TAPIS_SITE_ID. Site ID must now be set in the environment at service start.
Incremental improvements and preview of new feature.
- Support sharedAppCtx for listFiles operation.
- Add warning messages for when permission is denied.
- Add error messages for service exceptions.
- None.
Incremental improvements and preview of new feature.
- Support sharing of a path with one or more users. Includes support for sharing with all users in a tenant.
- Support sharedAppCtx for mkdir operation and transfer requests (srcSharedAppCtx, destSharedAppCtx).
- Perform DB migration at service startup.
- Update so MODIFY permission implies READ.
Incremental improvements, new preview features and bug fix.
- Add support for impersonationId query parameter to getTransferTaskDetails.
- Improve handling of paths in S3 support. Keys updated to never start with "/".
- Fix issue with S3 support by updating to the latest AWS java libraries.
Incremental improvements and bug fix.
- None.
- Fix issue with getSKClient() that breaks associate site support.
Incremental improvements and preview of new feature.
- Support impersonationId for service to service requests.
- None
Incremental improvements and preview of new feature.
- Replace skipTapisAuthorization with impersonationId for requests from Jobs and Apps services.
- None
Preview of new feature.
- Support skipTapisAuthorization for requests from Jobs and Apps services.
- None
Fix issues with Jackson mapper and with worker startup..
- None
- None
- Transfer worker failing to start.
- Api throwing exceptions due to empty bean during json serialization.
Updates for JDK 17.
- None
- Updates for JDK 17.
- None
Code cleanup and refactoring, fix issue with SSH client cache.
- None
- Code cleanup and refactoring.
- Fix concurrence problem with SSH client cache. Github issue #39
Incremental improvements, support optional flag for file transfers.
- Support optional flag for file transfers.
- Add debug logging of cache key loads.
- Update S3 client to mostly operate on single objects.
- Additional tests.
- None.
Incremental improvements.
- Support copy/move of a file to a directory without having to specify full path for destination.
Initial release supporting basic file operations in the Tapis ecosystem. Users can perform file listing, uploading, operations such as move/copy/delete and also transfer files between systems. All Tapis files APIs accept JSON as inputs.
Currently the files service includes support for S3 and SSH type file systems. Other storage systems like IRODS will be included in future releases.
- Initial release.
- Initial release.
- None.