v0.4.0-rc.2

🚧 PRE-RELEASE 🚧
This is the pre-release for the upcoming 0.4.0 release. The CHANGELOG below may be missing some information. It is not recommended that you update to this pre-release unless you have a good reason, or want to test new features before they are finished.

This release of datalab contains significant new functionality whilst broadly maintaining compatibility with the 0.3.x series.

It is also accompanied by the first release of the datalab Python API package (https://github.com/datalab-org/datalab-api), as well as the first release of the Ansible playbooks and Terraform rules (Azure only, for now) to automated datalab deployments (https://github.com/datalab-org/datalab-ansible-terraform).

Special thanks go to @BenjaminCharmes and @vrajpatel9988 who made their first contributions to datalab in this release!

Highlights

• User accounts: users can now update their name and contact info, as well as connect external accounts and regenerate API keys directly from the web UI. The ability to login via ORCID is now enabled by default (but must be configured at the instance level).
• Admin dashboard: adds the ability for admins to do user management from the UI directly.
• Electrochemistry block: support for Neware file formats, MPR files written by ECLab > 11.50 and cyclic voltammetry data.
• Equipment: A new entry type has been added to record the equipment in the lab used for certain operations
• Better default permissions on inventories: users can now create and edit inventory entries without requiring an admin.
• Automatic random IDs: The ability to generate random IDs for new samples was added, allowing datalab to be used as the source of IDs.
• General improvements to block error reporting and reactivity, as well as several bug fixes.

Notes for upgrading to v0.4.0

These notes will be populated for the final release, but not yet.

Full changelog

• Dynamically set production app container config in entrypoint by @ml-evs in #605
• Visual style improvements, test updates and improved block UI by @jdbocarsly in #608
• Refactor mail config settings by @ml-evs in #614
• Add Neware support and fix MPR issue by @ml-evs in #617
• Hotfix for block errors and warnings UI that arise over multiple renderings by @ml-evs in #615
• Use newly released galvani and NewareNDA packages by @ml-evs in #625
• Update xrdml parser so it works with v2.0 by @jdbocarsly in #631
• Restrict relationship types in UI synthesis and constituents tables by @ml-evs in #630
• Update copyright year in LICENSE by @ml-evs in #622
• Allow browser context menu in tinymce fields by @jdbocarsly in #635
• Allow any authenticated user to generate an API key by @ml-evs in #641
• Serve the identifier prefix in the /info response by @ml-evs in #638
• Return user role from /get-current-user endpoint by @ml-evs in #644
• Created component for editing account settings by @BenjaminCharmes in #627
• User registration: default display name to GitHub username when no profile name is set by @ml-evs in #655
• Fix typo where CI tests were never run in chrome by @ml-evs in #656
• Temporarily disable Firefox tests by @ml-evs in #659
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /webapp by @dependabot in #660
• Bump follow-redirects from 1.15.4 to 1.15.6 in /webapp by @dependabot in #647
• Allow echem block to plot non-cyclic data by @ml-evs in #665
• Add optional functionality to add starting materials from within the webapp by @jdbocarsly in #642
• Added validation for the user update route/UI by @BenjaminCharmes in #646
• Improve block warning/error reactivity in the UI by @ml-evs in #666
• Add ability to generate .csv files within block callbacks by @ml-evs in #621
• Debug flaky e2e tests by @jdbocarsly in #670
• Add option to generate ID automatically when creating items by @jdbocarsly in #668
• Add default config for VUE_APP_EDITABLE_INVENTORY by @ml-evs in #673
• Add codecov upload to CI by @ml-evs in #677
• Disable codecov PR annotations by @ml-evs in #681
• Add "equipment" item type by @jdbocarsly in #531
• Convert Whinchat to use langchain by @vrajpatel9988 in #661
• Bump langchain-core from 0.1.33 to 0.1.35 in /pydatalab by @dependabot in #686
• Bump prettier, pre-commit hooks and some other deps by @ml-evs in #685
• Allow starting materials and equipment to be edited by normal users by @ml-evs in #672
• Bump pymongo from 4.6.2 to 4.6.3 in /pydatalab by @dependabot in #689
• Use mocked API keys in API tests to fake different authentication scenarios by @ml-evs in #676
• Significant refactor of deployment docs by @ml-evs in #690
• Enable ORCID connection by default by @ml-evs in #693
• Only fix PRs with pre-commit CI when asked by @ml-evs in #697
• Upgrade cypress and other testing deps to allow Firefox testing by @ml-evs in #699
• Update README with links to Python API by @ml-evs in #701
• Add public deployment badge in README by @ml-evs in #702
• Separate each browser test into separate run and temporarily disable Firefox by @ml-evs in #703
• Add admin dashboard with user management controls by @BenjaminCharmes in #674
• Added a way to refresh API Key from Account Settings UI by @BenjaminCharmes in #700
• Recreate user index with new settings if already existing by @ml-evs in #707
• Fix for saving collection blocks and error handling for excessively large blocks by @ml-evs in #709
• Add account_status field (active, unverified or deactivated) to People model by @BenjaminCharmes in #687
• Update deployment instructions wrt. new datalab-ansible-terraform repo by @ml-evs in #712
• Add UI for email registration/login by @ml-evs in #528
• Serve data about available block types in API by @ml-evs in #667
• Add upper pin to rosettasciio by @ml-evs in #720
• Update tests for current user by @ml-evs in #723
• Add ability to select between different chat models in the whinchat by @vrajpatel9988 in #680
• Bump Flask and Werkzeug to the latest releases on the v3 series by @ml-evs in #722
• Added HelpBubble component to display inline documentation in the UI by @BenjaminCharmes in #706
• Fix API key help message by @ml-evs in #725
• Fix README badges by @ml-evs in #730

New Contributors

• @BenjaminCharmes made their first contribution in #627
• @vrajpatel9988 made their first contribution in #661

Full Changelog: v0.3.2...v0.4.0-rc.2

v0.4.0-rc.1

🚧 PRE-RELEASE 🚧
This is the pre-release for the upcoming 0.4.0 release. The CHANGELOG below may be missing some information. It is not recommended that you update to this pre-release unless you have a good reason, or want to test new features before they are finished.

This release of datalab contains significant new functionality whilst broadly maintaining compatibility with the 0.3.x series.

It is also accompanied by the first release of the datalab Python API package (https://github.com/datalab-org/datalab-api), as well as the first release of the Ansible playbooks and Terraform rules (Azure only, for now) to automated datalab deployments (https://github.com/datalab-org/datalab-ansible-terraform).

Special thanks go to @BenjaminCharmes and @vrajpatel9988 who made their first contributions to datalab in this release!

Highlights

• User accounts: users can now update their name and contact info, as well as connect external accounts and regenerate API keys directly from the web UI. The ability to login via ORCID is now enabled by default (but must be configured at the instance level).
• Admin dashboard: adds the ability for admins to do user management from the UI directly.
• Electrochemistry block: support for Neware file formats, MPR files written by ECLab > 11.50 and cyclic voltammetry data.
• Equipment: A new entry type has been added to record the equipment in the lab used for certain operations
• Better default permissions on inventories: users can now create and edit inventory entries without requiring an admin.
• Automatic random IDs: The ability to generate random IDs for new samples was added, allowing datalab to be used as the source of IDs.
• General improvements to block error reporting and reactivity, as well as several bug fixes.

Notes for upgrading to v0.4.0

These notes will be populated for the final release, but not yet.

Full changelog

• Dynamically set production app container config in entrypoint by @ml-evs in #605
• Visual style improvements, test updates and improved block UI by @jdbocarsly in #608
• Refactor mail config settings by @ml-evs in #614
• Add Neware support and fix MPR issue by @ml-evs in #617
• Hotfix for block errors and warnings UI that arise over multiple renderings by @ml-evs in #615
• Use newly released galvani and NewareNDA packages by @ml-evs in #625
• Update xrdml parser so it works with v2.0 by @jdbocarsly in #631
• Restrict relationship types in UI synthesis and constituents tables by @ml-evs in #630
• Update copyright year in LICENSE by @ml-evs in #622
• Allow browser context menu in tinymce fields by @jdbocarsly in #635
• Allow any authenticated user to generate an API key by @ml-evs in #641
• Serve the identifier prefix in the /info response by @ml-evs in #638
• Return user role from /get-current-user endpoint by @ml-evs in #644
• Created component for editing account settings by @BenjaminCharmes in #627
• User registration: default display name to GitHub username when no profile name is set by @ml-evs in #655
• Fix typo where CI tests were never run in chrome by @ml-evs in #656
• Temporarily disable Firefox tests by @ml-evs in #659
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /webapp by @dependabot in #660
• Bump follow-redirects from 1.15.4 to 1.15.6 in /webapp by @dependabot in #647
• Allow echem block to plot non-cyclic data by @ml-evs in #665
• Add optional functionality to add starting materials from within the webapp by @jdbocarsly in #642
• Added validation for the user update route/UI by @BenjaminCharmes in #646
• Improve block warning/error reactivity in the UI by @ml-evs in #666
• Add ability to generate .csv files within block callbacks by @ml-evs in #621
• Debug flaky e2e tests by @jdbocarsly in #670
• Add option to generate ID automatically when creating items by @jdbocarsly in #668
• Add default config for VUE_APP_EDITABLE_INVENTORY by @ml-evs in #673
• Add codecov upload to CI by @ml-evs in #677
• Disable codecov PR annotations by @ml-evs in #681
• Add "equipment" item type by @jdbocarsly in #531
• Convert Whinchat to use langchain by @vrajpatel9988 in #661
• Bump langchain-core from 0.1.33 to 0.1.35 in /pydatalab by @dependabot in #686
• Bump prettier, pre-commit hooks and some other deps by @ml-evs in #685
• Allow starting materials and equipment to be edited by normal users by @ml-evs in #672
• Bump pymongo from 4.6.2 to 4.6.3 in /pydatalab by @dependabot in #689
• Use mocked API keys in API tests to fake different authentication scenarios by @ml-evs in #676
• Significant refactor of deployment docs by @ml-evs in #690
• Enable ORCID connection by default by @ml-evs in #693
• Only fix PRs with pre-commit CI when asked by @ml-evs in #697
• Upgrade cypress and other testing deps to allow Firefox testing by @ml-evs in #699
• Update README with links to Python API by @ml-evs in #701
• Add public deployment badge in README by @ml-evs in #702
• Separate each browser test into separate run and temporarily disable Firefox by @ml-evs in #703
• Add admin dashboard with user management controls by @BenjaminCharmes in #674
• Added a way to refresh API Key from Account Settings UI by @BenjaminCharmes in #700
• Recreate user index with new settings if already existing by @ml-evs in #707
• Fix for saving collection blocks and error handling for excessively large blocks by @ml-evs in #709
• Add account_status field (active, unverified or deactivated) to People model by @BenjaminCharmes in #687

New Contributors

• @BenjaminCharmes made their first contribution in #627
• @vrajpatel9988 made their first contribution in #661

Full Changelog: v0.3.2...v0.4.0-rc.1

v0.3.2

v0.3.2

This is a build hotfix for 0.3.x, where the incorrect versions of navani and galvani were being installed in docker builds (leading to missing support for ECLab >= 11.50).

What's Changed

• Force galvani and navani update in lockfile by @ml-evs in #602

Full Changelog: v0.3.1...v0.3.2

v0.3.1

v0.3.1

This release makes a few fixes relative to 0.3.0, primarily around file uploads and internal database storage.

It also adds the ability for data blocks to pass errors and warnings to the front-end, which can be useful for helping describe incompatibilities in data, analysis or plotting dynamically.

What's Changed

• Pin prettier version in Webapp to same version that is specified in pre-commit by @jdbocarsly in #589
• Downgrade pre-commit prettier by @ml-evs in #593
• CSS cleanup by @jdbocarsly in #588
• Fix issue where plot data was always stored in db by @ml-evs in #599
• Add ability for block errors and warnings to be passed through API to UI by @ml-evs in #590
• Fix issue where previously uploaded file cannot be replaced by @ml-evs in #594

Full Changelog: v0.3.0...v0.3.1

v0.3.0

v0.3.0

This is the long overdue v0.3.0 release of datalab, which coincides with many new deployments popping up. It is strongly recommended to upgrade and continue to keep up-to-date with releases as they come out. There are no intentional breaking changes between this release and the previous release candidates, so please report any issues you come across on the GitHub issue tracker.

Thanks to new contributor @elbee99 who has added support for Raman spectroscopy, plus new contributors with as-of-yet unreleased changes!

Headline features (relative to v0.2.5)

• Support for Biologic data files created with the most recent versions of ECLab 11.50+, plus enhanced support for Arbin data files
• A new 1D Raman block that can parse both Renishaw WDF and Oxford Instrument's spectra
• Ability to group samples and other entries into collections
• Support for new authentication methods, such as magic links via email
• Significant refactoring and modularization to ease the process of new contributions
• Simplified deployment procedure via Docker
• Automated snapshot backups

Full changelog

• Add concept of user manager permissions by @ml-evs in #417
• Add UI for collections, fix local graphs and improve tables by @ml-evs in #404
• Adjust default echem subsampling by @ml-evs in #426
• Version bumps and linting tweaks by @ml-evs in #429
• Remote filesystem scanning improvements by @ml-evs in #430
• Implemented Raman block by @elbee99 in #422
• Move echem block to its own app submodule by @ml-evs in #433
• Switch from flake8+isort to ruff by @ml-evs in #434
• Update vue/cli to v5 by @ml-evs in #432
• Tweak to remote file system caching for improved page loads by @ml-evs in #435
• Enable Arbin res file parsing by adding mdbtools/0.7.1 build to server Dockerfile by @ml-evs in #436
• Better sanitize inputs for plot cycle selector by @ml-evs in #438
• Add sample/user/cell counters to deployment about page by @ml-evs in #383
• Fix last modified timezone and make value dynamic by @ml-evs in #439
• Restructure NMR block module and add tests by @ml-evs in #441
• Add basic permissions to files by @ml-evs in #445
• Fix broken README link and update affiliations by @ml-evs in #448
• Manual overhaul of API docs by @ml-evs in #451
• Fix bug where user cannot be created with blank display name by @ml-evs in #456
• Bump navani and try to accomodate other deps by @ml-evs in #457
• Update INSTALL.md by @jdbocarsly in #458
• Add subtitled intro video to README by @ml-evs in #459
• Bump @babel/traverse from 7.22.10 to 7.23.2 in /webapp by @dependabot in #463
• Bump apollo-server-core from 3.12.0 to 3.12.1 in /webapp by @dependabot in #446
• Bump tinymce from 5.10.7 to 5.10.8 in /webapp by @dependabot in #465
• Support .wdf files for 1D Raman by @elbee99 in #466
• Add more details to README by @ml-evs in #479
• Adding limits on file uploads by @ml-evs in #475
• Improve login/logout UI to support multiple authentication mechanisms by @ml-evs in #384
• Remove unecessary scope for GitHub OAuth by @ml-evs in #483
• Tweak to deployed documentation by @ml-evs in #485
• Revamp installation docs with additional configuration/administration/deployment info by @ml-evs in #490
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #493
• Bump browserify-sign from 4.2.1 to 4.2.2 in /webapp by @dependabot in #480
• Rework e2e tests by @ml-evs in #504
• CI build optimisations by @ml-evs in #497
• Fix file permissions issues by @ml-evs in #511
• Labspec 1D Raman compatibility by @elbee99 in #477
• Make sure cypress cloud runs are recorded by @ml-evs in #515
• Clarify refcode prefix docs and config by @ml-evs in #516
• Add note about GitHub callback URL in docs by @ml-evs in #517
• Bump to cypress 13 and remove flaky tests in CI by @ml-evs in #519
• Additionally e2e test Firefox & Chrome by default by @ml-evs in #521
• Improve GitHub OAuth connection by @ml-evs in #523
• Fix scope order for GitHub OAuth app by @ml-evs in #525
• Generate default SECRET_KEY from platform-specific info by @ml-evs in #527
• Make Flask session lifetime configurable by @ml-evs in #526
• Add ability to register and sign-in via email magic links by @ml-evs in #484
• Check and warn for missing secrets in server startup logs by @ml-evs in #529
• Miscellaneous tidying by @ml-evs in #532
• remove option to add a "Test Block" on the edit page by @jdbocarsly in #535
• Promote invoke to a real dependency rather than dev dependency by @ml-evs in #541
• Fix y_options bug when plotting 1D Raman by @ml-evs in #542
• Native support for automatic backups by @ml-evs in #467
• Add rotating file log handler and simplify streamed log messages by @ml-evs in #545
• Update backup docs by @ml-evs in #544
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webapp by @dependabot in #533
• Bump tinymce from 5.10.8 to 5.10.9 in /webapp by @dependabot in #508
• Update galvani to 0.3.0 by @ml-evs in #551
• Pin openai to 0.28 until we can migrate by @ml-evs in #555
• Bump aiohttp from 3.9.1 to 3.9.2 in /pydatalab by @dependabot in #559
• Add UI for adding an existing item to an existing collection by @ml-evs in #553
• Bump navani to support ECLab >= 11.50 by @ml-evs in #563
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #564
• Add spacing above logo by @jdbocarsly in #575
• Tweaks and bugfixes for collection assignment UI, badge styles and navbar by @jdbocarsly in #566
• Add drop shadow to badge hover by @ml-evs in #579
• Upgrade mdbtools per new galvani version by @ml-evs in #581
• Disable ORCID sign-in UI for now by @ml-evs in #580
• Prepare 0.3.0 release by @ml-evs in #583

New Contributors

• @elbee99 made their first contribution in #422
• @pre-commit-ci made their first contribution in #493

Full Changelog: v0.2.5...v0.3.0

v0.3.0-rc3

What's Changed

• Make sure cypress cloud runs are recorded by @ml-evs in #515
• Clarify refcode prefix docs and config by @ml-evs in #516
• Add note about GitHub callback URL in docs by @ml-evs in #517
• Bump to cypress 13 and remove flaky tests in CI by @ml-evs in #519
• Additionally e2e test Firefox & Chrome by default by @ml-evs in #521
• Improve GitHub OAuth connection by @ml-evs in #523
• Fix scope order for GitHub OAuth app by @ml-evs in #525
• Generate default SECRET_KEY from platform-specific info by @ml-evs in #527
• Make Flask session lifetime configurable by @ml-evs in #526
• Add ability to register and sign-in via email magic links by @ml-evs in #484
• Check and warn for missing secrets in server startup logs by @ml-evs in #529
• Miscellaneous tidying by @ml-evs in #532
• remove option to add a "Test Block" on the edit page by @jdbocarsly in #535
• Promote invoke to a real dependency rather than dev dependency by @ml-evs in #541
• Fix y_options bug when plotting 1D Raman by @ml-evs in #542
• Native support for automatic backups by @ml-evs in #467
• Add rotating file log handler and simplify streamed log messages by @ml-evs in #545
• Update backup docs by @ml-evs in #544
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webapp by @dependabot in #533
• Bump tinymce from 5.10.8 to 5.10.9 in /webapp by @dependabot in #508
• Update galvani to 0.3.0 by @ml-evs in #551
• Pin openai to 0.28 until we can migrate by @ml-evs in #555
• Bump aiohttp from 3.9.1 to 3.9.2 in /pydatalab by @dependabot in #559
• Add UI for adding an existing item to an existing collection by @ml-evs in #553
• Bump navani to support ECLab <=11.50 by @ml-evs in #563

Full Changelog: v0.3.0-rc2...v0.3.0-rc3

v0.3.0-rc2

This is a pre-release of the upcoming v0.3.0.

There are few outstanding issues before we are comfortable releasing, primarily that we have some spurious test failures that occur only in our CI. We estimate these will be resolved by the end of the year (given current leave schedules of certain core developers...) In the meantime please use this pre-release for deployments and let us know if there are any issues.

What's Changed

• Add concept of user manager permissions by @ml-evs in #417
• Add UI for collections, fix local graphs and improve tables by @ml-evs in #404
• Adjust default echem subsampling by @ml-evs in #426
• Version bumps and linting tweaks by @ml-evs in #429
• Remote filesystem scanning improvements by @ml-evs in #430
• Implemented Raman block by @elbee99 in #422
• Move echem block to its own app submodule by @ml-evs in #433
• Switch from flake8+isort to ruff by @ml-evs in #434
• Update vue/cli to v5 by @ml-evs in #432
• Tweak to remote file system caching for improved page loads by @ml-evs in #435
• Enable Arbin res file parsing by adding mdbtools/0.7.1 build to server Dockerfile by @ml-evs in #436
• Better sanitize inputs for plot cycle selector by @ml-evs in #438
• Add sample/user/cell counters to deployment about page by @ml-evs in #383
• Fix last modified timezone and make value dynamic by @ml-evs in #439
• Restructure NMR block module and add tests by @ml-evs in #441
• Add basic permissions to files by @ml-evs in #445
• Fix broken README link and update affiliations by @ml-evs in #448
• Manual overhaul of API docs by @ml-evs in #451
• Fix bug where user cannot be created with blank display name by @ml-evs in #456
• Bump navani and try to accomodate other deps by @ml-evs in #457
• Update INSTALL.md by @jdbocarsly in #458
• Add subtitled intro video to README by @ml-evs in #459
• Bump @babel/traverse from 7.22.10 to 7.23.2 in /webapp by @dependabot in #463
• Bump apollo-server-core from 3.12.0 to 3.12.1 in /webapp by @dependabot in #446
• Bump tinymce from 5.10.7 to 5.10.8 in /webapp by @dependabot in #465
• Support .wdf files for 1D Raman by @elbee99 in #466
• Add more details to README by @ml-evs in #479
• Adding limits on file uploads by @ml-evs in #475
• Improve login/logout UI to support multiple authentication mechanisms by @ml-evs in #384
• Remove unecessary scope for GitHub OAuth by @ml-evs in #483
• Tweak to deployed documentation by @ml-evs in #485
• Revamp installation docs with additional configuration/administration/deployment info by @ml-evs in #490
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #493
• Bump browserify-sign from 4.2.1 to 4.2.2 in /webapp by @dependabot in #480
• Rework e2e tests by @ml-evs in #504
• CI build optimisations by @ml-evs in #497
• Fix file permissions issues by @ml-evs in #511
• Labspec 1D Raman compatibility by @elbee99 in #477

New Contributors

• @elbee99 made their first contribution in #422
• @pre-commit-ci made their first contribution in #493

Full Changelog: v0.2.5...v0.3.0-rc2

v0.2.5

What's Changed

This release is primarily to trigger Zenodo archival, but also contains some improvements to the chat block and the initial API support for collections of entries.

• Add collections type to API by @ml-evs in #403
• make website title setable by environment variable by @jdbocarsly in #408
• NMR block process number bugfix by @jdbocarsly in #407
• Chat improvements by @jdbocarsly in #414
• Add CITATION.cff file for repository by @ml-evs in #416

Full Changelog: v0.2.4...v0.2.5

v0.2.4

What's Changed

This release introduces a change that should speed up page loads on most datalab pages considerably. The edit page now loads quickly, while blocks with slower-loading large data files finish load in parallel and come in as they are ready.

• Load blocks asynchronously by @jdbocarsly in #401

Full Changelog: v0.2.3...v0.2.4

v0.2.3

What's Changed

• Fix XRD plot wavelength interaction by @ml-evs in #389
• Block loading fixes by @jdbocarsly in #390
• Add release action to automatically bump version number by @ml-evs in #392
• Fix/slim down chatblock context by @jdbocarsly in #394

Full Changelog: v0.2.2...v0.2.3