Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: ssh/scp data for tcbsd PLCs #25

Merged
merged 17 commits into from
Mar 26, 2024
Merged

Conversation

ZLLentz
Copy link
Member

@ZLLentz ZLLentz commented Dec 18, 2023

Description

Big idea: we can upload files to and download files from the TcBSD PLCs using scp instead of ftp, which is more modern and secure and is enabled by default on these. The old ftp support stays for backwards compatibility, because the old PLCs cannot use ssh/scp.

Main changes:

  • ssh_data has been added with a similar API to ftp_data, but it uses ssh/scp as the transport mechanism.
  • plc_data has been added, which imports from ftp_data and ssh_data and picks the method to use. It does this by trying ssh, and then trying ftp if ssh fails, and caching which one succeeded for next time.
  • data_types has been added, which currently generalizes the file info from each data source so that functions that may expect either can do type checking. In the future this may also have structs to help give us more detailed pmps parameter comparisons. This also required me to rename created_time -> modified_time to more accurately support both ssh and ftp at the same time.
  • The cli and gui modules will import from plc_data instead of from ftp_data and are otherwise unchanged.
  • Relocate helper functions without direct relations to ftp from ftp_data to plc_data
  • Add fabric as a runtime dependency for the ssh/scp data transfer.

Other changes:

  • Remove PyQt5 from the ci testing extras in favor of the more general dev-requirements.txt file. This is the new place we're keeping the requirements, so that they are noticed when checked outside of the CI builds.
  • Create a conda recipe for the repo to use for the CI builds.
  • Make the debug logger show timestamps, make the non-debug logger ignore paramiko spam
  • General cleanup of docstrings and type annotations
  • Decrease data timeouts from 2s to 1s to help with long chain timeouts, especially in the gui
  • Add more test PLCs to the tst config
  • Remove setuptools-scm as a runtime dependency because it is not used at runtime.

Some remaining sloppiness:

  • There isn't 1:1 parity between the data structures in the ftp module vs the data structures in the ssh module. Some of this is intrinsic to the difference between the protocols, but some of it is simply messy and it could be cleaned up. This leads to some unsightly bifurcations elsewhere in the code. (resolved: make them more similar and clean up)
  • I made a data_types module last year (?) and then didn't use it anywhere. I think I was intending to augment the comparison operations but I never implemented anything further. I should probably delete it or banish it to its own branch. (resolved: remove unused elements from data_types, keep the ones useful for comparing files between PLCs)
  • The CI fails because the standard CI expects there to be an automatic docs build, but these are not present in this repo. (unresolved: make the CI not require the docs build and handle this later)

Motivation and Context

See above, but largely: scp is more secure and is enabled by default on the new PLCs. The rest of the code doesn't have to change very much to accommodate new data transfer methodologies.

Related to:
pcdshub/lcls-twincat-motion#215
pcdshub/twincat-bsd-ansible#19

How Has This Been Tested?

Interactively only

Where Has This Been Documented?

Setup for the ecs-user account is documented on this page: https://confluence.slac.stanford.edu/display/PCDS/TcBSD+PLC+Setup

Pre-merge checklist

  • Code works interactively
  • Code contains descriptive docstrings, including context and API
  • Pre-commit passes on GitHub Actions

@ZLLentz
Copy link
Member Author

ZLLentz commented Mar 22, 2024

I think I should clean up some of the messiness before asking for reviewers on this one

@ZLLentz
Copy link
Member Author

ZLLentz commented Mar 25, 2024

For now, I've made the docs build optional. I'll handle making a real docs build at some later date.
Once I check my confluence documentation, I'll mark this as ready for review and ask for reviewers.

@ZLLentz ZLLentz changed the title WIP: ssh/scp data for tcbsd PLCs ENH: ssh/scp data for tcbsd PLCs Mar 25, 2024
@ZLLentz ZLLentz marked this pull request as ready for review March 25, 2024 20:29
nrwslac
nrwslac previously approved these changes Mar 25, 2024
Copy link

@nrwslac nrwslac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read through the changes, lgtm!

@ZLLentz
Copy link
Member Author

ZLLentz commented Mar 26, 2024

I'm going to fix some subtle copy/paste issues I see in the docstrings and merge later today (giving some chance if others want to/have time to peek at it)

Copy link

@nrwslac nrwslac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫡

Copy link

@NSLentz NSLentz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Zach confirmed that this has been tested on the old PLCs as well as the new ones.

@ZLLentz ZLLentz merged commit 25a593d into pcdshub:master Mar 26, 2024
8 of 9 checks passed
@ZLLentz ZLLentz deleted the enh_ssh_data branch March 26, 2024 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants