Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

framecache writer yaml output #62

Open
donald-e-boyce opened this issue Mar 28, 2018 · 3 comments
Open

framecache writer yaml output #62

donald-e-boyce opened this issue Mar 28, 2018 · 3 comments
Assignees

Comments

@donald-e-boyce
Copy link
Collaborator

The framecache writer was recently updated to write a single npz file as well as the older yaml files. However, the option to write yaml was not included in the standard save.write() interface. This is easily corrected, but unit tests should updated appropritately and new ones should be developed for the npz writer.

@donald-e-boyce donald-e-boyce self-assigned this Mar 28, 2018
@donald-e-boyce donald-e-boyce mentioned this issue Mar 28, 2018
@donald-e-boyce
Copy link
Collaborator Author

The current framecache writer uses a keyword arg to write yaml, but in standard usage the writer doesn't called directly. It gets called through the save.write() function.

def write(self, output_yaml=False):

Instead, the save.write() function saves the keyword args, which are now accessible via the options property (new, formerly _opts).

        if self.options.get('output_yaml', False):
            self._write_yml()

@donald-e-boyce
Copy link
Collaborator Author

At this point, I have fixed the unit tests for the yaml-based frame cache and I have added new tests for npz-based frame cache. However, the npz-based frame cache has issues handling general metadata. It can save data of various python types to the array cache, but it converts them to a numpy array with a special dtype. When loading the file, there is no obvious way to recover the original type of the metadata.

In any case, the npz-based frame cache will need to be revisited if we want to use more general metadata. For now, it works fine with numpy arrays as metadata, and we could just merge in the changes and move on.

Any thoughts?

@joelvbernier
Copy link
Owner

The dtypes in the numpy arrays preserve the original data type (e.g. string vs int vs float), don't they?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants