Merge develop into master #89
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tems/DGP into feature/ui-development Merging upstream changes into local ui-development branch.
Fixed Flight.eotvos property to correctly call and calculate eotvos correction, and to return a DataFrame indexed by the input GPS index. Eotvos is then plotted on the 3rd horizontal plot in the main window.
Added file preview dialog for importing GPS/Gravity data, and ability to switch column data headers as needed using a drop-down combo box.
Added contribution guidelines in contribution.rst. Rewrote the README to provide more info for users and contributors.
Added contribution guidelines in contribution.rst. Rewrote the README to provide more info for users and contributors.
* Fixed and tested Eotvos function. Tests are successful on the limited set of data, need to test fully on a larger dataset and results generated using Daniel's MATLAB routines. * CLN: Code refactoring and clean up in eotvos code. * TST: Update test_eotvos and tested locally with full data file. Renamed test files for test_eotvos. Eotvos function was tested locally against a data/result set generated with MATLAB (~200k lines), a smaller sample will be generated and uploaded later as the full dataset is unwieldly to run unittests against, and the file size is inconveniently large. ENH: Debugging implementation of eotvos to plot GPS CLN: Add specific function to plot 3 main graphs TST: Added basic test to flight for gps and eotvos. Added testing to test_project to test flight import and processing of eotvos data via .eotvos property. Briefly tested eotvos plotting functionality with gps data imported. Exception handling needs to be added to deal with bad file inputs. ENH: Add Eotvos plotting to main flight plots. Fixed Flight.eotvos property to correctly call and calculate eotvos correction, and to return a DataFrame indexed by the input GPS index. Eotvos is then plotted on the 3rd horizontal plot in the main window.
Work in Progress: model base for project tree view rewritten to allow fine grained control over display and retrieval of data. Also integrating model into the base object representation, allowing for dynamic editing of the underlying data structure via the model. TST: Fix tests due to project model updates ENH: Add type-checking to Container class in project. Fix dependent function calls after implementing new ProjectModel model. ENH/CLN: Change flight lines storage, improve TreeView Changed FlightLine storage to use a Container in the Flight class. Moved gen_uuid from Flight class to dgp.lib.etc as a general function that can generate a uuid4 with an optional prefix (keeping the resultant string at len == 32). BUG: Bugfixes in new code to merge back into gui-development. Fix various references and bugs due to new View/Models code, and side effects in project.py. Tree Model should now have equivalent functionality to original implementation, but with the ability to dynamically add/remove objects from their various containers without the need to completely regenerate the view model. BUG: Fix issue plotting datetime index in new plot method
Merge Import Preview dialog feature into current tip of ui-development.
…ot area. - Extents of lines can be changed by stretching from either of the sides and by dragging the box for the line. TO DO: - Finish right-click context menu for removing patches and setting labels.
Line 488 in main.py was passing flight.uid to main.py::_new_plot_widget instead of the flight object, causing a runtime error when creating a new flight via the add_flight_dialog method. This has been changed to pass the flight object as an attribute instead of its uid.
Logic added in models.py to properly handle addition of children and sub-children to the project model. There are still some minor issues to be worked out, i.e. properly displaying the lines in a human-readable way. Also still need to re-draw all lines on the plots when loading a project.
Flight Line selection rectangles are now displayed and drawn when a project is loaded/application launched based on saved Flight Lines. TODO: Code Cleanup in Plotter.py, incorporate new draw_patch function into onclick method to reduce code duplication and outsource patch creation to single function.
Major GUI improvements including: - [ ] Implemented point-and-click line selection on plots - [ ] Implemented GPS data import to flight - [ ] Added a toolbar with common actions - [ ] Added a preview dialog for data import - [ ] Added Eötvös plots to main flight plots - [ ] Various project model improvements - [ ] Added decimation feature for plotting
* CLN/ENH: Code Cleanup and resolution of TODO's in various project files. Cleaned up old unused code, refactored some methods e.g. plot2 -> plot_series, some small adjustments to Container class. ENH: Added logic to remove FlightLines when new gravity data is imported. * ENH: Partial implementation of Channel selection Channels (for Gravity data) can be selected by dragging and dropping them beneath the respective plot titles in the Channel list. TODO: Implement ability to remove channels from the plot. TODO: Add context menu allowing user to clear all channels from specified plot. TODO: Replot channels based on the state saved in the Flight objects upon load, instead of the default (if there is saved state). * ENH: Prereq changes to project structure for channel selection. Updates to plotter and flight classes to facilitate addition/removal and update of data channels in future via graphical interface. Flight now stores plot state - that is, which of its available data channels are plotted on which axes, to be restored on project load. * CLN: Cleanup deprecated project/plot code. * ENH: Add data channel selection capability Ability to add/remove data channels to and from different plots in workspace. TODO: Performance optimization/fix for data loading in the Flight class. TODO: Enable adding of calculated data series i.e. Eotvos * ENH/FIX: Optimized data loading, fixed bugs in channel selection. ENH: Optimized project.py::Flight::get_channel_data function to cache DataFrame on first load, to reduce disk IO operations. FIX: Issue when moving a channel from one plot to another, this now works correctly. FIX: Bug in plotter code when all channels are removed from all plots, as the DateFormatter is unable to handle this situation. FIX: Line Selection removal in Project Tree, rewrote most of the ProjectModel::remove_child code to properly handle removal of items from the model.
A right-click context menu allows the user to set a label associated with any flight line.
ENH: Added label functionality for flight lines
…hen a label was not set.
…-attr BUG: Fixed several bugs in patch movement with no labels
* ENH/TST: New TreeItem Class design and tests. Added new TreeItem and interface in dgp/lib/types.py. The Abstract base AbstractTreeItem is designed to allow polymorphic use of classes derived from it, providing the required methods for use in a QT TreeView or similar model. The TreeItem implements AbstractTreeItem, providing default implementations for the functions defined in the Abstract class. The goal of this branch is to remove dependence on the ProjectItem class in dgp/gui/models.py, which currently provides a wrapper for objects added to the ProjectModel. The idea is then that any class/object to be displayed in the GUI via a QT model can inherit from TreeItem, and minimally implement the data() method. A higher level of control can be implemented by the inherting class over its display in the UI by overriding or extending methods from TreeItem. * CLN: Reformat models/project/types modules to 80 char lines. Reformatted project files to conform better with PEP8 80 char line widths. Removed old comments/general code cleanup. * ENH/CLN: Rewrite of ProjectModel and supporting code ENH: This commit rewrites a large portion of the ProjectModel and TreeItem classes, designed to improve the usability and reduce the complexity introduced by using a wrapper class previously (ProjectItem). The TreeItem class is designed to be inherited from for objects that need to be displayed in a GUI TreeView. TreeItem defines the default implementation sufficient for most objects to immediately be useable in the ProjectModel, only the data() method must be overriden to tell the model what to display for various roles. TST: A basic test suite has been added to test the functionality of the new TreeItem class. More rigorous tests should be added in the future. CLN: Various project files have also been cleaned up to conform to a 80 character line width standard. Deprecated code and comments have also been removed in various files. * FIX: Compatibility with Py3.5 change IntFlag to IntEnum. * ENH/FIX: Enhanced new model code after testing. Fixed bugs and enhanced new model code after further UI testing. Added ability to specify styles for individual TreeItems (icon, bg/fg color). Enhanced update code for adding/removing/modifying items in Tree Model - no longer need to manually specify update when adding/removing. If modifying an attribute, a setter property should be used to call super().update() if the attribute affects the visual appearance of an item. Main window UI cleanup. Adjusted margins of some widgets/panels for cleaner look.
Resolves #50 * ENH: Added tab functionality for Flights. New tab functionality: Double click flight or create new flight to display in its own tab. Tabs have sub-tabs for future use displaying maps, transformations etc. TODO: Implement close button on tabs, decide whether to destroy tab, or hide to show again later. * ENH: Rework channel std model into custom model. Design new model based on QAbstractModel for Channel data selection feature. New model enables customization of drag and drop, and simplifies plotting updates. Modified types.py and created new BaseTreeItem class that provides implementation of AbstractTreeItem, without adding any additional features like TreeItem. This allows the developer to create specialized Tree Item classes with a default implementation for the basic requirements, without including unnecesary features. * ENH/TST: Added datamanager module as global data interface. Add datamanager module, to act as a global interface to import/export/load/save data within the project. The datamanager is initialized once and then can be used by any other module in the project by calling datamanager.get_manager(). This simplifies the retrieval of data from various locations in the code, as each caller must only know the UID and type of data they need to retrieve, as opposed to storing references to the project to utilize its load method. TST: Basic test suite created for datamanager - more comprehensive tests need to be added as functionality is added to the module - plan to support the storing of metadata and other formats such as JSON, CSV. * ENH: Various enhancements to Tab UI model. A sprawling commit that lays the foundation for the UI as we proceed. UI Tabs are now mostly functional, each Flight is displayed in its own tab, and the contextual tree model (below the project tree) is automatically switched/updated as the user selects between tabs and sub-tabs. This commit and the preceeding commit also introduce the datamanager module, which was necessary at this point to reduce the complexity of calls between the Flights/Tabs and Project. Code in various files has been updated to utilize the new datamanager for the import/retrieval of DataFrames. * FIX: Fix handling of relative/absolute paths. Fix erroneous behavior of datamanager if project directory is moved. Data manager now constructs paths to data files based on its initialization path. * CLN/ENH: Code-cleanup and reintroduced state into Flight Tabs. General code cleanup Re-implemented state save/load into Flight tab code, so that the plot tab will remember and re-draw plot/lines when opened. * ENH/CLN: Improved datamanager and JSON registry. Improvements and tests added for basic functionality of DataManager and JSON registry component - ability to store/retrieve HDF5 files from module level instance. Added ability to close tabs in tab interface. Cleaned up and documented code in various files. TODO - should tab widgets be kept in memory for quicker redraw if reopened?
* ENH/FIX: ENhancements/Fixes prior to transform implementation. Enh: Implement twin Y axis scales on plots. Enh: Improve Plotter sub-plot generation code. Allow the layout to be re-initialized with different parameters. Enh: Add prototype interface for Transform functionality on Flight sub-tab. Fix: Fixed behavior of ChannelListModel when it is used in multiple instances - i.e. on the main plot, and the transform plot. The DataSource class was modified to return a unique set of DataChannels, instead of creating a single instance of each channel, which could then be mutated by multiple models - leading to unexpected behavior. * ENH: Re-write plotter.py, speed improvements, modularity. ENH: Large re-write of the plotter.py module, including most of LineGrabPlot class. Added AxesGroup and PatchGroup classes to conceptually group related Axes and Patches (Flight Lines) to ease the process of doing batch operations on the plots. * ENH/FIX: Updates and fixes to plotter re-write. ENH: Implemented contextual cursors that change dependant on mouse location on the plot (e.g. drag handles at edge of flight lines) FIX: Fixed/improved proximity calculations by using percentage system which scales the proximity values based on the X-limits of the axes. DOC: Added more documentation to new classes/methods. * ENH/FIX: Fixed plot scaling issues, implement re-sampling. Fixed plot scaling issues e.g. when user plotted a data line while zoomed in on an Axes, using the default Home button on the MPL toolbar would create undesired results. Re-implemented plot resampling method to up/down-sample data based on zoom level for some performance gains in the interactive plot. * FIX: Fixed no len() method in LineGrabPlot Fix bug when creating new flight due to code modification in the BasePlottingCanvas class. __len__ method has now been added to the LineGrabPlot. * FIX: Patch and label drawing issues. Fixed patch not drawing after label is set until mouse movement. Fixed text display when label is removed to display empty string. Fixed label position not updating/drawing on zoom in axes where zoom occured.
Simplified TreeItem and removed some duplicate/unneeded class parameters. Implemented new get_child method to search for child by UID, instead of allowing child() to accept either an Int index or str UID.
Cleaned up new Project Creation Dialog, fixed high-resolution issues due to fixed/static values. Minor spacing/resoulution fixes for add_flight_dialog and splash_screen
Usability improvement in splash.py - enable double click to select and open project. Redesign advanced_data_import ui and backend code, better import preview handling, and more information displayed. Preview/edit area is now separate pop-out dialog from import window. main.py updated to support new import functionality.
Major refactoring of advanced_data_import and it's import preview functionality. Propogated use of BaseDialog to further custom dialogs, and improved BaseDialog message and error facilities. Child classes can now easily display a message in QLabel, or display an error via an error Pop-Up. Also removed hacky logging hooks from BaseDialog, creating purposeful functions instead to handle messages/errors. Cleaned up code in CreateProject dialog, better invalid/empty input detection. Simplified accept handling and moved accept signal definition from code to .ui file. Began introduction of enums for defined type comparison, used to specify ProjectType, DataType etc. Replaced TableModel with TableModel2 (to be renamed and original deleted in future), with more sensible and less hacky handling of table data. Table accepts 2D List array - and possibly Numpy ndarray. In future expand capability to ingest pandas DataFrame as well.
Fix documentation build errors due to refactored sources
Re-order splash screen finish trigger and main window visibility so that the splash screen correctly disappears when the UI controls have loaded.
…tion Datafile uid property will now return None when there is no underlying datafile (i.e. we have a placeholder controller). Project model signal signature changed to permit None values as uid. main.py _tab_open_requested updated to ignore requests where UID is None. This may (should) be changed in future to remove the confusion of having 'empty' DataFileControllers. Perhaps by introducing a new type of QStandardItem subclass to be used as a placeholder, which can be replaced by a complete DFC when the data has been loaded. Fix invalid attempt to construct a pathlib.Path object from a None value in settings.py. Will now correctly check that the retrieved path is not None before constructing a Path object, which otherwise results in a TypeError.
Unsure of the root cause of this issue, but it occurs on Linux/Windows/OSX all the same. Using the toggleViewAction() method of QDockWidget to construct the QAction seems to fix the issue where the DockWidget is hidden after the Main Window has been minimized then restored.
…ents Feature/ui improvements
Refactor/rename IBaseController -> AbstractController. Add concrete methods to AbstractController to handle reference tracking, updates, deletes. Simplify the Interface hierarchy and delete IParent/IChild/DGPObject Functionality of above is rolled into AbstractController as these features are essentially consistent across all UI controllers.
Improved the observer registration method to enable passing of bound methods, and observation of specific state actions (update/delete) This allows UI objects (tabs/widgets etc) to explicitly register and subscribe to controller events - e.g. when a controller object is deleted, it will execute a callback notifying all observers of its destruction. Added clone tracking feature to AbstractController to standardize the update of clones, and integrate the state notifications with those of the observers.
Rename 'datamodel' property to 'entity' (AttributeProxy) Implement uid property in AbstractController Set common QStandardItem attributes in AbstractController Add project attribute to AbstractController as weakref Refactor/cleanup imports in controllers
Updated workspace base tabs (WorkspaceTab/SubTab) to provide base initializer which takes weak-reference of supplied AbstractController and registers itself as an observer in order to close when the controller is deleted. Fixed error in workspace_widget where invalid index (None) was passed if user attempted to close tab via shortcut (Ctrl+W) when no tabs were opened. Removed model tabCloseRequested signal as it is no longer needed with new observer model. Fix issues in plot segment signalling with observers
Refactored DataSetController to use the standard child interface defined by AbstractController for managing its DataSegment's. Child DataFiles are still managed as a special case, but this removes some unnecessary complication with two sets of similar methods.
Add documentation to AbstractController and other various changes. Fix breaking tests due to API changes/refactoring.
Updated and added to documentation for various dgp/core and dgp/gui packages/modules. Improved docstrings in controller_interfaces for AbstractController Add workspaces documentation for GUI workspace tabs.
All controllers now maintain a weakref to the project which owns them, this helps to simplify some actions and reduce chained get_parent calls when child objects need to call a project function or attribute.
Add new tests to verify observer behavior and clone updates. Fix tests due to AbstractController refactoring
There are round-off errors in the pandas to_timedelta function. This interim fix until, which will remain until the issue gets addressed in the pandas code base, rounds datetimes to the nearest 1000 ns. This only works for times up to a precision of about 10 us.
…ng_issue Fix for timestamp round-off issue in convert_gps_time
Many GUI actions/methods to activate them are not easily tested, and would only be minorly useful to have tested anyways. pragma: no cover annotations have been added and will be added in future to these sorts of blocks/methods as I believe it is a waste of time to attempt to test many of them.
Virtual more correctly expresses the implementation of the base controller vs Abstract in terms of OOP principles. As the BaseController now provides some implementations for common methods.
Merge feature branch adding observer/subscriber interface for GUI objects depdendent on project controller objects.
DanielAliod
approved these changes
Jul 20, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge latest develop into master to bring up to most recent state