Setting DB data types for text to fix bug occurring when a previous d…

…atabase import had less data; Grafana dashboards and screenshots; README update; other small changes/fixes
Petrichor-Labs · Jul 8, 2021 · d9b9027 · d9b9027
1 parent d51cb12
commit d9b9027
Show file tree

Hide file tree

Showing 23 changed files with 42,991 additions and 32 deletions.
diff --git a/README.md b/README.md
@@ -1,31 +1,73 @@
 This library makes use of [pynmea2](https://github.com/Knio/pynmea2) to parse through input NMEA 0183 data, organize it, and output it to CSV files or to a PostgreSQL database.
 
+## Table of Contents
+1. [Terminology](#terminology)
+1. [Setup](#setup)
+1. [Usage](#Usage)
+1. [Examples](#Examples)
+1. [Examples](#Examples)
+1. [Development Notes/Oddities](#Development Notes/Oddities)
+1. [Helpful References](#Helpful%20References)
+1. [Grafana](#Grafana)
+1. [Discussion](#Discussion)
+
+
+## Support
+If you find this tool useful, please consider supporting its development and the development of other tools like it. You can support us financially using the `Sponsor` button at the top of the [GitHub page](https://github.com/Petrichor-Labs/nmea_data_convert).
+
 ## Terminology
 * **`sentence`**:
-A line from your data file from a particular `talker` and of a particular `sentence_type` E.g.:
-`$GNRMC,,V,,,,,,,,,,N*4D`
-`$GNGGA,045824.00,3944.54025,N,10511.64604,W,1,03,4.93,1784.2,M,-21.5,M,,*49`
+A line from your GNSS module/NMEA data file from a particular `talker` and of a particular `sentence type` E.g.:
+  - `$GNRMC,,V,,,,,,,,,,N*4D`
+  - `$GNGGA,045824.00,3944.54025,N,10511.64604,W,1,03,4.93,1784.2,M,-21.5,M,,*49`
 
 * **`talker`**:
 The type of the transmitting unit. For the purposes of satellite navigation, this is the constellation from which data is being received.
-E.g.: `GA`: Galileo Positioning System; `GB`: BDS (BeiDou System); `GL`: GLONASS Receiver; `GN`: Global Navigation Satellite System (GNSS); `GP`: Global Positioning System (GPS)
-See: https://gpsd.gitlab.io/gpsd/NMEA.html#_talker_ids, or https://www.nmea.org/Assets/20190303%20nmea%200183%20talker%20identifier%20mnemonics.pdf
-
-* **`sentence_type`**:
+E.g.:
+  - `GA`: Galileo Positioning System
+  - `GB`: BDS (BeiDou System)
+  - `GL`: GLONASS Receiver
+  - `GN`: Global Navigation Satellite System (GNSS)
+  - `GP`: Global Positioning System (GPS)
+
+  See: https://gpsd.gitlab.io/gpsd/NMEA.html#_talker_ids, or https://www.nmea.org/Assets/20190303%20nmea%200183%20talker%20identifier%20mnemonics.pdf
+
+* **`sentence type`**:
 One of several types of NMEA sentences that can be received from the talker.
-E.g.: `RMC`, `VTG`, `GGA`, `GSA`, `GSV`, `GLL`
-See: https://gpsd.gitlab.io/gpsd/NMEA.html#_nmea_standard_sentences
+E.g.:
+  - `RMC`
+  - `VTG`
+  - `GGA`
+  - `GSA`
+  - `GSV`
+  - `GLL`
+
+  See: https://gpsd.gitlab.io/gpsd/NMEA.html#_nmea_standard_sentences
+
+* **`sentence cycle`**:
+A set of sentences output by a GNSS module on each iteration. For example, a module operating on a 1Hz cycle may output a set of sentences at the top of every second that looks like this:
+  ```
+  $GNRMC,045832.00,A,3944.53039,N,10511.64401,W,0.875,,221220,,,A*7A
+  $GNVTG,,T,,M,0.875,N,1.621,K,A*33
+  $GNGGA,045832.00,3944.53039,N,10511.64401,W,1,07,1.55,1785.9,M,-21.5,M,,*42
+  $GNGSA,A,3,19,02,14,28,24,06,,,,,,,2.77,1.55,2.30*1B
+  $GNGSA,A,3,73,,,,,,,,,,,,2.77,1.55,2.30*1A
+  $GPGSV,2,1,06,02,33,205,20,06,68,161,23,14,26,115,23,19,70,010,15*7B
+  $GPGSV,2,2,06,24,42,282,26,28,37,109,22*75
+  $GLGSV,1,1,03,73,85,280,31,74,33,326,,,,,31*61
+  $GNGLL,3944.53039,N,10511.64401,W,045832.00,A,A*68
+  ```
 
 ## Setup
 
 Input data files can contain either sentences having all the same `talker`+`sentence_type`, like that of `test_data/test_data_0_GLGSV.nmea`, `test_data_0_GNGGA.nmea`, etc., or cycles of sentences like that of `test_data/test_data_0_all.nmea`. Your input file should have a format similiar to those under `test_data`.
 
 To have your data datetime stamped, it must be in a format like that of `test_data/test_data_0_all.nmea`, with RMC sentences containing date and time stamps proceeding other sentences in the same cycle. 
 
-Useage of the `cycle_start` (`cs`), `num_sentences_per_cycle` (`spc`), and `backfill_datetimes` (`bfdt`) parameters will depend on the format of your data, and some combination of them is required. See below for examples. See the Usage section for explanations of the parameters.
+Usage of the `cycle_start` (`cs`), `num_sentences_per_cycle` (`spc`), and `backfill_datetimes` (`bfdt`) parameters will depend on the format of your data, and some combination of them is required. See below for examples. See the Usage section for explanations of the parameters.
 
 
-If working with a database, the database access information/credentials must be setup in `db_creds.py`.
+To import data to a Postgres database, have the database server installed and running where desired. The database access information/credentials must be setup in `db_creds.py`.
 
 
 ## Usage
@@ -42,15 +84,15 @@ positional arguments:
 
 optional arguments:
   -h, --help            show this help message and exit
-  --cycle_start CYCLE_START, --cs CYCLE_START
+  --cycle_start CYCLE_START, -cs CYCLE_START
                         talker+sentence_type, e.g. 'GNRMC'; used to key off of for sentence merging, and more; must appear once and only once in each cycle, and must be at the beginning of each cycle; must contain date and time information for sentences to be datetime
                         stamped
-  --num_sentences_per_cycle NUM_SENTENCES_PER_CYCLE, --spc NUM_SENTENCES_PER_CYCLE
+  --num_sentences_per_cycle NUM_SENTENCES_PER_CYCLE, -spc NUM_SENTENCES_PER_CYCLE
                         If the cycle_start argument is not provided, and sentences are not all of type GSV, cycles will be inferred from this argument. Every num_sentences_per_cycle will be given the same cycle_id starting with the first sentence. Sentence merging is
                         based on cycle_id.
-  --backfill_datetimes, --bfdt
+  --backfill_datetimes, -bfdt
                         backfill datetimes where missing by extrapolating from messages with datetime information
-  --drop_previous_db_tables, --dropt
+  --drop_previous_db_tables, -dropt
                         drop all previous DB tables before importing new data; only applies when output_method is 'db' or 'both'
 ```
 ## Examples
@@ -59,7 +101,7 @@ Output cycles of NMEA sentences to CSV files using GNRMC sentences as the cycle
 ```
 $ ls -l *.csv
 ls: *.csv: No such file or directory
-$ python nmea_data_convert.py test_data/test_data_0_all.nmea csv --cs GNRMC
+$ python nmea_data_convert.py test_data/test_data_0_all.nmea csv -cs GNRMC
 
 Reading in data... done.
 
@@ -91,7 +133,7 @@ MacBook-Pro-4:nmea_data_convert Thomas$ ls -l *.csv
 ### Example 2
 Output cycles of NMEA sentences to both CSV files and database using GNRMC sentences as the cycle start, backfill datetimes, and drop previous tables from database:
 ```
-$ python nmea_data_convert.py test_data/test_data_0_all.nmea both --bfdt --dropt --cs GNRMC
+$ python nmea_data_convert.py test_data/test_data_0_all.nmea both -bfdt -dropt -cs GNRMC
 
 Reading in data... done.
 
@@ -131,7 +173,7 @@ All done. Exiting.
 ### Example 3
 Convert sentences, all of the same `talker`+`sentence_type`, to database:
 ```
-$ python nmea_data_convert.py test_data/test_data_0_GNVTG.nmea db --spc 1
+$ python nmea_data_convert.py test_data/test_data_0_GNVTG.nmea db -spc 1
 
 Reading in data... done.
 
@@ -155,10 +197,21 @@ $ python nmea_data_convert.py test_data/test_data_0_GPGSV.nmea db
 ### Example 5
 Convert GSA sentences, all of the same `talker`, to database, where each sentence is part of a cycle containing two GSA sentences. Cycles may contain a GSA sentence for each constellation (see `test_data/test_data_0_GNGSA.nmea`:
 ```
-$ python nmea_data_convert.py test_data/test_data_0_GNGSA.nmea db --spc 2
+$ python nmea_data_convert.py test_data/test_data_0_GNGSA.nmea db -spc 2
 [output excluded for brevity]
 ```
 
+## Development Notes/Oddities
+- Sentence cycles contain a variable number of GPGSV sentences. A single GPGSV sentence contains information for 0-4 GPS satellites. A GNSS receiver could see up to 16 GPS satellites (https://stackoverflow.com/a/16415858/2909854, that may apply only on the ground?), resulting in up to four GPGSV sentences in a single cycle, e.g.:
+  ```
+  ...
+  $GPGSV,4,1,13,08,14,320,33,10,57,295,34,13,01,038,,15,26,049,35*7B
+  $GPGSV,4,2,13,18,60,119,41,23,68,019,34,24,29,097,35,27,36,294,27*71
+  $GPGSV,4,3,13,29,01,170,,32,31,211,34,46,38,215,36,48,40,210,37*73
+  $GPGSV,4,4,13,51,44,184,40*46
+  ...
+  ```
+  Columns used in the dataframes, CSV output, and database output are mostly determined by what is returned from pynmea2. For GPGSV sentences, we use `expand_GSV_fields()` to always create columns for up to 16 satellites in view. This way, the correct columns will exist in the appropriate database table and can be used to hold data from different and varying NMEA data sets. In the future, to make the database handling more dynamic, we could move to using something like Binary JSON instead.
 
 ## Helpful References
 * https://github.com/Knio/pynmea2/blob/master/README.md
@@ -169,8 +222,24 @@ $ python nmea_data_convert.py test_data/test_data_0_GNGSA.nmea db --spc 2
 * Glossary : https://www.unavco.org/help/glossary/glossary.html
 * https://gpsd.gitlab.io/gpsd/NMEA.html#_nmea_standard_sentences
 
-## Support
-If you find this tool useful, please consider supporting development of this tool and other tools like it. You can do so using the `Sponsor` button at the top of the [GitHub page](https://github.com/Petrichor-Labs/nmea_data_convert).
+## Grafana
+One of the great advantages of importing this data into a database is that you can use a system like [Grafana](https://github.com/grafana/grafana) to visualize it. Below are screenshots of the included Grafana dashboards showing the data parsed from `test_data/test_data_4.nmea` using this tool. These dashboards have been found to work in Grafana v7.5.7, but not in v8.0.4.
+
+![](grafana_screenshots/210705_GNRMC.png)
+
+![](grafana_screenshots/210705_GNVTG.png)
+
+![](grafana_screenshots/210705_GNGGA.png)
+
+![](grafana_screenshots/210705_GNGSA.png)
+
+![](grafana_screenshots/210705_GPGLGSV.png)
+
+![](grafana_screenshots/210705_GPGLGSV_selective.png)
+
+![](grafana_screenshots/210705_GNGLL.png)
+
+![](grafana_screenshots/210705_overview.png)
 
 
 ## Discussion

diff --git a/column_casting.py b/column_casting.py
@@ -20,6 +20,7 @@
 datatype_dict['Int16']   = sqlalchemy.types.SmallInteger()
 datatype_dict['Int32']   = sqlalchemy.types.Integer()
 datatype_dict['float32'] = sqlalchemy.types.Float(precision=6)
+datatype_dict['text']    = sqlalchemy.types.Text()
 
 
 # This db_datatypes dictionary is completed in dfs_to_db()
@@ -41,18 +42,26 @@
                                    'sv_prn_num_9',  'elevation_deg_9',  'azimuth_9',  'snr_9',
                                    'sv_prn_num_10', 'elevation_deg_10', 'azimuth_10', 'snr_10',
                                    'sv_prn_num_11', 'elevation_deg_11', 'azimuth_11', 'snr_11',
-                                   'sv_prn_num_12', 'elevation_deg_12', 'azimuth_12', 'snr_12',]
+                                   'sv_prn_num_12', 'elevation_deg_12', 'azimuth_12', 'snr_12',
+                                   'sv_prn_num_13', 'elevation_deg_13', 'azimuth_13', 'snr_13',
+                                   'sv_prn_num_14', 'elevation_deg_14', 'azimuth_14', 'snr_14',
+                                   'sv_prn_num_15', 'elevation_deg_15', 'azimuth_15', 'snr_15',
+                                   'sv_prn_num_16', 'elevation_deg_16', 'azimuth_16', 'snr_16',]
 
-columns_to_cast['RMC', 'Int32']  = ['datestamp']
+columns_to_cast['RMC', 'Int32']   = ['datestamp']
 columns_to_cast['RMC', 'float32'] = ['timestamp', 'lat', 'lon', 'spd_over_grnd', 'true_course', 'mag_variation']
+columns_to_cast['RMC', 'text']    = ['status', 'lat_dir', 'lon_dir', 'mode', 'mag_var_dir']
 
 columns_to_cast['GGA', 'float32'] = ['timestamp', 'lat', 'lon', 'horizontal_dil', 'altitude', 'geo_sep']
 columns_to_cast['GGA', 'Int16']   = ['gps_qual', 'num_sats']
+columns_to_cast['GGA', 'text']    = ['lat_dir', 'altitude_units', 'geo_sep_units']
 # TODO: For GGA, unsure about 'age_gps_data' and 'ref_station_id'
 
 columns_to_cast['GLL', 'float32'] = ['lat', 'lon']
+columns_to_cast['GLL', 'text']    = ['lat_dir', 'lon_dir', 'status', 'faa_mode']
 
 columns_to_cast['VTG', 'float32'] = ['true_track', 'mag_track', 'spd_over_grnd_kts', 'spd_over_grnd_kmph']
+columns_to_cast['VTG', 'text']    = ['true_track_sym', 'mag_track_sym', 'spd_over_grnd_kts_sym', 'spd_over_grnd_kmph_sym', 'faa_mode']
 
 columns_to_cast['GSA', 'Int16']   = ['mode_fix_type', 'gp_sv_id01', 'gp_sv_id02', 'gp_sv_id03', 'gp_sv_id04',
                                                       'gp_sv_id05', 'gp_sv_id06', 'gp_sv_id07', 'gp_sv_id08',
@@ -61,5 +70,6 @@
                                                       'gl_sv_id05', 'gl_sv_id06', 'gl_sv_id07', 'gl_sv_id08',
                                                       'gl_sv_id09', 'gl_sv_id10', 'gl_sv_id11', 'gl_sv_id12',]
 columns_to_cast['GSA', 'float32'] = ['pdop', 'hdop', 'vdop']
+columns_to_cast['GSA', 'text']    = ['mode']