Change recorder_prefilter from submodule to subtree

custom_components/places/recorder_history_prefilter/.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

custom_components/places/recorder_history_prefilter/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 gcobb321
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

custom_components/places/recorder_history_prefilter/README.md

@@ -0,0 +1,123 @@
+# Home Assistant Recorder History Prefilter
+
+The HA Recorder module was modified in HA 2023.6.0 to no longer allow a custom component to insert a sensor or other type of entity into the *_exclude_e* list that defined the entities that should not be added to the History database (home_assistant_v2.db).
+
+This module fixes that problem. 
+
+
+
+## How it Works
+
+#### The Home Assistant Recorder Component
+
+When Home Assistant starts, it loads the recorder component. The recorder component gets it's parameters from the HA configuration file (configuration.yaml) and builds the list of entities, globs, etc. to be filtered. It then starts a state change listener to call the filter module whenever an entity's state is changed. The filter determines if the entity should be added to the History database,
+
+For efficiency, Home Assistant builds the filter that is checked one time when Home Assistant is loaded. The filter can not be changed or added to by a custom_component.
+
+#### The Recorder History Prefilter
+
+This module injects it's own filter (actually a copy of the recorder's filter) into the HA Recorder component, removes the HA listener that was set up by the HA Recorder and sets up a new state change listener. It is called when an entity's state changes instead of the HA Recorder filter and checks to see if the entity should be filtered. It the entity is not in it's list, the entity is passed to the HA Recorder filter module as it normally would be.
+
+
+
+### Using the Recorder Prefilter
+
+Entities are added or removed from this list using function calls from a custom component. More than one custom component can use this.
+
+#### Adding Entities
+
+The recorder prefilter is contained in the *recorder_prefilter.py* module. 
+
+- Download it and add it to your code base.
+
+Import it into the your module that determines the entities to be filtered. This may be in  *\__init__.py*, *sensor.py* or another module.
+
+- `import recorder_prefilter`
+
+The a single entity or a list of entities can be added at one time using the *add_filter* function call.
+
+- `recorder_prefilter.add_filter(hass, 'icloud3_event_log')`
+- `recorder_prefilter.add_filter(hass, ['icloud3_event_log', 'icloud3_wazehist_track', 'gary_iphone_info'])`
+- `recorder_prefilter.add_filter(hass, [filtered_entities_list)`
+
+Notes: 
+
+- The entity_type (*input_boolean, light, etc* ) needs to be used if it is a non-sensor entity. 
+- The first custom component to use the *add_filter* function call will inject the *recorder_prefilter* into the HA Recorder component. All subsequent *add_filter* calls will update the filter list.
+
+#### Removing Entities
+
+Entities can be removed from the filter list using the *remove_filter* function call.
+
+The a single entity or a list of entities can be removed at one time.
+
+- `recorder_prefilter.remove_filter(hass, 'icloud3_event_log')`
+
+- `recorder_prefilter.remove_filter(hass, ['icloud3_event_log', 'icloud3_wazehist_track', 'gary_iphone_info'])`
+
+- `recorder_prefilter.remove_filter(hass, [filtered_entities_list)`
+
+
+
+### Logging
+
+Normal HA Logging is available for the *recorder_prefilter* module. Enable it in configuration.yaml as you would do with any other component.
+
+    logger:
+      default: info
+      logs:
+        custom_components.icloud3.support.recorder_prefilter: info
+        custom_components.places.recorder_prefilter: info
+
+**Info logging** - Add basic records to the *home-assistant.log* file
+
+```Recorder Prefilter Injection Started
+Recorder Prefilter Injection Started
+Recorder Prefilter Injection Completed
+Added Recorder Prefilter Entities (icloud3)-2
+Recorder Prefilter Entities Updated, Entities Filtered-2
+...
+...
+Added Recorder Prefilter Entities (places)-1
+Recorder Prefilter Entities Updated, Entities Filtered-3
+...
+...
+Added Recorder Prefilter Entities (icloud3)-6
+Recorder Prefilter Entities Updated, Entities Filtered-9
+```
+
+
+
+**Debug logging** - Add operational records to the *home-assistant.log* file.
+
+    custom_components.icloud3.support.recorder_prefilter: debug
+    custom_components.places.recorder_prefilter: debug
+
+```Recorder Prefilter Injection Started
+Recorder Prefilter Injection Started
+Injecting Custom Exclude Entity Prefilter into Recorder
+Removing Recorder Event Listener
+Reinitializing Recorder Event Listener
+Recorder Prefilter Injection Completed
+Added Prefilter Entities-['icloud3_event_log', 'icloud3_wazehist_track']
+Added Recorder Prefilter Entities (icloud3)-2
+All Prefiltered Entities-['sensor.icloud3_event_log', 'sensor.icloud3_wazehist_track']
+Recorder Prefilter Entities Updated, Entities Filtered-2
+...
+...
+Added Prefilter Entities-sensor.gary_place
+Added Recorder Prefilter Entities (places)-1
+All Prefiltered Entities-['sensor.gary_place', 'sensor.icloud3_event_log', 'sensor.icloud3_wazehist_track']
+Recorder Prefilter Entities Updated, Entities Filtered-3
+...
+...
+Added Prefilter Entities-['sensor.gary_iphone_info', 'sensor.gary_iphone_trigger', 'sensor.lillian_iphone_info', 'sensor.lillian_iphone_trigger', 'sensor.lillian_watch_info', 'sensor.lillian_watch_trigger']
+Added Recorder Prefilter Entities (icloud3)-6
+All Prefiltered Entities-['sensor.gary_iphone_info', 'sensor.gary_iphone_trigger', 'sensor.gary_place', 'sensor.icloud3_event_log', 'sensor.icloud3_wazehist_track', 'sensor.lillian_iphone_info', 'sensor.lillian_iphone_trigger', 'sensor.lillian_watch_info', 'sensor.lillian_watch_trigger']
+Recorder Prefilter Entities Updated, Entities Filtered-9
+
+```
+
+
+
+Developed by:  Gary Cobb, *iCloud3 iDevice Tracker custom component*, (aka geekstergary)

custom_components/places/recorder_history_prefilter/recorder_prefilter.py

@@ -0,0 +1,182 @@
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+#
+#   HA RECORDER - EXCLUDE entities FROM BEING ADDED TO HISTORY DATABASE
+#
+#
+#   The HA Recorder module was modified in HA 2023.6.0 to no longer allow a custom
+#   component to insert entity entity names in the '_exclude_e' list that defined
+#   entity entities to not be added to the History database (home_assistant_v2.db).
+#
+#   This module fixes that problem by using a code injection process to provide a
+#   local prefilter to determine if an entity should be added before the Recorder filter.
+#
+#
+#   This injection has two methods:
+#       add_filter - Add entities to the filter list
+#       ----------
+#           hass - HomeAssistant
+#           entities to be filtered -
+#                   single entity - entity_id (string)
+#                   multiple entities - list of entity ids
+#
+#                   'sensor.' will be added to the beginning of the entity id if
+#                   it's type is not specifid
+#
+#           recorder_prefilter.add_filter(hass, 'filter_id1')
+#           recorder_prefilter.add_filter(hass, ['filter_entity2', 'filter_entity3'])
+#
+#
+#       remove_filter - Remove entities from the filter list
+#       -------------
+#           Same arguments for add_filter
+#
+#           recorder_prefilter.remove_filter(hass, 'filter_id1')
+#           recorder_prefilter.remove_filter(hass, ['filter_entity2', 'filter_entity3'])
+#
+#
+#   Gary Cobb, iCloud3 iDevice Tracker, aka geekstergary
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+from homeassistant.core import HomeAssistant
+from inspect import getframeinfo, stack
+import logging
+_LOGGER = logging.getLogger(__name__)
+
+VERSION = 1.0
+
+def add_filter(hass: HomeAssistant, entities=None):
+    '''
+    Inject the entity prefilter into the Recorder, remove Recorder listeners,
+    reinitialize the Recorder
+
+    Arguments:
+        hass - HomeAssistant
+        entities - A list of entity entities
+                        (['gary_last_update', 'lillian_last_update', '*_next_update'])
+                      - A single entity entity ('gary_last_zone')
+
+    Returns:
+        True - The injection was successful
+        False - The injection was not successful
+    '''
+
+    ha_recorder = hass.data['recorder_instance']
+
+    if ha_recorder is None:
+        return False
+
+    if hass.data.get('recorder_prefilter') is None:
+        rp_data = hass.data['recorder_prefilter'] = {}
+        rp_data['injected'] = True
+        rp_data['legacy'] = True
+        rp_data['exclude_entities'] = []
+
+        try:
+            ha_recorder.entity_filter._exclude_e.add(entities)
+            return True
+        except:
+            pass
+
+        rp_data['legacy'] = False
+
+        if _inject_filter(hass) is False:
+            return
+
+    _update_filter(hass, entities)
+
+
+def remove_filter(hass: HomeAssistant, entities):
+    if hass.data['recorder_prefilter']['legacy']:
+        try:
+            ha_recorder = hass.data['recorder_instance']
+            ha_recorder.entity_filter._exclude_e.discard(entities)
+            return True
+        except Exception as err:
+            _LOGGER.exception(err)
+
+    _update_filter(hass, entities, remove=True)
+
+
+def _inject_filter(hass: HomeAssistant):
+    ha_recorder = hass.data['recorder_instance']
+    rp_data = hass.data['recorder_prefilter']
+    recorder_entity_filter   = ha_recorder.entity_filter
+    recorder_remove_listener = ha_recorder._event_listener
+
+    def entity_filter(entity_id):
+        """
+        Prefilter an entity to see if it should be excluded from
+        the recorder history.
+
+        This function is injected into the recorder, replacing the
+        original HA recorder_entity_filter module.
+
+        Return:
+            False - The entity should is in the filter list
+            Run the original HA recorder_entity_filter function -
+                The entity is not in the filter list.
+        """
+        if (entity_id
+                and entity_id in hass.data['recorder_prefilter']['exclude_entities']):
+            return False
+
+        return recorder_entity_filter(entity_id)
+
+    try:
+        _LOGGER.info("Recorder Prefilter Injection Started")
+        _LOGGER.debug("Injecting Custom Exclude Entity Prefilter into Recorder")
+        ha_recorder.entity_filter = entity_filter
+
+        _LOGGER.debug("Removing Recorder Event Listener")
+        recorder_remove_listener()
+
+        _LOGGER.debug("Reinitializing Recorder Event Listener")
+        hass.add_job(ha_recorder.async_initialize)
+
+        _LOGGER.info(f"Recorder Prefilter Injection Completed")
+
+        return True
+
+    except Exception as err:
+        _LOGGER.info(f"Recorder Prefilter Injection Failed ({err})")
+        _LOGGER.exception(err)
+
+    return False
+
+
+def _update_filter(hass: HomeAssistant, entities=None, remove=False):
+    """ Update the filtered entity list """
+
+    mode = 'Removed' if remove else 'Added'
+    cust_component = _called_from()
+    entities_cnt = 1 if type(entities) is str else len(entities)
+
+    _LOGGER.debug(f"{mode} Prefilter Entities ({cust_component})-{entities}")
+    _LOGGER.info(f"{mode} Recorder Prefilter Entities "
+                    f"({cust_component})-{entities_cnt}")
+
+    entities =  [entities]  if type(entities) is str else \
+                entities    if type(entities) is list else \
+                []
+
+
+    rp_data = hass.data.get('recorder_prefilter')
+    rp_exclude_entities = rp_data['exclude_entities']
+
+    for entity in entities:
+        if entity.find('.') == -1:
+            entity = f"sensor.{entity}"
+        if entity not in rp_exclude_entities:
+            if remove is False:
+                rp_exclude_entities.append(entity)
+            elif entity in rp_exclude_entities:
+                rp_exclude_entities.remove(entity)
+
+    _LOGGER.debug(f"All Prefiltered Entities-{sorted(rp_exclude_entities)}")
+    _LOGGER.info(f"Recorder Prefilter Entities Updated, "
+                    f"Entities Filtered-{len(rp_exclude_entities)}")
+
+
+def _called_from():
+    cust_component = getframeinfo(stack()[0][0]).filename
+    return cust_component.split('custom_components/')[1].split('/')[0]