Trendline for scatter plots

[Heronimonimo]

First off, what a great card. I'm really having a lot of fun making new visualizations of my data.

Describe the solution you'd like
I want to add a trendline to my scatter plots.

How would it be defined in yaml?

  - entity: ""
    x: $fn ({ vars }) => vars.temperature
    "y": $fn ({ vars }) => vars.heat
    filters:
       - trendline:
           type: linear
           show_formula: true
           show_r2: true
    type: scatter

Scribble
Examples of how this could look like in Plotly can be found here: https://plotly.com/python/linear-fits/

[dbuezas]

Hi there!
I'm not quite sure what you mean, the feature exists, has multiple different regression algorithms and offers the exact same api you mentioned

[dbuezas]

Is it because you are also setting the x axis?
Maybe try setting it inside a filter (e.g with map_x and map_y) instead of setting x and y in the entity.

Btw, happy to hear you are having fun with it!

[Heronimonimo]

Thanks for the quick reply.

An example, I'm using one entity (heat produced by heatpump) to set Y and another (outdoor temperature to set X. When I define a trendline as set above I'm getting a line from point to point in the scatter, but not a trendline fitted trough the dataset.

[Heronimonimo]

This is what my plot looks like. Using this code:

  - entity: ""
    x: $fn ({ vars }) => vars.temperature
    "y": $fn ({ vars }) => vars.heat
    filters:
      trendline:
       trendline: ols
    name: points+line
    marker:
      color: white
      size: 6

[dbuezas]

See my comment above, also make sure your yaml is well defined (you have issues with the filters there.

Try the yaml editor helper linked at the top of the readme to validate your yaml

[Heronimonimo]

See my comment above, also make sure your yaml is well defined (you have issues with the filters there.

Try the yaml editor helper linked at the top of the readme to validate your yaml

Unfortunately I don't understand how I could use 'map_y' and 'map_x' to get to a correct trendline in the scatter plot.

My full code looks like this:

type: custom:plotly-graph
title: Warmteproductie per week
entities:
  - entity: sensor.buienradar_temperature
    period: day
    internal: true
    fn: $fn ({ ys, vars }) => vars.temperature = ys
  - entity: sensor.panasonic_heat_pump_main_heat_energy_production
    period: day
    internal: true
    fn: $fn ({ ys, vars }) => vars.heat = ys
  - entity: ""
    x: $fn ({ vars }) => vars.temperature
    "y": $fn ({ vars }) => vars.heat
    mode: markers
    name: points
    marker:
      color: green
      size: 6
    type: scatter
hours_to_show: 12000
raw_plotly_config: true
layout:
  showlegend: true
  margin:
    r: 50
    l: 75
    t: 50
    b: 75
  legend:
    orientation: h
  xaxis:
    title:
      text: Temperature
    range:
      - -5
      - 18
    showgrid: false
    zeroline: false
  yaxis:
    title:
      text: Heat produced (W)
    range:
      - 0
      - 7500
    showgrid: false
    zeroline: false

[dbuezas]

Oh, sorry, not map_y/x, the fn filter:

filters:
  - fn: |
          ({vars}) => ({xs: vars.temperature, ys: vars.heat})
  - trendline:
           type: linear
           show_formula: true
           show_r2: true

I think this may work

[dbuezas]

Uhmmmm on a second thought, I think the problem is just that your x axis isn't sorted. Try this:

- fn: |
    ({vars}) => {
      const records = vars.temperature.map((x,i)=>({
          x,
          y:vars.heat[i]
      })).sort((a,b)=>a.x-b.x);
      return {
          xs: records.map(({x})=>x),
          ys: records.map(({y})=>y)
      };
    }

Followed by the Trendline filter

[Heronimonimo]

Uhmmmm on a second thought, I think the problem is just that your x axis isn't sorted. Try this:

- fn: |
    ({vars}) => {
      const records = vars.temperature.map((x,i)=>({
          x,
          y:vars.heat[i]
      }).sort((a,b)=>a.x-b.x);
      return {
          xs: records.map(({x})=>x),
          ys: records.map(({y})=>y)
      };
    }

Followed by the Trendline filter

Oke, just tried it. Getting this now:
Error: at [entities.3.filters.0]: Error in filter: TypeError: {(intermediate value)(intermediate value)}.sort is not a function

[dbuezas]

try again, corrected a missing parenthesis.

[Heronimonimo]

try again, corrected a missing parenthesis.

That fixes the code but still gives me a scatter with a line from dot to dot.

[dbuezas]

show me what you've got

[Heronimonimo]

Yes, I've got this now.

type: custom:plotly-graph
title: Warmteproductie per dag
entities:
  - entity: sensor.buienradar_temperature
    period: day
    internal: true
    fn: $fn ({ ys, vars }) => vars.temperature = ys
  - entity: sensor.panasonic_heat_pump_main_heat_energy_production
    period: day
    internal: true
    fn: $fn ({ ys, vars }) => vars.heat = ys
  - entity: ""
    x: $fn ({ vars }) => vars.temperature
    "y": $fn ({ vars }) => vars.heat
    mode: markers
    name: points
    marker:
      color: green
      size: 6
    type: scatter
  - entity: ""
    x: $fn ({ vars }) => vars.temperature
    "y": $fn ({ vars }) => vars.heat
    mode: line
    filters:
      - fn: >
          ({vars}) => { const records =
          vars.temperature.map((x,i)=>({x,y:vars.heat[i]})).sort((a,b)=>a.x-b.x);
          return {
           xs: records.map(({x})=>x),
           ys: records.map(({y})=>y)
          }; }
      - trendline:
          type: linear
          show_formula: true
          show_r2: true
  - entity: ""
    name: 375W/deg = 7.5kW @ -5 deg
    x:
      - 18
      - 15
      - -5
    "y":
      - 0
      - 0
      - 7500
    line:
      color: red
      shape: linear
  - entity: ""
    name: 325W/deg = 6.5kW @ -5 deg
    x:
      - 18
      - 15
      - -5
    "y":
      - 0
      - 0
      - 6500
    line:
      color: blue
      shape: linear
hours_to_show: 336
raw_plotly_config: true
layout:
  showlegend: true
  margin:
    r: 50
    l: 75
    t: 50
    b: 75
  legend:
    orientation: h
  xaxis:
    title:
      text: Temperature
    range:
      - -5
      - 18
    showgrid: false
    zeroline: false
  yaxis:
    title:
      text: Heat produced (W)
    range:
      - 0
      - 7500
    showgrid: false
    zeroline: false

[dbuezas]

Try with this:

type: custom:plotly-graph
$fn: |
  $ex {
    // ignore this block
    vars.temperature = [2,4,12,2,4,5,67,3,3,2];
    vars.heat =        [4000,2000,2000,3000,1000,4000,6000,7000,5000,3000,1];
  }
entities:
  - entity: ""
    mode: lines
    name: sorted data
    fn: |
      $ex {
        const records = vars.temperature
          .map((x, i) => ({
            x,
            y: vars.heat[i],
          }))
          .sort((a, b) => a.x - b.x);
        vars.sorted = {
          xs: records.map(({ x }) => x),
          ys: records.map(({ y }) => y),
        };
      }
    x: $ex vars.sorted.xs
    y: $ex vars.sorted.ys
  - entity: ""
    name: trendline
    mode: lines
    filters:
      - load_var: sorted
      - trendline:
          type: linear
          show_formula: true
          show_r2: true
      - store_var: trendline
    x: $ex vars.trendline.xs.map(x=>+x)
    y: $ex vars.trendline.ys
    
raw_plotly_config: true
layout:
  showlegend: true
  margin:
    r: 50
    l: 75
    t: 50
    b: 75
  legend:
    orientation: h
  xaxis:
    title:
      text: Temperature
    range:
      - -5
      - 18
    showgrid: false
    zeroline: false
  yaxis:
    title:
      text: Heat produced (W)
    range:
      - 0
      - 7500
    showgrid: false
    zeroline: false

[Heronimonimo]

This does give me a trendline, but unfortunately a piecewise one and not a single straight line.

[dbuezas]

I think that is just numerical error, you can mal it again to remove all but the first and last data points of the trendline

[Heronimonimo]

I think that is just numerical error, you can mal it again to remove all but the first and last data points of the trendline

I'm curious why your example code gave such a different trendline without this problem.

In this thread I also saw similar issues with the histogram plots having piecewise trendlines:
#261

Also the formula of the trendline isn't showing up.

[dbuezas]

The formula and error squared (r2) will go into the name of the trace, so to see them you can:

• Remove name: trendline from the entity in your yaml so it is not overwritten
• Grab it from meta.friendly_name after applying the filter and use it for something else (e.g on an indicator type trace).

Regarding the piece wise trend line, to be honest I don't know, maybe the lib I use for the linear trendline (https://www.npmjs.com/package/ml-regression-simple-linear) thinks it should use integers or something like that.

You could try using the force_numeric filter before the trendline in case the numbers come in as strings for some reason and that ends up creating issues.