Best Practices for Dynamic Data Fetching and Dealing with Large Datasets

[Lazywave]

Hi everyone,

I’m using Observable Framework to visualize a large dataset containing two years of 5-minute time spans and corresponding data flow amounts for "profiles" (e.g., groups of IXPs). The full dataset is around 50 GB, with ~70 million rows.

To improve performance, I pre-aggregated the data into more manageable chunks (e.g., daily aggregations across 8 regions), reducing it to ~6000 rows. This version works very well for initial visualizations since I can use Observable’s built-in data loaders and keep the data in memory.

However, I want users to be able to view 5-minute data for specific profiles. I set up a JavaScript fetch function that queries my API to load this detailed data dynamically, but I’ve encountered the following issue:

When the fetch is triggered (e.g., via a dropdown or other input), the entire plot reloads, causing flickering as the plot disappears during the request and then reappears once the data is received.
Ideally, I want the plot to stay visible and only update the marks corresponding to the new data without this full reload behavior.

Questions:
How can I structure this pipeline so that dynamic data fetching seamlessly updates only the marks inside the plot without causing the flicker?
Are there specific Observable techniques, like using mutable data or Views, that would be better suited for this scenario? I haven't really been able to find out how you're supposed to deal with large data.
Are there any examples or patterns I can follow to make this interaction smoother?

Thanks in advance for your guidance!

Here's my current code:

// API function to fetch data
async function fetchAggregatedData(startDate, endDate, regionsToShow) {
    const params = new URLSearchParams();

    if (startDate) {
        params.append('start_date', startDate.toISOString().split('T')[0]);
    }
    if (endDate) {
        params.append('end_date', endDate.toISOString().split('T')[0]);
    }
    if (regionsToShow && regionsToShow.length > 0) {
        regionsToShow.forEach(region => params.append('regions', region));
    }

    const response = await fetch(`http://localhost:8081/api/aggregated-traffic?${params}`);
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
    const result = await response.json();
    return result.data;
}

// Modified main plot function
export async function mainPlotApi(
    startDate,
    endDate,
    regionsToShow,
    regionColors,
    width
) {
    try {
        // Fetch data from API
        const data = await fetchAggregatedData(startDate, endDate, regionsToShow);

        // Create a line plot for each region with its respective color
        const marks = Object.keys(regionColors).map(region => {
        const regionData = data.filter(d => d.region_continent === region);
        return Plot.line(regionData, {
            x: "date",
            y: "total_data_in_tb",
            stroke: regionColors[region],
            strokeWidth: 1,
            strokeDasharray: region === 'Worldwide' ? '5,5' : 'none'
        });
        });

        // Render the plot
        return Plot.plot({
            width,
            marks,
            x: {
                type: "time",
                label: "Time",
                domain: [startDate, endDate],
                tickFormat: d => new Date(d).toLocaleString("en-GB", {
                    year: "2-digit", month: "short", day: "numeric"
                }),
            },
            y: {
                label: "Data Volume (TB)"
            }
        });
    }
    catch (error) {
        console.error('Error fetching or processing data:', error);
        throw error;
    }
}

[declann]

Are there specific Observable techniques, like using mutable data or Views, that would be better suited for this scenario?

Yes to mutable data: https://observablehq.com/framework/reactivity#mutables

To use the Observable reactive dataflow you would need to split this code into cells directly in the notebook.

You can make a cell like:

```js
const data = Mutable([]);

function setData(arr) {
  data.value = [...arr];
}
```

a separate cell with the plot that depends on data, and a separate cell to fetch & call setData.

I think this is what you had in mind.

[gcoch]

I have a simlilar scenario, where I have a large database in MongoDb. What I do is segment it by year and export only the fields I need to .parquet files, which are much smaller. I've seen a considerable performance improvement. For the current year, I append the last week to the historic data and that seems to work also.