Perf: Improve performance of top stats query with long time windows #4857
Conversation
We usually don't read events/pageviews from sessions table as sessions can start before or after the time window of the query. This is particularly pronounced with realtime queries. After this change we start reading events/pageviews from sessions table for longer time periods as the inaccuracies introduced by the session timing is minimized this way. The end result is that queries for e.g. top stats on dashboard load for 7 days or longer periods only requires reading sessions table.
| # When querying large time window aggregates, session-level event/pageview counts | ||
| # are accurate enough: the amount of sessions starting/ending outside of time window | ||
| # doesn't matter as much. | ||
| query_days > 5 and empty?(query.filters) and empty?(query.dimensions) -> :either |
There was a problem hiding this comment.
This condition would be true for some periods on virgin dashboards (of users that have just started using Plausible). When the counts are in the single digits, the count changing when cycling through various periods may be mistaken for a bug.
There was a problem hiding this comment.
It may be an acceptable tradeoff, but maybe it's possible to describe a condition where the differences in counts really don't matter? For whom is the events table query most expensive actually?
There was a problem hiding this comment.
Discussed this live.
A problem with this approach is that when user looks at montly view and sees that $DAY had 9 visitors and after clicking on $DAY sees they had 8 visitors, this might trigger some WTFs from our users.
Arturs suggestion was to change this logic to only not read from sessions in realtime view.
I think this is valid, but want to ponder this a bit more and figure out the implications. Marking as a draft for now.
We usually don't read events/pageviews from sessions table as sessions can start before or after the time window of the query. This is particularly pronounced with realtime queries.
After this change we start reading events/pageviews from sessions table for longer time periods as the inaccuracies introduced by the session timing is minimized this way.
The end result is that queries for e.g. top stats on dashboard load for 7 days or longer periods only requires reading sessions table.