Add feature to filter by date in users analysis table #1010

benzuzu · 2023-08-02T06:12:10Z

We can now select time range in the users tab of the usage analysis page. Most importantly, this allows us to see recent usage, rather than just cumulative.

This was done by fetching dayUsage and monthUsage from the backend. It does take a bit longer, but the difference is not actually too significant. The longest part is actually fetching the files themselves from S3, not parsing through. Some JHipster analysis may be good.

this fixes https://github.com/oncokb/oncokb-pipeline/issues/169

Some other small changes:

"Top Users" and "All Users" filter is removed. This doesn't change render/fetch time, so there is really not much purpose.
Most used endpoint also includes proportion of all usage.
Default sort is reset when returning to a specific time range

Some ideas for future:

The UsageText component we've made has the limits hardcoded. I think it could be good to accept a string for type of usage (daily, monthly, all-time), and adjust the color limits accordingly (1 mil in 1 day definitely more concerning than 1 mil all time).
Month usage can be slightly unclear, especially at the beginning of months. The usage is very small as it is only from a couple of days, but we don't display this context anywhere (e.g., all months just have format "2023-08"). Also, the s3 summary files are at least 1-2 days behind based on our cronjob (potentially more if cronjob fails), so using the current date is not always accurate. It may be nice to add an (x days) text next to months that are not fully analyzed, or maybe even all months.
JHipster runtime analysis to potentially speed up the fetching/rendering.

benzuzu · 2023-08-02T14:27:14Z

Regarding clearing up the confusion on S3 summary files and our backend methods:

S3 currently stores 4 different types of files:

	Monthly Usage	Yearly Usage	Monthly Resource	Yearly Resource
Fmt	<user>: month: <resource>: <amt> <resource>: <amt> day: <day>: <resource>: <amt> <user>: ...	<user>: year: <resource>: <amt> <resource>: <amt> month: <month>: <resource>: <amt> <user>: ...	month: <resource>: <amt> <resource>: <amt> day: <day>: <resource>: <amt> <day>:...	year: <resource>: <amt> <resource>: <amt> month: <month>: <resource>: <amt> <month>:...
Info	This pretty much contains everything (specific resource usage, grouped by day). Can manipulate to create all other files. We now use this for all user-based tables.	More general version of Monthly Usage. Used for summarizing cumulative usage, as the year data is very convenient for that, specifically in the all time usage tables.	Contains usage amounts for each resource (grouped by month and day), but no user data. Currently not used at all. Can consider removing this.	Contains usage amounts for each resource (grouped by year and month), but no user data. Currently used for resource pages.

So, I think the somewhat significant downfall is that our only useful day-grouped data is in the Monthly Usage file. Consequently, regardless of our use case, if we want to get user-specific, day-grouped data we must parse through the massive monthly usage files (12 if we want past year). However, it seems we haven't reached too big of a user base, so the fetching of the file from S3 still contributes more to the processing time than the backend parsing (but not by much, both are nontrivial).

Definitely something to look into regardless, maybe different formatting of our summary files to have more versatility and efficiency. Specifically, maybe we can look at a <user>: <amt> format instead of a <resource>: <amt> format for the monthly resource. This has much more utility as we can group resource usage by user and date.

zhx828 · 2023-08-10T13:14:38Z

@benzuzu thanks for the detailed comments. I agree doing <user>: <amt> format instead of a <resource>: <amt> seems more reasonable. But we might add annotate/bySample, annotate/clinicalTrials in the future which makes latter more attractive structure. Let's revisit in a year when we have more endpoints available.

zhx828 · 2023-08-10T13:23:08Z

src/main/webapp/app/pages/usageAnalysisPage/UsageAnalysisTable.tsx

+    this.filterToggled = value;
+  }
+
+  @computed get calculateData(): UserOverviewUsage[] | UsageRecord[] {


I would rather for you to calculate the data separately and render two tables so the code looks more organized. Attache a key to differentiate. Any performance concern here?

zhx828 · 2023-08-10T13:25:32Z

src/main/webapp/app/pages/usageAnalysisPage/UsageAnalysisTable.tsx

+    if (this.timeTypeToggleValue === ToggleValue.RESULTS_BY_MONTH) {
+      this.props.data.forEach((userOverviewUsage: UserOverviewUsage) => {
+        for (const curMonth in userOverviewUsage.monthUsage) {
+          if (Object.prototype.hasOwnProperty.call(userOverviewUsage.monthUsage, curMonth)) {


I'm confused by the usage of the condition here. Could you elaborate?

zhx828 · 2023-08-10T13:29:31Z

src/main/webapp/app/pages/usageAnalysisPage/UsageAnalysisTable.tsx

+                  Header: usageHeader,
+                  accessor: 'totalUsage',
+                  Cell(props: { original: UserOverviewUsage }) {
+                    return <UsageText usage={props.original.totalUsage ? props.original.totalUsage : 0} />;


I think as a primitive data type, they all have default values. Why we are checking here?

zhx828

The icon under Details is not clickable except for user public_website@localhost. But it should be enabled for all.
Can calendar tooltip be closed if use clicks anywhere on the page?
Could you add a link on the email to user page? It should apply to both Users tab and resource page Example : oncokb.org/admin/usage-analysis/resources/!api!v1!annotate!mutations!byProteinChange

benzuzu · 2023-08-21T04:31:17Z

The icon under Details is not clickable except for user public_website@localhost. But it should be enabled for all.

Can calendar tooltip be closed if use clicks anywhere on the page?

Could you add a link on the email to user page? It should apply to both Users tab and resource page Example : oncokb.org/admin/usage-analysis/resources/!api!v1!annotate!mutations!byProteinChange

Thanks for the review @zhx828. Regarding the first item, I'm pretty sure this is just because we are using the fake data to run locally. The backend searches the database for userid since we do not store in s3. Should be all good in production.

I'll work on the two frontend components you had comments on now.

Some logic was wrong in the default sorting rules. Also, the day and month usage hashmap does not have userId stored, so this fix adds an optional property that can be used for the info icon link.

Render two table instead of one based on data type (date grouping)

zhx828 · 2024-03-20T19:56:35Z

@benzuzu for the same user on the same day, why there are two entries?

benzuzu · 2024-03-20T20:55:34Z

@benzuzu for the same user on the same day, why there are two entries?

@zhx828 thanks for catching. all the users in fake data had the same email. just fixed

zhx828 · 2024-03-20T20:59:11Z

src/main/webapp/app/config/constants.tsx

@@ -104,7 +104,7 @@ export const SHORTEN_TEXT_FROM_LIST_THRESHOLD = 5;
 export const TOKEN_ABOUT_2_EXPIRE_NOTICE_IN_DAYS = 14;
 export const USAGE_TOP_USERS_LIMIT = 10000;
 export const USAGE_ALL_TIME_KEY = 'All';
-export const USAGE_ALL_TIME_VALUE = 'This year';
+export const USAGE_ALL_TIME_VALUE = 'Past Year';


When I see Past Year, I think about 2023. But the stats is for 2024. Or am I using Past Year wrong?

It does the past year, i.e. one year back from today. Is this confusing/not what we wanted?

maybe "last 12 months" is more clear - thoughts?

@benzuzu oh I see. Last 12 Months is definitely better.

@zhx828 Should we also change By Year tab on the top to

All Time (Last 12 Months)

Last 12 Months

All (Last 12 Months)

All Time

or does By Year make sense? I think one of the first 3 makes more sense

@benzuzu I'm a bit confused. Looking at the stats for the user, it's past 12 month. But for resource, it's from the starting of the year to current?

@zhx828 ah, sorry. I was wrong - it's been a bit since I wrote the logic. Both the user and resource stats are beginning of year to current (not last 12 months). This is because we store the analysis in yearly files, not constantly updating to last 12 months. If I remember correctly, we decided to change backed logic in userOverviewUsageGet specifically so that we could view specific user usage for the past year (not just year to date). I've changed the tab now below:

I can change the backend logic for resourceUsageGet as well to add all 12 months, but I'm not sure if it is worth the trouble.

…to include older months

zhx828 · 2024-03-21T13:37:00Z

@benzuzu looks good. Let me know once this is ready to be merged.

benzuzu · 2024-03-21T14:22:15Z

@benzuzu looks good. Let me know once this is ready to be merged.

@zhx828 good to go now. Thanks

benzuzu requested a review from zhx828 August 2, 2023 15:04

zhx828 mentioned this pull request Aug 2, 2023

Enable data usage grouping by date in usage analysis page #994

Closed

benzuzu force-pushed the usage-analysis-group-by-date branch 2 times, most recently from 6271a94 to 0e2f67e Compare August 4, 2023 20:08

zhx828 requested changes Aug 10, 2023

View reviewed changes

benzuzu requested a review from zhx828 August 15, 2023 14:59

zhx828 requested changes Aug 16, 2023

View reviewed changes

benzuzu force-pushed the usage-analysis-group-by-date branch from d5478e5 to 87f25c7 Compare March 18, 2024 09:04

benzuzu added 5 commits March 18, 2024 02:05

Filter by date in users tab

df9c694

Fix small logic errors in sorting and individual user page link

bab5128

Some logic was wrong in the default sorting rules. Also, the day and month usage hashmap does not have userId stored, so this fix adds an optional property that can be used for the info icon link.

Update screenshot test data

785dc27

Modify Table Logic

a7b6c63

Render two table instead of one based on data type (date grouping)

close calendar on outside click

dec2980

benzuzu force-pushed the usage-analysis-group-by-date branch from 87f25c7 to dec2980 Compare March 18, 2024 09:06

add hyperlinks to emails and endpoints

b41c23d

benzuzu requested a review from zhx828 March 20, 2024 18:37

benzuzu added 2 commits March 20, 2024 14:09

fix ALL resource bug

a10fffd

update screenshot test

96502a9

benzuzu added enhancement Enhancement tag for release Ready to merge labels Mar 20, 2024

benzuzu added 2 commits March 20, 2024 15:32

fix screenshot test data bug

ae30fe6

update screenshot

371d716

zhx828 approved these changes Mar 20, 2024

View reviewed changes

benzuzu added 2 commits March 20, 2024 18:59

change 'all time' to 'year to date' and adjust individual user logic …

f9e71ce

…to include older months

route userpage usage table to fake data

9f33e18

benzuzu added 2 commits March 20, 2024 19:56

route companypage usage table to fake data

41576e4

screenshots

b8f7132

zhx828 merged commit 4ea7581 into oncokb:master Mar 21, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add feature to filter by date in users analysis table #1010

Add feature to filter by date in users analysis table #1010

benzuzu commented Aug 2, 2023 •

edited

Loading

benzuzu commented Aug 2, 2023 •

edited

Loading

zhx828 commented Aug 10, 2023

zhx828 Aug 10, 2023

zhx828 Aug 10, 2023

zhx828 Aug 10, 2023

zhx828 left a comment •

edited by benzuzu

Loading

benzuzu commented Aug 21, 2023

zhx828 commented Mar 20, 2024 •

edited

Loading

benzuzu commented Mar 20, 2024

zhx828 Mar 20, 2024

benzuzu Mar 20, 2024 •

edited

Loading

zhx828 Mar 20, 2024

benzuzu Mar 20, 2024 •

edited

Loading

zhx828 Mar 20, 2024

benzuzu Mar 20, 2024 •

edited

Loading

zhx828 commented Mar 21, 2024

benzuzu commented Mar 21, 2024

Add feature to filter by date in users analysis table #1010

Add feature to filter by date in users analysis table #1010

Conversation

benzuzu commented Aug 2, 2023 • edited Loading

benzuzu commented Aug 2, 2023 • edited Loading

zhx828 commented Aug 10, 2023

zhx828 Aug 10, 2023

Choose a reason for hiding this comment

zhx828 Aug 10, 2023

Choose a reason for hiding this comment

zhx828 Aug 10, 2023

Choose a reason for hiding this comment

zhx828 left a comment • edited by benzuzu Loading

Choose a reason for hiding this comment

benzuzu commented Aug 21, 2023

zhx828 commented Mar 20, 2024 • edited Loading

benzuzu commented Mar 20, 2024

zhx828 Mar 20, 2024

Choose a reason for hiding this comment

benzuzu Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

zhx828 Mar 20, 2024

Choose a reason for hiding this comment

benzuzu Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

zhx828 Mar 20, 2024

Choose a reason for hiding this comment

benzuzu Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

zhx828 commented Mar 21, 2024

benzuzu commented Mar 21, 2024

benzuzu commented Aug 2, 2023 •

edited

Loading

benzuzu commented Aug 2, 2023 •

edited

Loading

zhx828 left a comment •

edited by benzuzu

Loading

zhx828 commented Mar 20, 2024 •

edited

Loading

benzuzu Mar 20, 2024 •

edited

Loading

benzuzu Mar 20, 2024 •

edited

Loading

benzuzu Mar 20, 2024 •

edited

Loading