Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MM-60600: CLI tools for deduplicated event table #1631

Merged
merged 32 commits into from
Oct 22, 2024
Merged

MM-60600: CLI tools for deduplicated event table #1631

merged 32 commits into from
Oct 22, 2024

Conversation

ifoukarakis
Copy link
Contributor

@ifoukarakis ifoukarakis commented Oct 10, 2024 •
edited
Loading

Summary

  • CLI utils for deferred merge.
  • Handle schema changes when applying merge.
  • Minor refactoring to avoid duplicate code.
  • DBT models for maintaining delta table and deduped view.

Doesn't include:

  • Replacing current model with new deduplicated one.
  • Automation

Ticket Link

https://mattermost.atlassian.net/browse/MM-60600

@ifoukarakis ifoukarakis added the Work In Progress Not yet ready for review label Oct 10, 2024
@ifoukarakis ifoukarakis force-pushed the MM-60600-deferred-merge-events branch from 4c967d7 to d7805b4 Compare October 14, 2024 14:40
@ifoukarakis ifoukarakis force-pushed the MM-60600-deferred-merge-events branch from d7805b4 to 5544f15 Compare October 18, 2024 07:58
@ifoukarakis ifoukarakis changed the title MM-60600: deduplicate events table MM-60600: CLI tools for deduplicated event table Oct 21, 2024
@ifoukarakis ifoukarakis marked this pull request as ready for review October 21, 2024 07:44
@ifoukarakis ifoukarakis added 2: Dev Review Requires review by a core committer and removed Work In Progress Not yet ready for review labels Oct 21, 2024
catalintomai
catalintomai reviewed Oct 22, 2024
View reviewed changes
...ttermost-analytics/models/staging/mm_telemetry_prod/stg_mm_telemetry_prod__event_deduped.sql
, plugin_id
, post_id
, privacy
, q
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be worth removing the spurious columns before running the process.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check values on these columns first. Let's do it in a separate PR in order to keep things simpler.

@ifoukarakis ifoukarakis added 3: Reviews Complete All reviewers have approved the pull request and removed 2: Dev Review Requires review by a core committer labels Oct 22, 2024
@ifoukarakis ifoukarakis merged commit 551e3a2 into master Oct 22, 2024
4 checks passed
@ifoukarakis ifoukarakis deleted the MM-60600-deferred-merge-events branch October 22, 2024 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@catalintomai catalintomai catalintomai approved these changes

Assignees
No one assigned
Labels
3: Reviews Complete All reviewers have approved the pull request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ifoukarakis @catalintomai