Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: remove empty fields from output #67

Merged
merged 5 commits into from
Dec 11, 2023
Merged

fix: remove empty fields from output #67

merged 5 commits into from
Dec 11, 2023

Conversation

ddlees
Copy link
Contributor

@ddlees ddlees commented Dec 8, 2023

  • updated struct tags to include omitempty attribute
  • created custom marshalers for types that serialize json.RawMessage such that the json.RawMessage value is also pruned of "empty" fields

Note: The following data was measured against a sample azure tenant and meant only to serve as an example. The output delta will vary depending on the size of the target environment.

Output size comparison by kind

Kind v2.1.6 bytes next bytes change
app-owners 5673039 2549207 3123832 (55.06%)
apps 4249214 2668793 1580421 (37.19%)
automation-accounts 5284 4780 504 (9.54%)
container-registries 1269 1239 30 (2.36%)
device-owners 165394 15501 149893 (90.63%)
devices 47398 32890 14508 (30.61%)
function-apps 11619 11462 157 (1.35%)
group-members 10908769 3818079 7090690 (65.0%)
group-owners 118139 15257 102882 (87.09%)
key-vault-access-policies 1618 1484 134 (8.28%)
key-vaults 4242 3184 1058 (24.94%)
logic-apps 9511 9380 131 (1.38%)
managed-clusters 963 858 105 (10.9%)
management-group-descendants 4382 4148 234 (5.34%)
role-assignments 127938 91097 36841 (28.8%)
roles 158463 153294 5169 (3.26%)
service-principal-owners 461866 418982 42884 (9.28%)
service-principals 4217700 3630377 587323 (13.93%)
storage-accounts 16141 15168 973 (6.03%)
users 300195 128529 171666 (57.18%)
virtual-machines 284739 160451 124288 (43.65%)
vm-scale-sets 3498 3348 150 (4.29%)

Output size comparison by aggregate collection

Kind v2.1.6 bytes next bytes change
azurehound list 37814662 24781862 13032800 (34.46%)
azurehound list az-ad 27154960 14248851 12906109 (47.53%)
azurehound list az-rm 10659848 10533079 126769 (1.19%)

Copy link
Contributor

@superlinkx superlinkx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to the specific testing comment, do we have any other details on how this was tested against BHCE/BHE with datasets and whether the outcome was in line with v2.1.6?

models/utils.go Outdated Show resolved Hide resolved
@ddlees
Copy link
Contributor Author

ddlees commented Dec 8, 2023

In addition to the specific testing comment, do we have any other details on how this was tested against BHCE/BHE with datasets and whether the outcome was in line with v2.1.6?

Yes, a collection with this changeset was run against a remote BHE instance and local BHE/BHCE instances successfully without complication.

Copy link
Contributor

@superlinkx superlinkx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a couple of places in the json.RawMessages where there's a mix of tabs and spaces used for indentation. It would be nice to format those for consistency, but there's no practical problems here. Tests are very detailed and make a lot of sense, I don't see any obvious gaps here so I say ship it!

models/utils_test.go Outdated Show resolved Hide resolved
@ddlees ddlees merged commit b739550 into main Dec 11, 2023
9 checks passed
@ddlees ddlees deleted the bed-3954 branch December 11, 2023 19:51
@github-actions github-actions bot locked and limited conversation to collaborators Dec 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants