Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #39: Add additional metadata to HAR #41

Merged
merged 2 commits into from
Jun 7, 2024
Merged

Fixes #39: Add additional metadata to HAR #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
967cf29
Fixes #39: Add additional metadata to HAR
baltpeter Apr 18, 2024
eaaaacf
squash me: Use new appstraction release and pin subtypes
zner0L Jun 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,67 @@ The following example collects the traffic for an app in the Android emulator. I

Take a look at the [`examples/`](examples) folder for some more examples of how to use cyanoacrylate.

## Additional metadata in exported HAR files

The HAR files that cyanoacrylate produces contain additional metadata about the analysis. We use this metadata, for example, to generate technical reports and complaints using [ReportHAR](https://github.com/tweaselORG/ReportHAR).

Here's an example of what such a HAR might look like (the actual requests in the `entries` field are omitted for brevity):

```json5
{
"log": {
"version": "1.2",
"creator": {
"name": "cyanoacrylate",
"version": "1.0.0"
},
"pages": [],
"entries": [
// [requests and responses omitted]
],
"_tweasel": {
"startDate": "2024-04-19T08:52:48.077Z",
"endDate": "2024-04-19T08:52:59.214Z",
"options": {
"mode": "allowlist",
"apps": ["de.hafas.android.db"]
},
"device": {
"platform": "android",
"runTarget": "device",
"osVersion": "13",
"osBuild": "lineage_ocean-userdebug 13 TQ2A.230505.002 8c3345902f",
"manufacturer": "motorola",
"model": "moto g(7) power",
"modelCodeName": "ocean",
"architectures": "arm64-v8a,armeabi-v7a,armeabi"
},
"versions": {
"appstraction": "1.0.0",
"cyanoacrylate": "1.0.0",
"node": "v18.13.0",
"python": "3.11.3",
"mitmproxy": "9.0.1"
},
"apps": [
{
"platform": "android",
"id": "de.hafas.android.db",
"name": "DB Navigator",
"version": "21.12.p03.04",
"versionCode": "211200007",
"architectures": [],
"md5": "4f4cf346a050ea23da0da60328a4dd10"
}
],
"metaVersion": "1.0"
}
}
}
```

See the [type definition](docs/README.md#tweaselharmetav1) for more details on which information is included.

## License

This code is licensed under the MIT license, see the [`LICENSE`](LICENSE) file for details.
Expand Down
112 changes: 91 additions & 21 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ cyanoacrylate
- [AppAnalysis](README.md#appanalysis)
- [AppAnalysisResult](README.md#appanalysisresult)
- [DeviceAttribute](README.md#deviceattribute)
- [DeviceV1](README.md#devicev1)
- [GetDeviceAttributeOptions](README.md#getdeviceattributeoptions)
- [IosPermission](README.md#iospermission)
- [MitmproxyCertificate](README.md#mitmproxycertificate)
Expand All @@ -27,7 +28,9 @@ cyanoacrylate
- [SupportedCapability](README.md#supportedcapability)
- [SupportedPlatform](README.md#supportedplatform)
- [SupportedRunTarget](README.md#supportedruntarget)
- [TrafficCollectionOptions](README.md#trafficcollectionoptions)
- [TrafficCollectionOptionsV1](README.md#trafficcollectionoptionsv1)
- [TweaselHar](README.md#tweaselhar)
- [TweaselHarMetaV1](README.md#tweaselharmetav1)

### Variables

Expand Down Expand Up @@ -64,13 +67,13 @@ Functions that can be used to instrument the device and analyze apps.
| `platform` | [`PlatformApi`](README.md#platformapi)<`Platform`, `RunTarget`, `Capabilities`\> | A raw platform API object as returned by [appstraction](https://github.com/tweaselORG/appstraction). |
| `resetDevice` | () => `Promise`<`void`\> | Reset the specified device to the snapshot specified in `targetOptions.snapshotName`. |
| `startAppAnalysis` | (`appIdOrPath`: `string` \| `AppPath`<`Platform`\>, `options?`: { `noSigint?`: `boolean` ; `resetApp?`: `boolean` }) => `Promise`<[`AppAnalysis`](README.md#appanalysis)<`Platform`, `RunTarget`, `Capabilities`\>\> | Start an app analysis. The app analysis is controlled through the returned object. Remember to call `stop()` on the object when you are done with the app to clean up and retrieve the analysis data. |
| `startTrafficCollection` | (`options?`: `Platform` extends ``"android"`` ? [`TrafficCollectionOptions`](README.md#trafficcollectionoptions) : `never`) => `Promise`<`void`\> | Start collecting the device's traffic. On Android, this will start a WireGuard proxy on the host computer on port `51820`. It will automatically configure the target to use the WireGuard proxy and trust the mitmproxy TLS certificate. You can configure which apps to include using the `options` parameter. On iOS, this will start a mitmproxy HTTP(S) proxy on the host computer on port `8080`. It will automatically configure the target to use the proxy and trust the mitmproxy TLS certificate. You can not restrict the traffic collection to specific apps. Only one traffic collection can be active at a time. |
| `startTrafficCollection` | (`options?`: `Platform` extends ``"android"`` ? `TrafficCollectionOptions` : `never`) => `Promise`<`void`\> | Start collecting the device's traffic. On Android, this will start a WireGuard proxy on the host computer on port `51820`. It will automatically configure the target to use the WireGuard proxy and trust the mitmproxy TLS certificate. You can configure which apps to include using the `options` parameter. On iOS, this will start a mitmproxy HTTP(S) proxy on the host computer on port `8080`. It will automatically configure the target to use the proxy and trust the mitmproxy TLS certificate. You can not restrict the traffic collection to specific apps. Only one traffic collection can be active at a time. |
| `stop` | () => `Promise`<`void`\> | Stop the analysis. This is important for clean up, e.g. stopping the emulator if it is managed by this library. |
| `stopTrafficCollection` | () => `Promise`<`Har`\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |
| `stopTrafficCollection` | () => `Promise`<[`TweaselHar`](README.md#tweaselhar)\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |

#### Defined in

[src/index.ts:48](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L48)
[src/index.ts:104](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L104)

___

Expand All @@ -90,7 +93,7 @@ The options for the `startAnalysis()` function.

#### Defined in

[src/index.ts:272](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L272)
[src/index.ts:328](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L328)

___

Expand All @@ -110,18 +113,23 @@ ___

Ƭ **App**: `Object`

Metadata about an app.
Metadata about an app, as returned by parseAppMeta.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `id` | `string` | The app's ID. |
| `version?` | `string` | The app's version. |
| `architectures` | (``"arm64"`` \| ``"arm"`` \| ``"x86"`` \| ``"x86_64"`` \| ``"mips"`` \| ``"mips64"``)[] | A list of the architectures that the app supports. The identifiers for the architectures are normalized across Android and iOS. On Android, this will be empty for apps that don't have native code. |
| `id` | `string` | The app/bundle ID. |
| `md5?` | `string` | The MD5 hash of the app's package file. In the case of split APKs on Android, this will be the hash of the main APK. In the case of custom APK bundle formats (`.xapk`, `.apkm` and `.apks`), this will be the hash of the entire bundle. **Be careful when interpreting this value.** App stores can deliver different distributions of the exact same app. For example, apps downloaded from the App Store on iOS include the user's Apple ID, thus leading to different hashes even if different users download the very same version of the same app. |
| `name?` | `string` | The app's display name. |
| `platform` | [`SupportedPlatform`](README.md#supportedplatform) | The platform the app is for. |
| `version?` | `string` | The app's human-readable version. |
| `versionCode?` | `string` | The app's version code. |

#### Defined in

[src/index.ts:32](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L32)
node_modules/appstraction/dist/index.d.ts:85

___

Expand Down Expand Up @@ -155,7 +163,7 @@ Functions that can be used to control an app analysis.

#### Defined in

[src/index.ts:121](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L121)
[src/index.ts:177](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L177)

___

Expand All @@ -171,17 +179,17 @@ The result of an app analysis.
| :------ | :------ | :------ |
| `app` | [`App`](README.md#app) | The app's metadata. |
| `mitmproxyEvents` | [`MitmproxyEvent`](README.md#mitmproxyevent)[] | The mitmproxy events that were observed during the traffic collection. Note that this is not a stable API. |
| `traffic` | `Record`<`string`, `Har`\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |
| `traffic` | `Record`<`string`, [`TweaselHar`](README.md#tweaselhar)\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |

#### Defined in

[src/index.ts:208](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L208)
[src/index.ts:264](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L264)

___

### DeviceAttribute

Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? ``"apiLevel"`` \| ``"manufacturer"`` \| ``"model"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"`` : ``"idfv"`` \| ``"manufacturer"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"``
Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? ``"apiLevel"`` \| ``"architectures"`` \| ``"manufacturer"`` \| ``"model"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"`` : ``"architectures"`` \| ``"idfv"`` \| ``"manufacturer"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"``

A supported attribute for the `getDeviceAttribute()` function, depending on the platform.

Expand All @@ -193,7 +201,32 @@ A supported attribute for the `getDeviceAttribute()` function, depending on the

#### Defined in

node_modules/appstraction/dist/index.d.ts:444
node_modules/appstraction/dist/index.d.ts:445

___

### DeviceV1

Ƭ **DeviceV1**: `Object`

Metadata about the device the analysis was run on.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `architectures` | `string` | Architectures/ABIs supported by the device. |
| `manufacturer?` | `string` | The device's manufacturer. |
| `model?` | `string` | The device's model. |
| `modelCodeName?` | `string` | The device's model code name. |
| `osBuild?` | `string` | The build string of the OS. |
| `osVersion` | `string` | The version of the OS. |
| `platform` | [`SupportedPlatform`](README.md#supportedplatform) | The device's operating system. |
| `runTarget` | [`SupportedRunTarget`](README.md#supportedruntarget)<[`SupportedPlatform`](README.md#supportedplatform)\> | The type of device (emulator, physical device). |

#### Defined in

[src/index.ts:41](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L41)

___

Expand All @@ -212,7 +245,7 @@ The options for each attribute available through the `getDeviceAttribute()` func

#### Defined in

node_modules/appstraction/dist/index.d.ts:446
node_modules/appstraction/dist/index.d.ts:447

___

Expand Down Expand Up @@ -479,7 +512,7 @@ The options for a specific platform/run target combination.

#### Defined in

[src/index.ts:226](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L226)
[src/index.ts:282](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L282)

___

Expand All @@ -497,7 +530,7 @@ A capability supported by this library.

#### Defined in

[src/index.ts:25](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L25)
[src/index.ts:34](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L34)

___

Expand Down Expand Up @@ -531,9 +564,9 @@ node_modules/appstraction/dist/index.d.ts:66

___

### TrafficCollectionOptions
### TrafficCollectionOptionsV1

Ƭ **TrafficCollectionOptions**: { `mode`: ``"all-apps"`` } \| { `apps`: `string`[] ; `mode`: ``"allowlist"`` \| ``"denylist"`` }
Ƭ **TrafficCollectionOptionsV1**: { `mode`: ``"all-apps"`` } \| { `apps`: `string`[] ; `mode`: ``"allowlist"`` \| ``"denylist"`` }

Options for a traffic collection that specifies which apps to collect traffic from.

Expand All @@ -543,7 +576,44 @@ Options for a traffic collection that specifies which apps to collect traffic fr

#### Defined in

[src/index.ts:46](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L46)
[src/index.ts:101](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L101)

___

### TweaselHar

Ƭ **TweaselHar**: `Har` & { `log`: { `_tweasel`: [`TweaselHarMetaV1`](README.md#tweaselharmetav1) } }

A HAR file with additional tweasel metadata containing information about the analysis that the traffic was collected
through.

#### Defined in

[src/index.ts:63](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L63)

___

### TweaselHarMetaV1

Ƭ **TweaselHarMetaV1**: `Object`

Metadata about the traffic collection as included in a [TweaselHar](README.md#tweaselhar).

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `apps?` | [`App`](README.md#app)[] | Details about the app(s) that was/were analyzed. Currently only populated if the traffic was recorded through an app analysis. |
| `device` | [`DeviceV1`](README.md#devicev1) | Details about the device that the analysis was run on. |
| `endDate` | `string` | The time and date at which the traffic collection was stopped. |
| `metaVersion` | ``"1.0"`` | The version of the tweasel-specific metadata format. Currently, `1.0` is the only version. If the format is ever changed or extended in the future, this version will be incremented. |
| `options` | [`TrafficCollectionOptionsV1`](README.md#trafficcollectionoptionsv1) | The options that were used for the traffic collection. |
| `startDate` | `string` | The time and date at which the traffic collection was started. |
| `versions` | `Record`<`string`, `string`\> | The versions of the dependencies used in the analysis. |

#### Defined in

[src/index.ts:70](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L70)

## Variables

Expand Down Expand Up @@ -622,4 +692,4 @@ An object that can be used to instrument the device and analyze apps.

#### Defined in

[src/index.ts:306](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L306)
[src/index.ts:362](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L362)
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
"dependencies": {
"@types/har-format": "^1.2.10",
"andromatic": "^1.1.1",
"appstraction": "^1.0.1",
"appstraction": "^1.2.0",
"autopy": "^1.1.1",
"cross-fetch": "^3.1.5",
"ctrlc-windows": "^2.1.0",
Expand Down
Loading