Skip to content

Commit

Permalink
Fixes #39: Add additional metadata to HAR
Browse files Browse the repository at this point in the history
  • Loading branch information
@baltpeter
baltpeter committed May 13, 2024
1 parent 2ba1e02 commit ae87d5e
Show file tree
Hide file tree
Showing 6 changed files with 313 additions and 49 deletions.
61 changes: 61 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,67 @@ The following example collects the traffic for an app in the Android emulator. I

Take a look at the [`examples/`](examples) folder for some more examples of how to use cyanoacrylate.

## Additional metadata in exported HAR files

The HAR files that cyanoacrylate produces contain additional metadata about the analysis. We use this metadata, for example, to generate technical reports and complaints using [ReportHAR](https://github.com/tweaselORG/ReportHAR).

Here's an example of what such a HAR might look like (the actual requests in the `entries` field are omitted for brevity):

```json5
{
"log": {
"version": "1.2",
"creator": {
"name": "cyanoacrylate",
"version": "1.0.0"
},
"pages": [],
"entries": [
// [requests and responses omitted]
],
"_tweasel": {
"startDate": "2024-04-19T08:52:48.077Z",
"endDate": "2024-04-19T08:52:59.214Z",
"options": {
"mode": "allowlist",
"apps": ["de.hafas.android.db"]
},
"device": {
"platform": "android",
"runTarget": "device",
"osVersion": "13",
"osBuild": "lineage_ocean-userdebug 13 TQ2A.230505.002 8c3345902f",
"manufacturer": "motorola",
"model": "moto g(7) power",
"modelCodeName": "ocean",
"architectures": "arm64-v8a,armeabi-v7a,armeabi"
},
"versions": {
"appstraction": "1.0.0",
"cyanoacrylate": "1.0.0",
"node": "v18.13.0",
"python": "3.11.3",
"mitmproxy": "9.0.1"
},
"apps": [
{
"platform": "android",
"id": "de.hafas.android.db",
"name": "DB Navigator",
"version": "21.12.p03.04",
"versionCode": "211200007",
"architectures": [],
"md5": "4f4cf346a050ea23da0da60328a4dd10"
}
],
"metaVersion": 1
}
}
}
```

See the [type definition](docs/README.md#tweaselharmetav1) for more details on which information is included.

## License

This code is licensed under the MIT license, see the [`LICENSE`](LICENSE) file for details.
Expand Down
133 changes: 108 additions & 25 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ cyanoacrylate
- [App](README.md#app)
- [AppAnalysis](README.md#appanalysis)
- [AppAnalysisResult](README.md#appanalysisresult)
- [Device](README.md#device)
- [DeviceAttribute](README.md#deviceattribute)
- [GetDeviceAttributeOptions](README.md#getdeviceattributeoptions)
- [IosPermission](README.md#iospermission)
Expand All @@ -28,6 +29,9 @@ cyanoacrylate
- [SupportedPlatform](README.md#supportedplatform)
- [SupportedRunTarget](README.md#supportedruntarget)
- [TrafficCollectionOptions](README.md#trafficcollectionoptions)
- [TweaselHar](README.md#tweaselhar)
- [TweaselHarMetaV1](README.md#tweaselharmetav1)
- [VersionMeta](README.md#versionmeta)

### Variables

Expand Down Expand Up @@ -66,11 +70,11 @@ Functions that can be used to instrument the device and analyze apps.
| `startAppAnalysis` | (`appIdOrPath`: `string` \| `AppPath`<`Platform`\>, `options?`: { `noSigint?`: `boolean` ; `resetApp?`: `boolean` }) => `Promise`<[`AppAnalysis`](README.md#appanalysis)<`Platform`, `RunTarget`, `Capabilities`\>\> | Start an app analysis. The app analysis is controlled through the returned object. Remember to call `stop()` on the object when you are done with the app to clean up and retrieve the analysis data. |
| `startTrafficCollection` | (`options?`: `Platform` extends ``"android"`` ? [`TrafficCollectionOptions`](README.md#trafficcollectionoptions) : `never`) => `Promise`<`void`\> | Start collecting the device's traffic. On Android, this will start a WireGuard proxy on the host computer on port `51820`. It will automatically configure the target to use the WireGuard proxy and trust the mitmproxy TLS certificate. You can configure which apps to include using the `options` parameter. On iOS, this will start a mitmproxy HTTP(S) proxy on the host computer on port `8080`. It will automatically configure the target to use the proxy and trust the mitmproxy TLS certificate. You can not restrict the traffic collection to specific apps. Only one traffic collection can be active at a time. |
| `stop` | () => `Promise`<`void`\> | Stop the analysis. This is important for clean up, e.g. stopping the emulator if it is managed by this library. |
| `stopTrafficCollection` | () => `Promise`<`Har`\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |
| `stopTrafficCollection` | () => `Promise`<[`TweaselHar`](README.md#tweaselhar)\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |

#### Defined in

[src/index.ts:48](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L48)
[src/index.ts:105](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L105)

___

Expand All @@ -90,7 +94,7 @@ The options for the `startAnalysis()` function.

#### Defined in

[src/index.ts:272](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L272)
[src/index.ts:329](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L329)

___

Expand All @@ -102,26 +106,31 @@ An ID of a known permission on Android.

#### Defined in

node_modules/appstraction/dist/index.d.ts:68
node_modules/appstraction/dist/index.d.ts:58

___

### App

Ƭ **App**: `Object`

Metadata about an app.
Metadata about an app, as returned by parseAppMeta.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `id` | `string` | The app's ID. |
| `version?` | `string` | The app's version. |
| `architectures` | (``"arm64"`` \| ``"arm"`` \| ``"x86"`` \| ``"x86_64"`` \| ``"mips"`` \| ``"mips64"``)[] | A list of the architectures that the app supports. The identifiers for the architectures are normalized across Android and iOS. On Android, this will be empty for apps that don't have native code. |
| `id` | `string` | The app/bundle ID. |
| `md5?` | `string` | The MD5 hash of the app's package file. In the case of split APKs on Android, this will be the hash of the main APK. In the case of custom APK bundle formats (`.xapk`, `.apkm` and `.apks`), this will be the hash of the entire bundle. **Be careful when interpreting this value.** App stores can deliver different distributions of the exact same app. For example, apps downloaded from the App Store on iOS include the user's Apple ID, thus leading to different hashes even if different users download the very same version of the same app. |
| `name?` | `string` | The app's display name. |
| `platform` | [`SupportedPlatform`](README.md#supportedplatform) | The platform the app is for. |
| `version?` | `string` | The app's human-readable version. |
| `versionCode?` | `string` | The app's version code. |

#### Defined in

[src/index.ts:32](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L32)
node_modules/appstraction/dist/index.d.ts:85

___

Expand Down Expand Up @@ -155,7 +164,7 @@ Functions that can be used to control an app analysis.

#### Defined in

[src/index.ts:121](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L121)
[src/index.ts:178](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L178)

___

Expand All @@ -171,17 +180,42 @@ The result of an app analysis.
| :------ | :------ | :------ |
| `app` | [`App`](README.md#app) | The app's metadata. |
| `mitmproxyEvents` | [`MitmproxyEvent`](README.md#mitmproxyevent)[] | The mitmproxy events that were observed during the traffic collection. Note that this is not a stable API. |
| `traffic` | `Record`<`string`, `Har`\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |
| `traffic` | `Record`<`string`, [`TweaselHar`](README.md#tweaselhar)\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |

#### Defined in

[src/index.ts:208](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L208)
[src/index.ts:265](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L265)

___

### Device

Ƭ **Device**: `Object`

Metadata about the device the analysis was run on.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `architectures` | `string` | Architectures/ABIs supported by the device. |
| `manufacturer?` | `string` | The device's manufacturer. |
| `model?` | `string` | The device's model. |
| `modelCodeName?` | `string` | The device's model code name. |
| `osBuild?` | `string` | The build string of the OS. |
| `osVersion` | `string` | The version of the OS. |
| `platform` | [`SupportedPlatform`](README.md#supportedplatform) | The device's operating system. |
| `runTarget` | [`SupportedRunTarget`](README.md#supportedruntarget)<[`SupportedPlatform`](README.md#supportedplatform)\> | The type of device (emulator, physical device). |

#### Defined in

[src/index.ts:41](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L41)

___

### DeviceAttribute

Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? `never` : `Platform` extends ``"ios"`` ? ``"idfv"`` : `never`
Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? ``"apiLevel"`` \| ``"architectures"`` \| ``"manufacturer"`` \| ``"model"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"`` : ``"architectures"`` \| ``"idfv"`` \| ``"manufacturer"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"``

A supported attribute for the `getDeviceAttribute()` function, depending on the platform.

Expand All @@ -193,7 +227,7 @@ A supported attribute for the `getDeviceAttribute()` function, depending on the

#### Defined in

node_modules/appstraction/dist/index.d.ts:418
node_modules/appstraction/dist/index.d.ts:445

___

Expand All @@ -212,7 +246,7 @@ The options for each attribute available through the `getDeviceAttribute()` func

#### Defined in

node_modules/appstraction/dist/index.d.ts:420
node_modules/appstraction/dist/index.d.ts:447

___

Expand All @@ -224,7 +258,7 @@ An ID of a known permission on iOS.

#### Defined in

node_modules/appstraction/dist/index.d.ts:72
node_modules/appstraction/dist/index.d.ts:62

___

Expand Down Expand Up @@ -421,7 +455,7 @@ Functions that are available for the platforms.
| :------ | :------ | :------ |
| `clearStuckModals` | `Platform` extends ``"android"`` ? () => `Promise`<`void`\> : `never` | Clear any potential stuck modals by pressing the back button followed by the home button. This is currently broken on iOS (see https://github.com/tweaselORG/appstraction/issues/12). Requires the `ssh` capability on iOS. |
| `ensureDevice` | () => `Promise`<`void`\> | Assert that the selected device is connected and ready to be used with the selected capabilities, performing necessary setup steps. This should always be the first function you call. Note that depending on the capabilities you set, the setup steps may make permanent changes to your device. For Android, you can set the url to the WireGuard APK which should be installed in the `WIREGUARD_APK_URL` environment variable. Note that it is only used if WireGuard isn’t installed already. |
| `getDeviceAttribute` | <Attribute\>(`attribute`: `Attribute`, ...`options`: `Attribute` extends keyof [`GetDeviceAttributeOptions`](README.md#getdeviceattributeoptions) ? [options: GetDeviceAttributeOptions[Attribute]] : [options?: undefined]) => `Promise`<`string`\> | Get the value of the given attribute of the device. Requires the `frida` capability on iOS. |
| `getDeviceAttribute` | <Attribute\>(`attribute`: `Attribute`, ...`options`: `Attribute` extends keyof [`GetDeviceAttributeOptions`](README.md#getdeviceattributeoptions) ? [options: GetDeviceAttributeOptions[Attribute]] : [options?: undefined]) => `Promise`<`string`\> | Get the value of the given device attribute. |
| `getForegroundAppId` | () => `Promise`<`string` \| `undefined`\> | Get the app ID of the running app that is currently in the foreground. Requires the `frida` capability on iOS. |
| `getPidForAppId` | (`appId`: `string`) => `Promise`<`number` \| `undefined`\> | Get the PID of the app with the given app ID if it is currently running. Requires the `frida` capability on iOS. |
| `getPrefs` | (`appId`: `string`) => `Promise`<`Record`<`string`, `unknown`\> \| `undefined`\> | Get the preferences (`SharedPreferences` on Android, `NSUserDefaults` on iOS) of the app with the given app ID. Requires the `frida` capability on Android and iOS. |
Expand All @@ -445,7 +479,7 @@ Functions that are available for the platforms.

#### Defined in

node_modules/appstraction/dist/index.d.ts:95
node_modules/appstraction/dist/index.d.ts:116

___

Expand Down Expand Up @@ -479,7 +513,7 @@ The options for a specific platform/run target combination.

#### Defined in

[src/index.ts:226](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L226)
[src/index.ts:283](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L283)

___

Expand All @@ -497,7 +531,7 @@ A capability supported by this library.

#### Defined in

[src/index.ts:25](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L25)
[src/index.ts:34](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L34)

___

Expand All @@ -509,7 +543,7 @@ A platform that is supported by this library.

#### Defined in

node_modules/appstraction/dist/index.d.ts:74
node_modules/appstraction/dist/index.d.ts:64

___

Expand All @@ -527,7 +561,7 @@ A run target that is supported by this library for the given platform.

#### Defined in

node_modules/appstraction/dist/index.d.ts:76
node_modules/appstraction/dist/index.d.ts:66

___

Expand All @@ -543,7 +577,56 @@ Options for a traffic collection that specifies which apps to collect traffic fr

#### Defined in

[src/index.ts:46](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L46)
[src/index.ts:103](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L103)

___

### TweaselHar

Ƭ **TweaselHar**: `Har` & { `log`: { `_tweasel`: [`TweaselHarMetaV1`](README.md#tweaselharmetav1) } }

A HAR file with additional tweasel metadata containing information about the analysis that the traffic was collected
through.

#### Defined in

[src/index.ts:66](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L66)

___

### TweaselHarMetaV1

Ƭ **TweaselHarMetaV1**: `Object`

Metadata about the traffic collection as included in a [TweaselHar](README.md#tweaselhar).

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `apps?` | [`App`](README.md#app)[] | Details about the app(s) that was/were analyzed. Currently only populated if the traffic was recorded through an app analysis. |
| `device` | [`Device`](README.md#device) | Details about the device that the analysis was run on. |
| `endDate` | `string` | The time and date at which the traffic collection was stopped. |
| `metaVersion` | ``"1.0"`` | The version of the tweasel-specific metadata format. Currently, `1.0` is the only version. If the format is ever changed or extended in the future, this version will be incremented. |
| `options` | [`TrafficCollectionOptions`](README.md#trafficcollectionoptions) | The options that were used for the traffic collection. |
| `startDate` | `string` | The time and date at which the traffic collection was started. |
| `versions` | [`VersionMeta`](README.md#versionmeta) | The versions of the dependencies used in the analysis. |

#### Defined in

[src/index.ts:73](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L73)

___

### VersionMeta

Ƭ **VersionMeta**: `Record`<`string`, `string`\>

The versions of the dependencies used in the analysis.

#### Defined in

[src/index.ts:60](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L60)

## Variables

Expand All @@ -555,7 +638,7 @@ The IDs of known permissions on Android.

#### Defined in

node_modules/appstraction/dist/index.d.ts:66
node_modules/appstraction/dist/index.d.ts:56

___

Expand All @@ -567,7 +650,7 @@ The IDs of known permissions on iOS.

#### Defined in

node_modules/appstraction/dist/index.d.ts:70
node_modules/appstraction/dist/index.d.ts:60

## Functions

Expand Down Expand Up @@ -622,4 +705,4 @@ An object that can be used to instrument the device and analyze apps.

#### Defined in

[src/index.ts:306](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L306)
[src/index.ts:363](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L363)
2 changes: 1 addition & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
"dependencies": {
"@types/har-format": "^1.2.10",
"andromatic": "^1.1.1",
"appstraction": "^1.0.1",
"appstraction": "file:.yalc/appstraction",
"autopy": "^1.1.1",
"cross-fetch": "^3.1.5",
"ctrlc-windows": "^2.1.0",
Expand Down
Loading

0 comments on commit ae87d5e

Please sign in to comment.