Skip to content

Commit

Permalink
Fixes #39: Add additional metadata to HAR
Browse files Browse the repository at this point in the history
  • Loading branch information
@baltpeter
baltpeter committed Apr 19, 2024
1 parent c8078a5 commit c0f90a3
Show file tree
Hide file tree
Showing 5 changed files with 302 additions and 45 deletions.
58 changes: 58 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,64 @@ The following example collects the traffic for an app in the Android emulator. I

Take a look at the [`examples/`](examples) folder for some more examples of how to use cyanoacrylate.

## Additional metadata in exported HAR files

The HAR files that cyanoacrylate produces contain additional metadata about the analysis. We use this metadata, for example, to generate technical reports and complaints using [ReportHAR](https://github.com/tweaselORG/ReportHAR).

Here's an example of what such a HAR might look like (the actual requests in the `entries` field are omitted for brevity):

```json5
{
"log": {
"version": "1.2",
"creator": {
"name": "cyanoacrylate",
"version": "1.0.0"
},
"pages": [],
"entries": [
// [requests and responses omitted]
],
"_tweasel": {
"startDate": "2024-04-19T08:52:48.077Z",
"endDate": "2024-04-19T08:52:59.214Z",
"options": {
"mode": "allowlist",
"apps": ["de.hafas.android.db"]
},
"device": {
"platform": "android",
"runTarget": "device",
"osVersion": "13",
"osBuild": "lineage_ocean-userdebug 13 TQ2A.230505.002 8c3345902f",
"manufacturer": "motorola",
"model": "moto g(7) power",
"modelCodeName": "ocean"
},
"versions": {
"appstraction": "1.0.0",
"cyanoacrylate": "1.0.0",
"node": "v18.13.0",
"python": "3.11.3",
"mitmproxy": "9.0.1"
},
"apps": [
{
"id": "de.hafas.android.db",
"name": "DB Navigator",
"version": "21.12.p03.04",
"versionCode": "211200007",
"architectures": []
}
],
"metaVersion": 1
}
}
}
```

See the [type definition](docs/README.md#tweaselharmetav1) for more details on which information is included.

## License

This code is licensed under the MIT license, see the [`LICENSE`](LICENSE) file for details.
Expand Down
142 changes: 116 additions & 26 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ cyanoacrylate
- [App](README.md#app)
- [AppAnalysis](README.md#appanalysis)
- [AppAnalysisResult](README.md#appanalysisresult)
- [Device](README.md#device)
- [DeviceAttribute](README.md#deviceattribute)
- [GetDeviceAttributeOptions](README.md#getdeviceattributeoptions)
- [IosPermission](README.md#iospermission)
Expand All @@ -28,6 +29,9 @@ cyanoacrylate
- [SupportedPlatform](README.md#supportedplatform)
- [SupportedRunTarget](README.md#supportedruntarget)
- [TrafficCollectionOptions](README.md#trafficcollectionoptions)
- [TweaselHar](README.md#tweaselhar)
- [TweaselHarMetaV1](README.md#tweaselharmetav1)
- [VersionMeta](README.md#versionmeta)

### Variables

Expand Down Expand Up @@ -66,11 +70,11 @@ Functions that can be used to instrument the device and analyze apps.
| `startAppAnalysis` | (`appIdOrPath`: `string` \| `AppPath`<`Platform`\>, `options?`: { `noSigint?`: `boolean` ; `resetApp?`: `boolean` }) => `Promise`<[`AppAnalysis`](README.md#appanalysis)<`Platform`, `RunTarget`, `Capabilities`\>\> | Start an app analysis. The app analysis is controlled through the returned object. Remember to call `stop()` on the object when you are done with the app to clean up and retrieve the analysis data. |
| `startTrafficCollection` | (`options?`: `Platform` extends ``"android"`` ? [`TrafficCollectionOptions`](README.md#trafficcollectionoptions) : `never`) => `Promise`<`void`\> | Start collecting the device's traffic. On Android, this will start a WireGuard proxy on the host computer on port `51820`. It will automatically configure the target to use the WireGuard proxy and trust the mitmproxy TLS certificate. You can configure which apps to include using the `options` parameter. On iOS, this will start a mitmproxy HTTP(S) proxy on the host computer on port `8080`. It will automatically configure the target to use the proxy and trust the mitmproxy TLS certificate. You can not restrict the traffic collection to specific apps. Only one traffic collection can be active at a time. |
| `stop` | () => `Promise`<`void`\> | Stop the analysis. This is important for clean up, e.g. stopping the emulator if it is managed by this library. |
| `stopTrafficCollection` | () => `Promise`<`Har`\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |
| `stopTrafficCollection` | () => `Promise`<[`TweaselHar`](README.md#tweaselhar)\> | Stop collecting the device's traffic. This will stop the proxy on the host computer. |

#### Defined in

[src/index.ts:48](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L48)
[src/index.ts:109](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L109)

___

Expand All @@ -90,7 +94,7 @@ The options for the `startAnalysis()` function.

#### Defined in

[src/index.ts:272](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L272)
[src/index.ts:333](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L333)

___

Expand All @@ -102,26 +106,29 @@ An ID of a known permission on Android.

#### Defined in

node_modules/appstraction/dist/index.d.ts:68
node_modules/appstraction/dist/index.d.ts:58

___

### App

Ƭ **App**: `Object`

Metadata about an app.
Metadata about an app, as returned by parseAppMeta.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `id` | `string` | The app's ID. |
| `version?` | `string` | The app's version. |
| `architectures` | (``"arm64"`` \| ``"arm"`` \| ``"x86"`` \| ``"x86_64"`` \| ``"mips"`` \| ``"mips64"``)[] | A list of the architectures that the app supports. The identifiers for the architectures are normalized across Android and iOS. On Android, this will be empty for apps that don't have native code. |
| `id` | `string` | The app/bundle ID. |
| `name?` | `string` | The app's display name. |
| `version?` | `string` | The app's human-readable version. |
| `versionCode?` | `string` | The app's version code. |

#### Defined in

[src/index.ts:32](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L32)
node_modules/appstraction/dist/index.d.ts:85

___

Expand Down Expand Up @@ -155,7 +162,7 @@ Functions that can be used to control an app analysis.

#### Defined in

[src/index.ts:121](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L121)
[src/index.ts:182](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L182)

___

Expand All @@ -171,17 +178,41 @@ The result of an app analysis.
| :------ | :------ | :------ |
| `app` | [`App`](README.md#app) | The app's metadata. |
| `mitmproxyEvents` | [`MitmproxyEvent`](README.md#mitmproxyevent)[] | The mitmproxy events that were observed during the traffic collection. Note that this is not a stable API. |
| `traffic` | `Record`<`string`, `Har`\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |
| `traffic` | `Record`<`string`, [`TweaselHar`](README.md#tweaselhar)\> | The collected traffic, accessible by the specified name. The traffic is available as a JSON object in the HAR format (https://w3c.github.io/web-performance/specs/HAR/Overview.html). |

#### Defined in

[src/index.ts:208](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L208)
[src/index.ts:269](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L269)

___

### Device

Ƭ **Device**: `Object`

Metadata about the device the analysis was run on.

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `manufacturer?` | `string` | The device's manufacturer. |
| `model?` | `string` | The device's model. |
| `modelCodeName?` | `string` | The device's model code name. |
| `osBuild?` | `string` | The build string of the OS. |
| `osVersion` | `string` | The version of the OS. |
| `platform` | [`SupportedPlatform`](README.md#supportedplatform) | The device's operating system. |
| `runTarget` | [`SupportedRunTarget`](README.md#supportedruntarget)<[`SupportedPlatform`](README.md#supportedplatform)\> | The type of device (emulator, physical device). |

#### Defined in

[src/index.ts:41](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L41)

___

### DeviceAttribute

Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? `never` : `Platform` extends ``"ios"`` ? ``"idfv"`` : `never`
Ƭ **DeviceAttribute**<`Platform`\>: `Platform` extends ``"android"`` ? ``"apiLevel"`` \| ``"manufacturer"`` \| ``"model"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"`` : ``"idfv"`` \| ``"manufacturer"`` \| ``"modelCodeName"`` \| ``"name"`` \| ``"osBuild"`` \| ``"osVersion"``

A supported attribute for the `getDeviceAttribute()` function, depending on the platform.

Expand All @@ -193,7 +224,7 @@ A supported attribute for the `getDeviceAttribute()` function, depending on the

#### Defined in

node_modules/appstraction/dist/index.d.ts:415
node_modules/appstraction/dist/index.d.ts:431

___

Expand All @@ -212,7 +243,7 @@ The options for each attribute available through the `getDeviceAttribute()` func

#### Defined in

node_modules/appstraction/dist/index.d.ts:417
node_modules/appstraction/dist/index.d.ts:433

___

Expand All @@ -224,7 +255,7 @@ An ID of a known permission on iOS.

#### Defined in

node_modules/appstraction/dist/index.d.ts:72
node_modules/appstraction/dist/index.d.ts:62

___

Expand Down Expand Up @@ -420,8 +451,8 @@ Functions that are available for the platforms.
| Name | Type | Description |
| :------ | :------ | :------ |
| `clearStuckModals` | `Platform` extends ``"android"`` ? () => `Promise`<`void`\> : `never` | Clear any potential stuck modals by pressing the back button followed by the home button. This is currently broken on iOS (see https://github.com/tweaselORG/appstraction/issues/12). Requires the `ssh` capability on iOS. |
| `ensureDevice` | () => `Promise`<`void`\> | Assert that the selected device is connected and ready to be used with the selected capabilities, performing necessary setup steps. This should always be the first function you call. Note that depending on the capabilities you set, the setup steps may make permanent changes to your device. |
| `getDeviceAttribute` | <Attribute\>(`attribute`: `Attribute`, ...`options`: `Attribute` extends keyof [`GetDeviceAttributeOptions`](README.md#getdeviceattributeoptions) ? [options: GetDeviceAttributeOptions[Attribute]] : [options?: undefined]) => `Promise`<`string`\> | Get the value of the given attribute of the device. Requires the `frida` capability on iOS. |
| `ensureDevice` | () => `Promise`<`void`\> | Assert that the selected device is connected and ready to be used with the selected capabilities, performing necessary setup steps. This should always be the first function you call. Note that depending on the capabilities you set, the setup steps may make permanent changes to your device. For Android, you can set the url to the WireGuard APK which should be installed in the `WIREGUARD_APK_URL` environment variable. Note that it is only used if WireGuard isn’t installed already. |
| `getDeviceAttribute` | <Attribute\>(`attribute`: `Attribute`, ...`options`: `Attribute` extends keyof [`GetDeviceAttributeOptions`](README.md#getdeviceattributeoptions) ? [options: GetDeviceAttributeOptions[Attribute]] : [options?: undefined]) => `Promise`<`string`\> | Get the value of the given device attribute. |
| `getForegroundAppId` | () => `Promise`<`string` \| `undefined`\> | Get the app ID of the running app that is currently in the foreground. Requires the `frida` capability on iOS. |
| `getPidForAppId` | (`appId`: `string`) => `Promise`<`number` \| `undefined`\> | Get the PID of the app with the given app ID if it is currently running. Requires the `frida` capability on iOS. |
| `getPrefs` | (`appId`: `string`) => `Promise`<`Record`<`string`, `unknown`\> \| `undefined`\> | Get the preferences (`SharedPreferences` on Android, `NSUserDefaults` on iOS) of the app with the given app ID. Requires the `frida` capability on Android and iOS. |
Expand All @@ -445,7 +476,7 @@ Functions that are available for the platforms.

#### Defined in

node_modules/appstraction/dist/index.d.ts:95
node_modules/appstraction/dist/index.d.ts:103

___

Expand Down Expand Up @@ -479,7 +510,7 @@ The options for a specific platform/run target combination.

#### Defined in

[src/index.ts:226](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L226)
[src/index.ts:287](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L287)

___

Expand All @@ -497,7 +528,7 @@ A capability supported by this library.

#### Defined in

[src/index.ts:25](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L25)
[src/index.ts:34](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L34)

___

Expand All @@ -509,7 +540,7 @@ A platform that is supported by this library.

#### Defined in

node_modules/appstraction/dist/index.d.ts:74
node_modules/appstraction/dist/index.d.ts:64

___

Expand All @@ -527,7 +558,7 @@ A run target that is supported by this library for the given platform.

#### Defined in

node_modules/appstraction/dist/index.d.ts:76
node_modules/appstraction/dist/index.d.ts:66

___

Expand All @@ -543,7 +574,66 @@ Options for a traffic collection that specifies which apps to collect traffic fr

#### Defined in

[src/index.ts:46](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L46)
[src/index.ts:107](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L107)

___

### TweaselHar

Ƭ **TweaselHar**: `Har` & { `log`: { `_tweasel`: [`TweaselHarMetaV1`](README.md#tweaselharmetav1) } }

A HAR file with additional tweasel metadata containing information about the analysis that the traffic was collected
through.

#### Defined in

[src/index.ts:70](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L70)

___

### TweaselHarMetaV1

Ƭ **TweaselHarMetaV1**: `Object`

Metadata about the traffic collection as included in a [TweaselHar](README.md#tweaselhar).

#### Type declaration

| Name | Type | Description |
| :------ | :------ | :------ |
| `apps?` | [`App`](README.md#app)[] | Details about the app(s) that was/were analyzed. Currently only populated if the traffic was recorded through an app analysis. |
| `device` | [`Device`](README.md#device) | Details about the device that the analysis was run on. |
| `endDate` | `string` | The time and date at which the traffic collection was stopped. |
| `metaVersion` | ``1`` | The version of the tweasel-specific metadata format. Currently, `1` is the only version. If the format is ever changed or extended in the future, this version will be incremented. |
| `options` | [`TrafficCollectionOptions`](README.md#trafficcollectionoptions) | The options that were used for the traffic collection. |
| `startDate` | `string` | The time and date at which the traffic collection was started. |
| `versions` | [`VersionMeta`](README.md#versionmeta) | The versions of the dependencies used in the analysis. |

#### Defined in

[src/index.ts:77](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L77)

___

### VersionMeta

Ƭ **VersionMeta**: `Object`

The versions of the dependencies used in the analysis.

#### Type declaration

| Name | Type |
| :------ | :------ |
| `appstraction` | `string` |
| `cyanoacrylate` | `string` |
| `mitmproxy` | `string` |
| `node` | `string` |
| `python` | `string` |

#### Defined in

[src/index.ts:58](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L58)

## Variables

Expand All @@ -555,7 +645,7 @@ The IDs of known permissions on Android.

#### Defined in

node_modules/appstraction/dist/index.d.ts:66
node_modules/appstraction/dist/index.d.ts:56

___

Expand All @@ -567,7 +657,7 @@ The IDs of known permissions on iOS.

#### Defined in

node_modules/appstraction/dist/index.d.ts:70
node_modules/appstraction/dist/index.d.ts:60

## Functions

Expand Down Expand Up @@ -622,4 +712,4 @@ An object that can be used to instrument the device and analyze apps.

#### Defined in

[src/index.ts:306](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L306)
[src/index.ts:367](https://github.com/tweaselORG/cyanoacrylate/blob/main/src/index.ts#L367)
2 changes: 1 addition & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
"dependencies": {
"@types/har-format": "^1.2.10",
"andromatic": "^1.1.1",
"appstraction": "^1.0.0",
"appstraction": "file:.yalc/appstraction",
"autopy": "^1.1.1",
"cross-fetch": "^3.1.5",
"ctrlc-windows": "^2.1.0",
Expand Down
Loading

0 comments on commit c0f90a3

Please sign in to comment.