Skip to content

Commit

Permalink
Merge pull request #12 from Q42/bugfix/float-serialization-inconsistency
Browse files Browse the repository at this point in the history
Bugfix/float serialization inconsistency
  • Loading branch information
Techwolf12 authored Sep 14, 2022
2 parents 3b10cc8 + b2a61db commit d3123e9
Show file tree
Hide file tree
Showing 9 changed files with 176 additions and 78 deletions.
97 changes: 69 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# Q42Stats.Android
# Q42Stats.Android

[![Release](https://jitpack.io/v/Q42/Q42Stats.Android.svg)](https://jitpack.io/#Q42/Q42Stats.Android)
[![](https://jitci.com/gh/Q42/Q42Stats.Android/svg)](https://jitci.com/gh/Q42/Q42Stats.Android)


Collect stats for Q42 internal usage, shared across multiple Android projects.

An iOS version is also available: https://github.com/Q42/Q42Stats

## Installation

Add the Jitpack repo and include the library:

```gradle
Expand Down Expand Up @@ -43,30 +44,44 @@ Add the Jitpack repo and include the library:
}
}
```
This can be safely called from the main thread since all work (both collecting statistics and sending them to the server) are done on an IO thread.
This can be safely called from the main thread since all work (both collecting statistics and sending them to the server) are done on an IO thread.

It is safe to call this function multiple times, as it will exit immediately if it is already running or when a data collection interval has not passed yet.
It is safe to call this function multiple times, as it will exit immediately if it is already
running or when a data collection interval has not passed yet.

### Debug Logging
By default, Q42Stats only logs errors. For debugging purposes, set the log level before using Q42Stats:

By default, Q42Stats only logs errors. For debugging purposes, set the log level before using
Q42Stats:

```
Q42Stats.logLevel = Q42StatsLogLevel.Debug
```
## Performance
- Q42Stats is tiny, because it only depends on Kotlin. The library size is about 50kB.
- Data consumption is also very modest. For each `minimumSubmitInterval` that you configure, about
5kB of data is transferred.
- Data collection is run on an IO thread, so it doesn't block your application.
## Data collected
Not all fields are supported on all versions of Android. If unsupported, the corresponding key is omitted.
Q42Stats does not collect any personally identifiably information and is fully GDPR compliant. This
has been verified by legal counsel. It should not be necesaary to ask users for permission before
invoking Q42Stats.
Below is a listing of all information gathered by Q42Stats. Not all fields are supported on all
versions of Android. If unsupported, the corresponding key is omitted.
### Accessibliity
| Key | Value | Notes |
|-|-|-|
| `isAccessibilityManagerEnabled` | bool | true when any accessibility service (eg. Talkback) is Enabled |
| `isClosedCaptioningEnabled` | bool | Live transcription of any spoken audio (min sdk 19) |
| `isTouchExplorationEnabled` | bool | Whether any assistive feature is enabled where the user navigates the interface by touch. Most probably TalkbBack, or similar
| `isTalkBackEnabled` | bool | iOS: VoiceOver
| `isSamsungTalkBackEnabled` | bool | Specifically checks whether com.samsung.android.app.talkback.talkbackservice is enabled
| Key | Value | Notes | |-|-|-| | `isAccessibilityManagerEnabled` | bool | true when any
accessibility service (eg. Talkback) is Enabled | | `isClosedCaptioningEnabled` | bool | Live
transcription of any spoken audio (min sdk 19) | | `isTouchExplorationEnabled` | bool | Whether any
assistive feature is enabled where the user navigates the interface by touch. Most probably
TalkbBack, or similar | `isTalkBackEnabled` | bool | iOS: VoiceOver | `isSamsungTalkBackEnabled` |
bool | Specifically checks whether com.samsung.android.app.talkback.talkbackservice is enabled
| `isSelectToSpeakEnabled` | bool | iOS: Speak Selection
| `isSwitchAccessEnabled` | bool | Control the device by a switch such as a foot pedal
| `isBrailleBackEnabled` | bool | Navigate the screen with an external Braille display
Expand All @@ -92,34 +107,60 @@ Not all fields are supported on all versions of Android. If unsupported, the cor
### System
| Key | Value | Notes |
|-|-|-|
| `applicationId` | String | identifier for the app for which data is collected, as set in the app's Manifest. iOS: bundleId | nl.hema.mobiel |
| `defaultLanguage`| en, nl, ... |
| `sdkVersion` | int | 29 for Android 10. [See this list](https://source.android.com/setup/start/build-numbers)
|`manufacturer`|String|eg. `samsung`|
|`modelName`|String| May be a marketing name, but more often an internal code name. eg. `SM-G980F` for a particular variant of a Samsung Galaxy S10|
| Key | Value | Notes | |-|-|-| | `applicationId` | String | identifier for the app for which data
is collected, as set in the app's Manifest. iOS: bundleId | nl.hema.mobiel | | `defaultLanguage`|
en, nl, ... | | `sdkVersion` | int | 29 for Android
10. [See this list](https://source.android.com/setup/start/build-numbers)
|`manufacturer`|String|eg. `samsung`| |`modelName`|String| May be a marketing name, but more often
an internal code name. eg. `SM-G980F` for a particular variant of a Samsung Galaxy S10|
## Development
All classes and functions that are not used by implementing apps should have `internal` visibility.
Since this a library, all errors should be caught so that implementing apps don't crash. During
development, you can use the debug flavors to allow the sample app to crash in cause of an
Exception (see `handleException()`). When testing, use the release variants to make sure that
exceptions don't crash the implementing apps.
Catch Throwable; not Exception. Since Throwabl is the superclass of Exception, this will make the
lib more resilient to crashes.
### Setup
1. Get the API key from [The Api project](https://github.com/Q42/accessibility-pipeline/tree/main/api). Use this key in the next step.
2. Create a file called `secrets.properties` in the root of the project (not in the app folder). Contents:
1. Get the API key
from [The Api project](https://github.com/Q42/accessibility-pipeline/tree/main/api). Use this key
in the next step.
2. Create a file called `secrets.properties` in the root of the project (not in the app folder).
Contents:
```
apikey="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
```
Note that this file will be ignored by git.
Note that this file will be ignored by git.
3. Change the SampleApplication to construct a Q42Stats object for a real firestore collection.
### Publishing
This library is distributed using [JitPack](https://jitpack.io/#q42/q42stats.android). This makes publishing a new version very easy:
This library is distributed using [JitPack](https://jitpack.io/#q42/q42stats.android). This makes
publishing a new version very easy:
1. In `Q42Stats.kt`, increment `DATA_MODEL_VERSION` by 1 if any changes to collected data is made.
1. Push the code for the new version to the `main` branch
1. Unit tests will be run automatically. Check [JitCI](https://jitci.com/gh/Q42/Q42Stats.Android) for status
1. Unit tests will be run automatically. Check [JitCI](https://jitci.com/gh/Q42/Q42Stats.Android)
for status
1. Create a tag in the semver format: `x.x.x` without the preceding `v.`
1. On GitHub, create a release from that tag. Give it the same name; `x.x.x`
1. If everything went well the release will be visible on [JitPack](https://jitpack.io/#q42/q42stats.android) and the version number in the badge at the top of this page will update.
1. In the Sample app build.gradle, Change the line `jitpackImplementation 'com.github.q42:q42stats.android:x.x.x` to the latest version.
1. If everything went well the release will be visible
on [JitPack](https://jitpack.io/#q42/q42stats.android) and the version number in the badge at the
top of this page will update.
1. In the Sample app build.gradle, Change the
line `jitpackImplementation 'com.github.q42:q42stats.android:x.x.x` to the latest version.
### Troubleshooting
- JitCi build failing while it can be successfully built locally
Perhaps something is broken in JitCi. JitPack can also be used for building and might be more
stable. To disable JitCi, select "Stop Building"
in [JitCi's](https://jitci.com/gh/Q42/Q42Stats.Android) settings page You will lose thebuild
status indicators in GitHub, however
15 changes: 7 additions & 8 deletions q42stats/src/main/java/com/q42/q42stats/library/HttpService.kt
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package com.q42.q42stats.library

import androidx.annotation.WorkerThread
import org.json.JSONObject
import java.io.BufferedReader
import java.io.BufferedWriter
import java.io.InputStreamReader
Expand All @@ -14,18 +13,18 @@ import javax.net.ssl.HttpsURLConnection
internal object HttpService {

/** Sends stats and returns body as String if successful */
fun sendStatsSync(config: Q42StatsConfig, data: JSONObject, lastBatchId: String?): JSONObject? =
fun sendStatsSync(config: Q42StatsConfig, data: String, lastBatchId: String?): String? =
httpPost(
"https://q42stats.ew.r.appspot.com/add/${config.firestoreCollectionId}",
data,
config.apiKey,
lastBatchId
)?.let { JSONObject(it) }
)
}

private fun httpPost(
url: String,
jsonObject: JSONObject,
data: String,
apiKey: String,
lastBatchId: String?
): String? {
Expand All @@ -43,7 +42,7 @@ private fun httpPost(
lastBatchId?.let {
conn.setRequestProperty("batchId", it)
}
return sendPostRequestContent(conn, jsonObject)
return sendPostRequestContent(conn, data)
} catch (e: Throwable) {
Q42StatsLogger.e(TAG, "Could not send stats to server", e)
} finally {
Expand All @@ -53,12 +52,12 @@ private fun httpPost(
return null
}

private fun sendPostRequestContent(conn: HttpURLConnection, jsonObject: JSONObject): String? {
private fun sendPostRequestContent(conn: HttpURLConnection, data: String): String? {
try {
conn.outputStream.use { os ->
BufferedWriter(OutputStreamWriter(os, "UTF-8")).use { writer ->
writer.write(jsonObject.toString())
Q42StatsLogger.d(TAG, "Sending JSON: $jsonObject")
writer.write(data.toString())
Q42StatsLogger.d(TAG, "Sending JSON: $data")
writer.flush()
}
}
Expand Down
36 changes: 23 additions & 13 deletions q42stats/src/main/java/com/q42/q42stats/library/Q42Stats.kt
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ import androidx.annotation.WorkerThread
import com.q42.q42stats.library.collector.AccessibilityCollector
import com.q42.q42stats.library.collector.PreferencesCollector
import com.q42.q42stats.library.collector.SystemCollector
import com.q42.q42stats.library.util.deserializeMeasurement
import com.q42.q42stats.library.util.filterValueNotNull
import com.q42.q42stats.library.util.serializeMeasurement
import kotlinx.coroutines.*
import org.json.JSONObject
import java.io.Serializable

internal const val TAG = "Q42Stats"
Expand Down Expand Up @@ -56,20 +59,27 @@ class Q42Stats(private val config: Q42StatsConfig) {

val currentMeasurement = collect(context)

val payload: Map<String, Any> = mapOf<String, Any?>(
"Stats Version" to "Android ${BuildConfig.LIB_BUILD_DATE}",
"currentMeasurement" to currentMeasurement,
"previousMeasurement" to prefs.previousMeasurement,
val previousMeasurement: Map<String, Any?>? =
prefs.previousMeasurement?.let { deserializeMeasurement(it) }
val payload: Map<String, Any> = mapOf<String, Any?>(
"Stats Version" to "Android ${BuildConfig.LIB_BUILD_DATE}",
"currentMeasurement" to currentMeasurement,
"previousMeasurement" to previousMeasurement,
).filterValueNotNull()
val responseBody = HttpService.sendStatsSync(
config,
payload.toQ42StatsApiFormat(),
prefs.lastBatchId
)
responseBody?.let {
val batchId = it.getString("batchId") // throws if not found
prefs.lastBatchId = batchId
prefs.previousMeasurement = currentMeasurement
val serializedPayload = serializeMeasurement(payload.toQ42StatsApiFormat())
val responseBody = HttpService.sendStatsSync(
config,
serializedPayload,
prefs.lastBatchId
)
responseBody?.let { body ->
val batchId = JSONObject(body).getString("batchId") // throws if not found
prefs.lastBatchId = batchId
prefs.previousMeasurement = currentMeasurement
.toQ42StatsApiFormat()
.let { q42StatsCurrentMeasurement ->
serializeMeasurement(q42StatsCurrentMeasurement)
}
}
} catch (e: Throwable) {
handleException(e)
Expand Down
22 changes: 3 additions & 19 deletions q42stats/src/main/java/com/q42/q42stats/library/Q42StatsPrefs.kt
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@ package com.q42.q42stats.library

import android.content.Context
import android.content.SharedPreferences
import org.json.JSONArray
import org.json.JSONObject

private const val SHARED_PREFS_NAME = "Q42StatsPrefs"
private const val LAST_SUBMIT_TIMESTAMP_KEY = "lastSubmitTimestamp"
Expand All @@ -14,14 +12,12 @@ internal class Q42StatsPrefs(context: Context) {
private val prefs: SharedPreferences =
context.getSharedPreferences(SHARED_PREFS_NAME, Context.MODE_PRIVATE)

var previousMeasurement: Map<String, Any?>?
var previousMeasurement: String?
get() =
prefs.getString(PREVIOUS_MEASUREMENT_KEY, null)?.let {
JSONObject(it).toMap()
}
prefs.getString(PREVIOUS_MEASUREMENT_KEY, null)
set(value) = with(prefs.edit()) {
value?.let {
putString(PREVIOUS_MEASUREMENT_KEY, JSONObject(value).toString())
putString(PREVIOUS_MEASUREMENT_KEY, value)
} ?: run {
remove(PREVIOUS_MEASUREMENT_KEY)
}
Expand All @@ -48,16 +44,4 @@ internal class Q42StatsPrefs(context: Context) {
apply()
}

private fun JSONObject.toMap(): Map<String, Any?> = keys().asSequence().associateWith {
when (val value = this[it]) {
is JSONArray -> {
val map = (0 until value.length()).associate { Pair(it.toString(), value[it]) }
JSONObject(map).toMap().values.toList()
}
is JSONObject -> value.toMap()
JSONObject.NULL -> null
else -> value
}
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ internal object AccessibilityCollector {
name,
0
) == 1
} catch (e: Exception) {
} catch (e: Throwable) {
Q42StatsLogger.e(TAG, "Could not read system int $name. Returning null", e)
null
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
package com.q42.q42stats.library

import org.json.JSONObject
import java.io.Serializable

internal fun Map<String, Any>.toQ42StatsApiFormat(): JSONObject {
val fireStoreMap = this.mapValues { entry ->
/** Returns a version of the input where all values are stringified
* We do this to prevent serialize -> deserialize issues where the float 1.0 is transformed to 1,
* for example. (this change is undesirable because the changed data type causes the object to have
* a different hash, not being equal in comparisons etc.
*/
internal fun Map<String, Any>.toQ42StatsApiFormat(): Map<String, Any> {
val q42StatsMap = this.mapValues { entry ->
mapFieldValue(entry)
}
return JSONObject(fireStoreMap)
return q42StatsMap
}

@Suppress("UNCHECKED_CAST")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package com.q42.q42stats.library.util

import org.json.JSONArray
import org.json.JSONObject

internal fun serializeMeasurement(value: Map<String, Any?>) =
JSONObject(value).toString()

internal fun deserializeMeasurement(it: String): Map<String, Any?> =
JSONObject(it).toMap()

private fun JSONObject.toMap(): Map<String, Any?> = keys().asSequence().associateWith {
when (val value = this[it]) {
is JSONArray -> {
val map = (0 until value.length()).associate { Pair(it.toString(), value[it]) }
JSONObject(map).toMap().values.toList()
}
is JSONObject -> value.toMap()
JSONObject.NULL -> null
else -> {
value
}
}
}
Original file line number Diff line number Diff line change
@@ -1,22 +1,34 @@
package com.q42.q42stats.library

import org.junit.Assert.assertEquals
import org.junit.Test

class ApiFormatUtilTest {
@Test
fun testToQ42StatsApiFormat() {
val expected =
"""{"screen width":"360","textScale":"1.6","language":"nl","currentMeasurement":{"createdAt":"1337"},"talkbackEnabled":"true"}"""
val actual = mapOf(
"screen width" to 360,
"fontSize" to 1.0, // expected is "1.0". should not be truncated to "1"
"textScale" to 1.6,
"talkbackEnabled" to true,
"language" to "nl",
"currentMeasurement" to mapOf(
"createdAt" to 1337L
)
).toQ42StatsApiFormat().toString()
assertEquals(expected, actual)
).toQ42StatsApiFormat()

testQ42StatsMap(actual)
}

private fun testQ42StatsMap(actual: Map<*, *>) {
actual.forEach {
when (it.value) {
is Map<*, *> -> {
testQ42StatsMap(it.value as Map<*, *>)
}
else -> {
assert(it.value is String) { "Expected $it to have a string type but was ${it.value}" }
}
}
}
}
}
Loading

0 comments on commit d3123e9

Please sign in to comment.