Outbound Webhooks

changelog.d/945.feature

@@ -0,0 +1,3 @@
+Add support for new connection type "Outgoing Webhooks". This feature allows you to send outgoing HTTP requests to other services
+when a message appears in a Matrix room. See [the documentation](https://matrix-org.github.io/matrix-hookshot/latest/setup/webhooks.html)
+for help with this feature.

config.sample.yml

@@ -100,6 +100,7 @@ listeners:
 #  #'allowJsTransformationFunctions' will allow users to write short transformation snippets in code, and thus is unsafe in untrusted environments
 
 #  enabled: false
+#  outbound: false
 #  enableHttpGet: false
 #  urlPrefix: https://example.com/webhook/
 #  userIdPrefix: _webhooks_

docs/setup/webhooks.md

@@ -1,7 +1,7 @@
 # Webhooks
 
-Hookshot supports generic webhook support so that services can send messages into Matrix rooms without being aware of the Matrix protocol. This works
-by having services hit a unique URL that then transforms a HTTP payload into a Matrix message.
+Hookshot supports two kinds of webhooks, inbound (previously known as Generic Webhooks) and outbound.
+
 
 ## Configuration
 
@@ -10,13 +10,19 @@ You will need to add the following configuration to the config file.
 ```yaml
 generic:
   enabled: true
+  outbound: true # For outbound webhook support
   urlPrefix: https://example.com/mywebhookspath/
   allowJsTransformationFunctions: false
   waitForComplete: false
   enableHttpGet: false
   # userIdPrefix: webhook_
 ```
 
+## Inbound Webhooks
+
+Hookshot supports generic webhook support so that services can send messages into Matrix rooms without being aware of the Matrix protocol. This works
+by having services hit a unique URL that then transforms a HTTP payload into a Matrix message.
+
 <section class="notice">
 Previous versions of the bridge listened for requests on `/` rather than `/webhook`. While this behaviour will continue to work,
 administators are advised to use `/webhook`.
@@ -50,15 +56,15 @@ namespaces:
       exclusive: true
 ```
 
-## Adding a webhook
+### Adding a webhook
 
 To add a webhook to your room:
   - Invite the bot user to the room.
   - Make sure the bot able to send state events (usually the Moderator power level in clients)
   - Say `!hookshot webhook example` where `example` is a name for your hook.
   - The bot will respond with the webhook URL to be sent to services.
 
-## Webhook Handling
+### Webhook Handling
 
 Hookshot handles `POST` and `PUT` HTTP requests by default.
 
@@ -76,7 +82,7 @@ If the body *also* contains a `username` key, then the message will be prepended
 If the body does NOT contain a `text` field, the full payload will be sent to the room. This can be adapted into a message by creating a **JavaScript transformation function**.
 
 
-### Payload formats
+#### Payload formats
 
 If the request is a `POST`/`PUT`, the body of the request will be decoded and stored inside the event. Currently, Hookshot supports:
 
@@ -88,7 +94,7 @@ If the request is a `POST`/`PUT`, the body of the request will be decoded and st
 Decoding is done in the order given above. E.g. `text/xml` would be parsed as XML. Any formats not described above are not
 decoded.
 
-### GET requests
+#### GET requests
 
 In previous versions of hookshot, it would also handle the `GET` HTTP method. This was disabled due to concerns that it was too easy for the webhook to be
 inadvertently triggered by URL preview features in clients and servers. If you still need this functionality, you can enable it in the config.
@@ -102,7 +108,7 @@ to a string representation of that value. This change is <strong>not applied</st
 variable, so it will contain proper float values.
 </section>
 
-### Wait for complete
+#### Wait for complete
 
 It is possible to choose whether a webhook response should be instant, or after hookshot has handled the message. The reason
 for this is that some services expect a quick response time (like Slack) whereas others will wait for the request to complete. You
@@ -111,7 +117,7 @@ can specify this either globally in your config, or on the widget with `waitForC
 If you make use of the `webhookResponse` feature, you will need to enable `waitForComplete` as otherwise hookshot will
 immeditately respond with it's default response values.
 
-## JavaScript Transformations
+### JavaScript Transformations
 
 <section class="notice">
 Although every effort has been made to securely sandbox scripts, running untrusted code from users is always risky. Ensure safe permissions
@@ -130,7 +136,7 @@ Please seek out documentation from your client on how to achieve this.
 
 The script string should be set within the state event under the `transformationFunction` key.
 
-### Script API
+#### Script API
 
 Transformation scripts have a versioned API. You can check the version of the API that the hookshot instance supports
 at runtime by checking the `HookshotApiVersion` variable. If the variable is undefined, it should be considered `v1`.
@@ -141,7 +147,7 @@ Scripts are executed synchronously and expect the `result` variable to be set.
 If the script contains errors or is otherwise unable to work, the bridge will send an error to the room. You can check the logs of the bridge
 for a more precise error.
 
-### V2 API
+#### V2 API
 
 The `v2` api expects an object to be returned from the `result` variable.
 
@@ -176,7 +182,7 @@ if (data.counter === undefined) {
 ```
 
 
-### V1 API
+#### V1 API
 
 The v1 API expects `result` to be a string. The string will be automatically interpreted as Markdown and transformed into HTML. All webhook messages
 will be prefixed with `Received webhook:`. If `result` is falsey (undefined, false or null) then the message will be `No content`.
@@ -192,3 +198,36 @@ if (data.counter > data.maxValue) {
     result = `*Everything is fine*, the counter is under by ${data.maxValue - data.counter}`
 }
 ```
+
+## Outbound webhooks
+
+You can also configure Hookshot to send outgoing requests to other services when a message appears
+on Matrix. To do so, you need to configure hookshot to enable outgoing messages with:
+
+```yaml
+generic:
+  outbound: true
+```
+
+### Request format
+
+Requests can be sent to any service that accepts HTTP requests. You may configure Hookshot to either use the HTTP `PUT` (default)
+or `POST` methods.
+
+Each request will contain 3 headers which you may use to authenticate and direct traffic:
+
+  - 'X-Matrix-Hookshot-EventId' contains the event's ID.
+  - 'X-Matrix-Hookshot-RoomId' contains the room ID where the message was sent.
+  - 'X-Matrix-Hookshot-Token' is the unique authentication token provided when you created the webhook. Use this
+    to verify that the message came from Hookshot.
+
+The payloads are formatted as `multipart/form-data`.
+
+The first file contains the event JSON data, proviced as the `event` file. This is a raw representation of the Matrix event data. If the
+event was encrypted, this will be the **decrypted** body.
+
+If any media is linked to in the event, then a second file will be present named `media` which will contain the media referenced in
+the event.
+
+All events that occur in the room will be sent to the outbound URL, so be careful to ensure your remote service can filter the
+traffic appropriately (e.g. check the `type` in the event JSON)

package.json

@@ -48,8 +48,8 @@
     "@octokit/rest": "^20.0.2",
     "@octokit/webhooks": "^12.0.10",
     "@sentry/node": "^7.52.1",
-    "@vector-im/compound-design-tokens": "^0.1.0",
-    "@vector-im/compound-web": "^0.9.4",
+    "@vector-im/compound-design-tokens": "^1.3.0",
+    "@vector-im/compound-web": "^4.8.0",
     "ajv": "^8.11.0",
     "axios": "^1.6.3",
     "cors": "^2.8.5",
@@ -86,6 +86,7 @@
     "@rollup/plugin-alias": "^5.1.0",
     "@tsconfig/node18": "^18.2.2",
     "@types/ajv": "^1.0.0",
+    "@types/busboy": "^1.5.4",
     "@types/chai": "^4.2.22",
     "@types/cors": "^2.8.12",
     "@types/express": "^4.17.14",
@@ -100,6 +101,7 @@
     "@typescript-eslint/eslint-plugin": "^6.17.0",
     "@typescript-eslint/parser": "^6.17.0",
     "@uiw/react-codemirror": "^4.12.3",
+    "busboy": "^1.6.0",
     "chai": "^4.3.4",
     "eslint": "^8.49.0",
     "eslint-config-preact": "^1.3.0",
@@ -116,5 +118,6 @@
     "ts-node": "^10.9.1",
     "typescript": "^5.3.3",
     "vite": "^5.0.13"
-  }
+  },
+  "packageManager": "[email protected]+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"
 }

spec/webhooks.spec.ts

@@ -0,0 +1,201 @@
+import { E2ESetupTestTimeout, E2ETestEnv, E2ETestMatrixClient } from "./util/e2e-test";
+import { describe, it, beforeEach, afterEach } from "@jest/globals";
+import { OutboundHookConnection } from "../src/Connections";
+import { TextualMessageEventContent } from "matrix-bot-sdk";
+import { IncomingHttpHeaders, createServer } from "http";
+import busboy, { FileInfo } from "busboy";
+import { TEST_FILE } from "./util/fixtures";
+
+async function createOutboundConnection(user: E2ETestMatrixClient, botMxid: string, roomId: string) {
+    const join = user.waitForRoomJoin({ sender: botMxid, roomId });
+    const connectionEvent = user.waitForRoomEvent({
+        eventType: OutboundHookConnection.CanonicalEventType,
+        stateKey: 'test',
+        sender: botMxid
+    });
+    await user.inviteUser(botMxid, roomId);
+    await user.setUserPowerLevel(botMxid, roomId, 50);
+    await join;
+
+    // Note: Here we create the DM proactively so this works across multiple
+    // tests.
+    // Get the DM room so we can get the token.
+    const dmRoomId = await user.dms.getOrCreateDm(botMxid);
+
+    await user.sendText(roomId, '!hookshot outbound-hook test http://localhost:8111/test-path');
+    // Test the contents of this.
+    await connectionEvent;
+
+    const msgPromise = user.waitForRoomEvent({ sender: botMxid, eventType: "m.room.message", roomId: dmRoomId });
+    const { data: msgData } =  await msgPromise;
+
+    const [_match, token ] = /<code>(.+)<\/code>/.exec((msgData.content as unknown as TextualMessageEventContent).formatted_body ?? "") ?? [];
+    return token;
+}
+
+/**
+ * 
+ * @returns 
+ */
+function awaitOutboundWebhook() {
+    return new Promise<{headers: IncomingHttpHeaders, files: {name: string, file: Buffer, info: FileInfo}[]}>((resolve, reject) => {
+        const server = createServer((req, res) => {
+            const bb = busboy({headers: req.headers});
+            const files: {name: string, file: Buffer, info: FileInfo}[] = [];
+            bb.on('file', (name, stream, info) => {
+                const buffers: Buffer[] = [];
+                stream.on('data', d => {
+                    buffers.push(d)
+                });
+                stream.once('close', () => {
+                    files.push({name, info, file: Buffer.concat(buffers)})
+                });
+            });
+
+            bb.once('close', () => {
+                res.writeHead(200, { 'Content-Type': 'text/plain' });
+                res.end('OK');
+                resolve({
+                    headers: req.headers,
+                    files,
+                });
+                clearTimeout(timer);
+                server.close();
+            });
+
+            req.pipe(bb);
+        });
+        server.listen(8111);
+        let timer: NodeJS.Timeout;
+        timer = setTimeout(() => {
+            reject(new Error("Request did not arrive"));
+            server.close();
+        }, 10000);
+
+    });
+}
+
+describe('OutboundHooks', () => {
+    let testEnv: E2ETestEnv;
+
+    beforeAll(async () => {
+        const webhooksPort = 9500 + E2ETestEnv.workerId;
+        testEnv = await E2ETestEnv.createTestEnv({
+            matrixLocalparts: ['user'],
+            config: {
+                generic: {
+                    enabled: true,
+                    outbound: true,
+                    urlPrefix: `http://localhost:${webhooksPort}`
+                },
+                listeners: [{
+                    port: webhooksPort,
+                    bindAddress: '0.0.0.0',
+                    // Bind to the SAME listener to ensure we don't have conflicts.
+                    resources: ['webhooks'],
+                }],
+            }
+        });
+        await testEnv.setUp();
+    }, E2ESetupTestTimeout);
+
+    afterAll(() => {
+        return testEnv?.tearDown();
+    });
+
+    it('should be able to create a new webhook and push an event.', async () => {
+        const user = testEnv.getUser('user');
+        const roomId = await user.createRoom({ name: 'My Test Webhooks room'});
+        const token = await createOutboundConnection(user, testEnv.botMxid, roomId);
+        const gotWebhookRequest = awaitOutboundWebhook();
+
+        const eventId = await user.sendText(roomId, 'hello!');
+        const { headers, files } = await gotWebhookRequest;
+        expect(headers['x-matrix-hookshot-roomid']).toEqual(roomId);
+        expect(headers['x-matrix-hookshot-eventid']).toEqual(eventId);
+        expect(headers['x-matrix-hookshot-token']).toEqual(token);
+
+        // And check the JSON payload
+        const [event, media] = files;
+        expect(event.name).toEqual('event');
+        expect(event.info.mimeType).toEqual('application/json');
+        expect(event.info.filename).toEqual('event_data.json');
+        const eventJson = JSON.parse(event.file.toString('utf-8'));
+
+        // Check that the content looks sane.
+        expect(eventJson.room_id).toEqual(roomId);
+        expect(eventJson.event_id).toEqual(eventId);
+        expect(eventJson.sender).toEqual(await user.getUserId());
+        expect(eventJson.content.body).toEqual('hello!');
+
+        // No media should be present.
+        expect(media).toBeUndefined();
+    });
+
+    it('should be able to create a new webhook and push a media attachment.', async () => {
+        const user = testEnv.getUser('user');
+        const roomId = await user.createRoom({ name: 'My Test Webhooks room'});
+        await createOutboundConnection(user, testEnv.botMxid, roomId);
+        const gotWebhookRequest = awaitOutboundWebhook();
+
+        const mxcUrl = await user.uploadContent(TEST_FILE, 'image/svg+xml', "matrix.svg");
+        await user.sendMessage(roomId, {
+            url: mxcUrl,
+            msgtype: "m.file",
+            body: "matrix.svg",
+        })
+        const { files } = await gotWebhookRequest;
+        const [event, media] = files;
+        expect(event.info.mimeType).toEqual('application/json');
+        expect(event.info.filename).toEqual('event_data.json');
+        const eventJson = JSON.parse(event.file.toString('utf-8'));
+        expect(eventJson.content.body).toEqual('matrix.svg');
+
+
+        expect(media.info.mimeType).toEqual('image/svg+xml');
+        expect(media.info.filename).toEqual('matrix.svg');
+        expect(media.file).toEqual(TEST_FILE);
+    });
+
+    // TODO: This requires us to support Redis in test conditions, as encryption is not possible
+    // in hookshot without it at the moment.
+
+    // it.only('should be able to create a new webhook and push an encrypted media attachment.', async () => {
+    //     const user = testEnv.getUser('user');
+    //     const roomId = await user.createRoom({ name: 'My Test Webhooks room', initial_state: [{
+    //         content: {
+    //             "algorithm": "m.megolm.v1.aes-sha2"
+    //         },
+    //         state_key: "",
+    //         type: "m.room.encryption"
+    //     }]});
+    //     await createOutboundConnection(user, testEnv.botMxid, roomId);
+    //     const gotWebhookRequest = awaitOutboundWebhook();
+
+    //     const encrypted = await user.crypto.encryptMedia(Buffer.from(TEST_FILE));
+    //     const mxc = await user.uploadContent(TEST_FILE);
+    //     await  user.sendMessage(roomId, {
+    //         msgtype: "m.image",
+    //         body: "matrix.svg",
+    //         info: {
+    //             mimetype: "image/svg+xml",
+    //         },
+    //         file: {
+    //             url: mxc,
+    //             ...encrypted.file,
+    //         },
+    //     });
+
+    //     const { headers, files } = await gotWebhookRequest;
+    //     const [event, media] = files;
+    //     expect(event.info.mimeType).toEqual('application/json');
+    //     expect(event.info.filename).toEqual('event_data.json');
+    //     const eventJson = JSON.parse(event.file.toString('utf-8'));
+    //     expect(eventJson.content.body).toEqual('matrix.svg');
+
+
+    //     expect(media.info.mimeType).toEqual('image/svg+xml');
+    //     expect(media.info.filename).toEqual('matrix.svg');
+    //     expect(media.file).toEqual(TEST_FILE);
+    // });
+});