Add support for replaying network requests from a captured HAR file (r…

…ancher#10149)

* Add support for replaying from har file

* Fix merge

* Improvements

* Add export and folder support

* Minor tidy ups

* Fix lint

* Add doc

* Update doc with correct fallback HTTP status code

* Fix typo

docs/developer/har-files.md

@@ -0,0 +1,113 @@
+# Using HAR Files
+
+A HAR file can be created from a web browser (such as Chrome) and can contain performance data but also the network requests captured over a period of time.
+
+A HAR file can therefore be a useful tool for debugging and analyzing a UI problem.
+
+There are a couple of tools that assist developers with using HAR files.
+
+The tools below assume that you have a HAR file that has network request/response data included in it.
+
+## yarn dev
+
+You can run the locally built UI in dev mode and replay the network requests from a captured HAR file. To so, set the `HAR_FILE` environment variable to the path of the HAR file to use, e.g.
+
+```
+HAR_FILE=test-data.har yarn dev
+```
+
+The UI will build and run as normal, but API network requests be intercepted and the responses will be sent from the matching requests in the HAR file.
+
+When you run `yarn dev` you'll see a list of the network requests that are stored in the HAR file and you'll also see the page that was loaded in the browser during the HAR file capture, eg.
+
+```
+Loading HAR file: <file-name>
+Network requests:
+    GET /api/v1/namespaces/cattle-ui-plugin-system/services/http:ui-plugin-operator:80/proxy/index.json
+    GET /rancherversion
+    GET /v1/management.cattle.io.setting
+    GET /v3/users?me=true
+    GET /v3/principals
+    GET /v1/userpreferences
+Page:
+    https://127.0.0.1:8005/c/local/explorer
+``````
+
+You should then take the URL listed under `Page` and load that into your web browser.
+
+## Running a built version of the UI
+
+Similar to above, but instead of the UI being served up from the local development build, the UI will be served from the production build for a given version of the UI.
+
+This is much quicker than `yarn dev` since the UI does not need to be built. This also allows easy checking of a bug or issue in different versions of the UI.
+
+Use the `har` script to do this, eg.
+
+```
+./scripts/har <har_file> <version>
+```
+
+Where:
+
+- `<har_file>` is the path to the HAR file to use
+- `<version>` is the version number of the UI to use (eg. `v2.8.0`, `v2.7.6`, `latest`)
+
+You'll get a similar print out to above, but the UI is loaded from the production build.
+
+## Exporting requests/responses
+
+You may want to inspect the network requests/responses contained in a HAR file. To support this, you can export each request into a separate json file via the `har-export` script, eg.
+
+```
+./scripts/har-export <har-file> <folder>
+```
+
+This will create sub-folders beneath `folder` for the path of the API request and create a file for the request of the form:
+
+```<name>.<method>.json```
+
+Where `<method>` is the HTTP version such as `get`` or `post`.
+
+Note that if the file contains multiple requests of the same path and method, only the last one in the HAR file will be written to disk (the previous ones will be overwritten).
+
+## Missing Requests
+
+In some cases the HAR file may not include a request or the data for the request may be missing. For empty request data, you will a message similar to the following when you run `yarn dev` or the `har` script:
+
+```
+    GET /v1/schemas
+        Warning: Omitting this response as there is no content - UI may not work as expected
+```
+
+To work around this, you can provide fallback data to be used in the case that a request is missing from the HAR file.
+
+Fallback data must be in a folder and use the same structure as that generated by `har-export`. The general idea is you would capture and then export a HAR file containing the requests you need to a folder.
+
+When running either `yarn dev` or `har` you can specify the folder location of fallback request data. By default the request from the HAR file is used. If this is not present, data from the fallback folder will be used.
+
+eg.
+
+```
+HAR_FILE=test-data.har HAR_DIR=fallback-folder yarn dev
+```
+
+or
+
+```
+./scripts/har <har_file> <version> <fallback-folder>
+```
+
+## Logging
+
+When running `yarn dev` or the `har` script as detalied above, the network requests will be logged, eg:
+
+```
+GET  404 /api/v1/namespaces/cattle-ui-plugin-system/services/http:ui-plugin-operator:80/proxy/index.json
+``````
+
+Each log entry starts the with HTTP verb being used and is followed by a single character, the meaning of which is as follows:
+
+- ' ' Empty space indicates request was sent from the HAR file
+- '*' Indicates that the request was sent from the HAR file but that all requests for that path had already been sent and the last request is being used again
+- '?' Indicates that there was no matching request - in this case a 404 will be returned
+- 'f' Indicates that the fallback data from file was used for the request

scripts/har

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+
+const fs = require('fs');
+const https = require('https');
+const path = require('path');
+const { createProxyMiddleware } = require('http-proxy-middleware');
+const express = require('express');
+const request = require('request');
+
+const base = path.resolve(__dirname, '..');
+const shell = path.resolve(base, 'shell');
+
+const har = require(`${ shell }/server/har-file`);
+
+console.log(path.resolve(shell, 'server.key'));
+
+const options = {
+  key:  fs.readFileSync(path.resolve(shell, 'server', 'server.key')),
+  cert: fs.readFileSync(path.resolve(shell, 'server', 'server.crt'))
+};
+
+let PORT = process.env.PORT || '8005';
+
+PORT = parseInt(PORT, 10);
+
+if (isNaN(PORT)) {
+  console.log('Invalid port'); // eslint-disable-line no-console
+  process.exit(1);
+}
+
+// Need at least two arguments
+
+if (process.argv.length < 4) {
+  console.log('Requires 2 arguments. Can also take an optional folder:');
+  console.log('  <har_file>  path to HAR file to load');
+  console.log('  <version>   dashboard version number to use for static assets (e.g v2.8.0, latest)');
+  console.log('  [<folder>]  folder containing json files to use as a fallback when HAR file does not contain request');
+  console.log('');
+
+  process.exit(1);
+}
+
+const harFile = process.argv[2];
+const version = process.argv[3].trim();
+const harData = har.loadFile(harFile, PORT, '/dashboard');
+const dashboardUrl = `https://releases.rancher.com/dashboard/${ version }/index.html`;
+
+let harFolder = process.argv.length > 4 ? process.argv[4] : '';
+
+console.log(dashboardUrl);
+
+const dev = (process.env.NODE_ENV !== 'production');
+const devPorts = dev || process.env.DEV_PORTS === 'true';
+
+const app = express();
+
+// Catch reload on a dynamic page
+// Check that the requestor will accept html and send them the index file
+app.use('*', (req, res, next) => {
+  const accept = req.headers.accept || '';
+  const acceptArray = accept.split(',');
+  const html = acceptArray.find((h) => h.trim() === 'text/html');
+
+  if (html) {
+    request(dashboardUrl, function (error, response, body) {
+      res.type('text/html');
+      res.status(response.statusCode);
+      res.send(Buffer.from(body));
+      res.end();
+    });
+
+    return;
+  }
+
+  next();
+});
+
+app.use('/dashboard', createProxyMiddleware(proxyWsOpts(dashboardUrl)));
+
+// Add in handler for har file requests
+app.use(har.harProxy(harData, harFolder));
+
+const server = https.createServer(options, app);
+const appServer = server.listen(PORT);
+
+console.log(`Running Dashboard web server on port ${ PORT }`); // eslint-disable-line no-console
+
+appServer.on('upgrade', (req, socket, head) => {
+  const responseHeaders = ['HTTP/1.1 101 Web Socket Protocol Handshake', 'Upgrade: WebSocket', 'Connection: Upgrade'];
+
+  socket.write(`${ responseHeaders.join('\r\n') }\r\n\r\n`);
+});
+
+// ===============================================================================================
+// Functions for the request proxying used in dev
+// ===============================================================================================
+
+function proxyOpts(target) {
+  return {
+    target,
+    secure:       !devPorts,
+    ws:           false,
+    changeOrigin: true,
+    onProxyReq,
+    onProxyReqWs,
+    onError,
+    onProxyRes,
+  };
+}
+
+function onProxyRes(proxyRes, req, res) {
+  if (devPorts) {
+    proxyRes.headers['X-Frame-Options'] = 'ALLOWALL';
+  }
+}
+
+function proxyWsOpts(target) {
+  return {
+    ...proxyOpts(target),
+    ws:           false,
+    changeOrigin: true,
+    secure:       false,
+  };
+}
+
+function onProxyReq(proxyReq, req) {
+  if (!(proxyReq._currentRequest && proxyReq._currentRequest._headerSent)) {
+    proxyReq.setHeader('x-api-host', req.headers['host']);
+    proxyReq.setHeader('x-forwarded-proto', 'https');
+    // console.log(proxyReq.getHeaders());
+  }
+}
+
+function onProxyReqWs(proxyReq, req, socket, options, head) {
+  req.headers.origin = options.target.href;
+  proxyReq.setHeader('origin', `${ options.target.href }/`);
+  proxyReq.setHeader('x-api-host', req.headers['host']);
+  proxyReq.setHeader('x-forwarded-proto', 'https');
+
+  socket.on('error', (err) => {
+    console.error('Proxy WS Error:', err); // eslint-disable-line no-console
+  });
+}
+
+function onError(err, req, res) {
+  res.statusCode = 598;
+  console.error('Proxy Error:', err); // eslint-disable-line no-console
+  res.write(JSON.stringify(err));
+}

scripts/har-export

@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+
+const path = require('path');
+
+const base = path.resolve(__dirname, '..');
+const shell = path.resolve(base, 'shell');
+
+const har = require(`${ shell }/server/har-file`);
+
+// Need two arguments
+if (process.argv.length < 4) {
+  console.log('Export HAR network requests to file')
+  console.log('');
+  console.log('Need 2 arguments:');
+  console.log('  <har_file> path to HAR file to load');
+  console.log('  <folder>  folder to write the HAR network request files to');
+  console.log('');
+
+  process.exit(1);
+}
+
+const harFile = process.argv[2].trim();
+const outFolder = process.argv[3].trim();
+
+const harData = har.loadFile(harFile, 8005);
+
+har.exportToFiles(harData, outFolder);