Change snippet extraction to use regex, add support for inline snippe…

…t tokens, update readme

README.md

@@ -2,7 +2,7 @@
 
 `doc-snippets` is a simple tool that allows you to extract and inject snippets from code into markdown files.
 
-## Installation
+# Installation
 
 Using NPM:
 
@@ -16,9 +16,9 @@ Using Yarn:
 yarn add -D doc-snippets
 ```
 
-## Usage
+# Usage
 
-### Marking and injecting
+## Marking and injecting
 
 `doc-snippets` extracts and injects snippets by using tokens:
 
@@ -28,7 +28,7 @@ yarn add -D doc-snippets
 - To inject a snippet:
   - `$snippet: snippet-name` - This gets replaced by the snippet with the given name.
 
-### Configuration
+## Configuration
 
 `doc-snippets` is, by defaut, configured using a JSON file that contains a `doc-snippets` object. By default, this file is `package.json`.
 
@@ -46,14 +46,24 @@ The configuration object has the following structure (and default values):
     "ignore": [],
     "dir": "./src/docs"
   },
-  "startToken": "$start: ",
-  "endToken": "$end",
+  "startTokens": [
+    {
+      "pattern": "$start: ",
+      "inline": false
+    }
+  ],
+  "endTokens": [
+    {
+      "pattern": "$end",
+      "inline": false
+    }
+  ],
   "injectToken": "$snippet: ",
   "outputDir": "./docs"
 }
 ```
 
-#### `extract` and `inject`
+### `extract` and `inject`
 
 The `extract` and `inject` objects both have the same structure:
 
@@ -66,29 +76,107 @@ The `extract` object specifies which files will be parsed for snippets to extrac
 The `inject` object specifies which files will be copied into `outputDir` and have snippets injected into them.
 
 **Example `extract` object:**
+
 ```JSON
 "extract": {
   "include": ["sample.sql", "./**/*.{js,ts}"],
   "ignore": "./**/node_modules/**",
   "dir": "./src"
 }
 ```
+
 In this example, `extract` will perform its search within the `./src` directory.
 
 It will include:
+
 - `./src/sample.sql`
 - All files within `./src` and its subdirectories ending in `.js` and `.ts`
 
 It will ignore:
+
 - All directories and subdirectories of any `node_modules` directory found within `./src` and its subdirectories.
 
 The same principles apply for the `inject` object.
 
-#### `outputDir`
+### Extraction tokens (`startTokens` and `endTokens`)
+
+Extraction tokens mark the beginning and end of our snippets.
+
+The tokens have two properties:
+- `pattern` - the **exact string** that denotes the token
+- `inline` - defines whether the token is a **regular** or **inline** extraction token
+
+#### Regular and inline extraction tokens
+
+By default, extraction tokens are designed to be written within their own lines. Anything else written within the token's line will be ignored when extracting snippets.
+
+Let's take a look at an example:
+
+```typescript
+// $start: hello-snippet we can add comments here
+const greeting = "Hello World!";
+console.log(greeting);
+// $end
+```
+
+Using the default configuration, this code will yield a snippet called `hello-snippet` with the following contents:
+```
+const greeting = "Hello World!";
+console.log(greeting);
+```
+
+To allow for more flexible snippet extraction, we can use **inline** extraction tokens.
+
+For example, let's configure our extraction tokens as follows:
+
+```json
+{
+  //...
+  "startTokens": [
+    {
+      "pattern": "/* $start: {SNIPPET_NAME} */",
+      "inline": true
+    }
+  ],
+  "endTokens": [
+    {
+      "pattern": "/* $end */",
+      "inline": true
+    }
+  ]
+}
+```
+
+Note that there is a special token within the start token's `pattern` property: **`{SNIPPET_NAME}`**. This denotes where in the inline token the snippet name will appear.
+
+In addition, let's assume that we have a file with the following code:
+
+```typescript
+const greeting = /* $start: hello-inline-snippet */"Hello World!";
+console.log(greeting);/* $end */
+```
+
+Snippet extraction would now yield a snippet named `hello-inline-snippet` with the following contents:
+```
+"Hello World!";
+console.log(greeting);
+```
+
+Inline extraction tokens can be useful when you need to extract only part of your line.
+
+Regular and inline extraction tokens can be used at the same time, and snippet extraction will always search for **any** end token after it encounters **any** start token.
+
+For reference:
+- When a  **regular start token** is encoutered, snippet extraction starts at the beginning of the next line.
+- When an **inline start token** is encountered, snippet extraction starts immediately after the token ends.
+- When a **regular end token** is encountered, snippet extraction ends at the end of the previous line.
+- When an **inline end token** is encountered, snippet extraction ends at the beginning of the token.
+
+### `outputDir`
 
 The `outputDir` is the output directory for the documentation injected with snippets.
 
-### Running `doc-snippets`
+## Running `doc-snippets`
 
 `doc-snippets` comes with a CLI tool which is designed to handle most scenarios.
 
@@ -100,7 +188,7 @@ doc-snippets combine
 
 The `combine` command reads a `"doc-snippets"` section from a configuration file (by default this is `package.json`) and performs snippet extraction and injection, and outputs documentation with injected snippets into an `outputDir`.
 
-#### 'combine' command options
+### 'combine' command options
 
 - `-c --config <path>` - Path to configuration file (default: './package.json')
 - `-o --output-dir <path>` - Combined documentation output directory
@@ -110,12 +198,14 @@ The `combine` command reads a `"doc-snippets"` section from a configuration file
 - `--inject-dir <path>` - The base directory within which to search for injectable files
 - `--inject-include <paths...>` - Include specified paths or glob patterns in snippet injection
 - `--inject-ignore <paths...>` - Ignore specified paths or glob patterns in snippet injection
-- `--start-token <token>` - Token marking the start of the snippet
-- `--end-token <token>` - Token marking the end of the snippet
+- `--start-tokens <tokens...>` - Tokens marking the start of the snippet
+- `--end-tokens <tokens...>` - Tokens marking the end of the snippet
+- `--inline-start-tokens <tokens...>` - Inline Tokens marking the start of the snippet
+- `--inline-end-tokens <tokens...>` - Inline Tokens marking the end of the snippet
 - `--inject-token <token>` - Token marking the point of snippet injection
 
 
-### In your own code
+## In your own code
 
 If you want to use `doc-snippets` programatically, it offers two exported functions:
 
@@ -125,11 +215,31 @@ import { extractSnippets, injectSnippetsIntoFile } from "doc-snippets";
 const searchOptions = {
   include: ["sample.sql", "./**/*.{js,ts}"],
   ignore: "./**/node_modules/**",
-  dir: "./src"
-}
+  dir: "./src",
+};
 
 //Returns snippets as `Record<string, string>`.
-const snippets = await extractSnippets(searchOptions, "$start: ", "$end");
+const startTokens = [
+  {
+    pattern: "$start: "
+  },
+  {
+    pattern: "/* $start: {SNIPPET_NAME} */",
+    inline: true
+  }
+];
+
+const endTokens = [
+  {
+    pattern: "$end"
+  },
+  {
+    pattern: "/* $end */",
+    inline: true
+  }
+];
+
+const snippets = await extractSnippets(searchOptions, startTokens, endTokens);
 
 //Injects `snippets` into `./dest/readme.md` replacing all instances of `$snippet: snippet-name` with the appropriate snippet
 await injectSnippetsIntoFile(snippets, "./dest/readme.md", "$snippet: ");