FMark by ZestyLogic 🍋♟

FSharp markdown implementation with additional functionality.

Namely:

Macros
Includes
Spreadsheet functionality
Table of Contents generation
Citations and footnotes

See example.fmark for usage examples.

How to use

FMark Plugin Visual Studio Code

Functions:

Open Preview (ctrl+shift+M)

Open a live preview of the HTML that will be generated by FMark.

Create HTML File

Create an HTML file in the same directory as the .fmark file, with .html instead of .fmark.

Command Line Interface

USAGE: FMark [--help] [--output <path>] [--loglevel <debug|info|warning|error|fatal>] [--format <html|markdown>] [--test [<sequential>]] [<path>]

INPUT:
    <path>                specify input file path.

OPTIONS:
    --input, -i <path> CLI flag for input (not necessary, default argument is input.)
    --output, -o <path> specify output file path.
    --loglevel, -l <debug|info|warning|error|fatal> set the log level  ('0:DEBUG', '1:INFO', '2:WARNING', '3:ERROR' ,'4:FATAL').
    --format, -f <html|markdown> specify format, by default: html.
    --test, -t [<sequential>] run CI tests.
    --help display this list of options.

Note: Markdown generation is incomplete and should only be used for property based testing.

Modules

Overall Flowchart:

            ┌─────────────────────┐                      ┌───────────────┐
Source ───> │ Lex and Preprocessor│ ───> Token list ───> │ TOCite Parse  │ ──> Token list with identifiers ┐
            └─────────────────────┘           │          └───────────────┘                                 │
                                    │         │                                                            │
                                    │         └──────────────────────────────> Header+Footer list ────>────┤
                                    │                                                                      │
                                    │          ┌──────────────┐                                            │
                                    └────────> │ Markalc Parse│────────── Table ────────────>──────────────│
                                               └──────────────┘                                            │
                                                                                                           │
                                   ┌─────────┐                            ┌─────────────┐                  │
             Final Document  <──── │ HTMLGen │ <──── ParsedObj list <──── │ Main Parser │ <────────────────┘
                                   └─────────┘                            └─────────────┘

Vanilla Markdown

Implemented

HTML passthrough
Lists
Links (relative links within file don't work in VSCode, use Create HTML File and preview that.)
Images (images don't work in VSCode, use Create HTML File and preview that.)
Headers
Styling (bold, italic, bold and italic, strikethrough)
Paragraphs
Quotes
Code blocks
Tables

Design decisions

Lists

A list ends with two endlines, i.e., \r\n\r\n
List items are separated by one endline
List type is determined by the first item.
- NUMBER ; DOT will be ordered list
- - or * will be unordered list
If the first item does not start with NUMDER;DOT;Space| "- " | "* ", the list type will be default to unordered list
Ordered list start number is determined by the first item
- The subsequent counting is +1 for each list item
Two spaces as one level of indentation.
A tab, \t character will be interpreted as four spaces, therefore two levels of indentation.

Header

All headers must be preceded by two endlines, with an exception made for the first line in a file.

\n\n# header\n is a header, otherwise unless the header is the first line in a file it won't be processed as a header.

Styling

When using underscores for styling, there must be a space (or endline on the right) on either side in order for it to be recognised.
Asterisks for styling work as usual.

e.g.

[ ]_em_[ ], [ ]__a strong__[ ], or [ ]_em_$, [ ]__a strong__$

Paragraphs

A paragraph is some characters that does not match:

CodeBlock
Header
ContentTable (Table of Contents object)
List
Quote
Table
Reference

It terminates with two endlines. Any elements mentioned above cannot exist in a paragraph. Otherwise, they will become normal text, and will not be rendered as expected.

Building

To build fmark, dotnet and fable have to be installed first. To build the javascript as well, yarn or npm are needed.

Using the Build script

# build only javascript
build -b js

# build and test every single module
build -b testall

# build js and cli
build -b all

Markdown extensions

Macros

Supported Constructs

These are the supported constructs in the preprocessor.

Supported	Syntax	Description	Tested
Simple Macro	`{% macro name value %}`	Sets the Macro `name` equal to the string `value`	Unit Test
Function Macro	`{% macro name(arg1; arg2) value %}`	Sets the Macro `name` equal to the string `value` with two parameters.	Unit Test
Simple Evaluation	`{{ macro_name }}`	Evaluates the macro `macro_name` and replaces the evaluation with the evaluated body of the macro.	Unit Test
Function Evaluation	`{{ macro_name(arg 1; arg 2) }}`	Evaluates the macro `macro_name` with the arguments `arg 1` and `arg 2` and replaces the evaluation with the evaluated body of the macro.	Unit Test
File Include	`{{ include relative/path/to/file }}`	Includes and preprocesses a file using a relative or absolute path. The macros declared in that file will then be available in the current file	Unit Test
Complex Macro Evaluation	`{{ x( {{ y( {{z}} ; Hello ) }} ; {{z}} ) }}`	Nested macro evaulations are supported. This way, default arguments can be created for other macros.	Unit Test

Supported Features

These are the features that are currently supported by the preprocessor.

Feature	Example	Description	Tested
Simple whitespace control	`{% macro x y %}` evaluates to `y` and not `y` .	Removes whitespace and newlines in macros where one wouldn't expect them to be added to the macro body.	Unit Test
Shadowing of macros through	`{% macro x x %} {% macro y(x) {{ x }} %}` with `{{ y(z) }}` will evaluate to `z` but `{{ x }}` outside of the macro will always evaluate to `x`.	Macros can be shadowed by arguments of other macros.	Unit Test
Nested macros	`{% macro x {% macro y %} %}`	Macro y is only defined inside macro x and cannot be seen outside of the scope of x.	Unit Test
Shadowing of macros through	`{% macro x x %} {% macro y {% macro x z %} {{x}} %} y: {{ y }}, x: {{ x }}` will evaluate to `y: z, x: x`	Macros can be shadowed by other macros which will be used instead for evaluation.	Unit Test
Evaluation of large strings	`{{ x(This is the first argument; This is the second argument) }}`	One can pass large strings as arguments to the macros.	Unit Test
Escaping of characters inside argument	`{{ x(arg 1 with a \); arg 2 with a \;) }}`	One can esape all the special characters inside macros and substitutions	Unit Test
Escaping macros	`\{% macro x y %}`	This will escape the whole macro and not evaluate it	Unit Test
Escaping Subsitutions	`\{{ x }}`	will not evaluate the substitution but instead output it literally	Unit Test
Outputting unmatched subsituttion	`{{ x }}` -> `{{ x }}` if not in scope	If the subsitution is not matched, it will output it as it got it	Unit Test
Nested Evaluations	`{{ x( {{y}} ) }}`	Arguments can now be evaluated inside them.	Unit Test

Usage

To use the preprocessor and the lexer, a string or a list of strings can be used, depending on if there are multiple lines or not. For a single string, the following can be used.

For string, the preprocess and lex functions.

[<EntryPoint>]
let main =
    let inputString = (* Read the string *)

    inputString
    |> preprocess
    |> lex
    ...

For a list of strings, one can use the preprocessList and lexList functions.

[<EntryPoint>]
let main =
    let inputStringList = (* Read the string list *)
    
    inputStringList
    |> preprocessList
    |> lexList
    ...

Example

In markdown using the preprocessor, one can then write the following:

Text before macro
{% macro Hello(arg1; arg2)
This is text inside the macro, with semicolons;
{% macro local(arg1; arg2)
This is the second macro
%}
Now back in the first macro.
{{ local(arg1; arg2) }}
%}
Outside both macros
Should be printed as not in scope: {{ local(arg1; arg2) }}

{{ Hello(arg1; arg2) }}

which then evaluates to

Text before macro
Outside both macros
Should be printed as not in scope: {{ local(arg1; arg2) }}


This is text inside the macro, with semicolons;
Now back in the first macro.

This is the second macro

More complicated macros can also be created by writing html in the macros. Due to the html passthrough in the lexer, the html will be copied over literally to the output html.

Lexer

The lexer supports HTML pass through, which can be used to display raw html in markdown. Using this with macros can give very useful and interesting functions.

Interface to the Parser

The interface to the parser was done using the following Token type, which the parser takes in and can parse.

type Token =
    | CODEBLOCK of string * Language
    | LITERAL of string
    | WHITESPACE of size: int
    | NUMBER of string
    | HASH | PIPE | EQUAL | MINUS | PLUS | ASTERISK | DOT
    | DASTERISK | TASTERISK | UNDERSCORE | DUNDERSCORE | TUNDERSCORE | TILDE | DTILDE
    | TTILDE | LSBRA | RSBRA | LBRA | RBRA | BSLASH | SLASH | LABRA | RABRA | LCBRA
    | RCBRA | BACKTICK | TBACKTICK | EXCLAMATION | ENDLINE | COLON | CARET | PERCENT

Features

Supports escaping of all the special characters defined in Types. This is done by adding a \ in front of the character that should be escaped.

Tokens that match multiple characters can also be escaped by just putting a \ before it. For example, *** can be escaped by writing \***.

Extensibility

It can easily be extended by adding the type of the token to Token above. Then the string has to be linked to the token by adding it as a tuple of type string * Token to a list called charList in the Lexer.

Spreadsheet functionality

Spreadsheet functions will evaluate in-place, if they are incorrectly formatted then Markalc will leave the cell unchanged as if it were normal text inside.

To delineate an expression, start the cell with the = operator, e.g.

Calcs	39	42
	`=6*5+SUM{4,5}`	`=[1,1]+3`

Supported

Arithmetic

Function	Syntax	Effect	Tested
Modulo	`a%b`	a modulo b	Property based, all integer inputs.
Power	`a^b`	a to the power of b	Property based, all integer inputs.
Multiply	`a*b`	a times b	Property based, all integer inputs.
Division	`a/b`	a divided by b	Property based, all integer inputs.
Subtraction	`a-b`	a minus b	Property based, all integer inputs.
Addition	`a+b`	a plus b	Property based, all integer inputs.
Brackets	`a+(b-c)`	Specify order of operation.	Unit tested.
Cell References	`[0,2]`	Evaluates to contents of cell specified (row 0, col 2)	Unit tested
Range Cell Ref	`[0,0]:[2,0]`	Evaluates to list of cell references.	Unit tested.
Keyword Cell Ref	`[col=2,row=0]`	Alternative syntax, order doens't matter.	Unit tested
Specify Decimal Places	`=5.333,2`	Comma then a number specifies number of decimal places.	Unit tested

Cell reference behaviour

Referenced cell	Return
Contains valid expression `=...`	Contents of cell
Does not contain valid expression	nan
Is not in table	nan
Contains circular reference	nan

Functions

Supported functions:

Function name	Effect	Tested
`SUM`	Adds all arguments.	Unit tested.
`AVG`	Calculates mean of all arguments.	Unit tested.
`MIN`	Returns minimum of all arguments.	Unit tested.
`MAX`	Returns maximum of all arguments.	Unit tested.

Functions support Range Cell References: [0,0]:[2,0] will evaluate to [0,0],[1,0],[2,0]. This can be used with other arguments, e.g. SUM{[0,0]:[2,0],1,3+2,[0,0]}

Unsupported

Feature	Example	Workaround
Negative numbers	`=-a+b`	Subtract number from zero in brackets e.g. `=(0-a)+b`
Integer operations	`=5+2`	All calculations done in floating point.
Allow formatting around equation	`=5+2`	None
String operations	e.g. Excel CONCAT	None
Assume empty cells are zero	Summing over column with empty cells	Put `=0` in each cell

Citations

FMark supports simple footers and styled references.

If multiple citations with the same reference ID is found, the first citation will be used inline.
- One [^fmark] inline, and two [^fmark], some author. The first will be used.
Reference to non-exist citation will result in "[Reference: %refID not found!]" in the HTML generated.
The list is sorted, with footnotes in order of their numerical IDs first, References in order of apperance after. Thus the order each explanatory text comes in the fmark file does not matter.

Simple Footers

Simple footers have numerical IDs.

this is inline[^1], and so on.

[^1], this is the text to explain such line.
      this line as well.
But not this line.

The body will show up with a superscripted link:

this is inline¹ , and so on.

But not this line.

The footer texts will be collected at the end of the document.

1: this is the text to explain such line. this line as well.

Styled References

Styled references have alphabetic IDs.

Supported rendering styles:

	Chicago	Harvard	IEEE
Book	Yes	Yes	-
Website	Yes	Yes	-

Supported data fields

Field	Explanation
type	`'Book'` or `'Website'`
author	Author with surname at the end
title	Title
year	The year it is written
url	Address for website
access	Date of access for websites, in `yyyy-mm-dd` format

Pick a style with %%RefStyle, or it will default to Harvard.

%%RefStyle = Harvard

Then follow field1=data1, field2=data2, ... to use references.

Spaces around equal sign are allowed, so f = d, f= d, f =d or f=d all have the same effect.

This is a citation[^Mark]. This is another[^FMark] one.

[^Mark], type= Book, author= Mark Smith, title= Not a real book, year= 2018
[^FMark], type= Website, author= FMark Smith, title= Not a real website, year= 2017 url= www.example.com/website access= 2018-3-4

With Harvard, it will look like this:

This is a citation(Smith, 2018). This is another(Smith, 2017) one.

At the end of the document:

Smith, M. (2018) Not a real book.

Smith, F. (2017) Not a real website. Available from: www.example.com/website [Accessed 4th March 2018].

if Chicago style is chosen:

This is a citation(Smith 2018). This is another(Smith 2017) one.

At the end of the document:

Mark Smith. 2018. Not a real book.

FMark Smith. 2017. "Not a real website." Accessed March 3, 2018. https://www.example.com/website

Latex maths rendering

Fmark includes support for Latex maths rendering.

Surround an equation written in Latex with $$ in order to use this feature.

Example: $$2+2=4-1=3$$

Name		Name	Last commit message	Last commit date
Latest commit History 571 Commits
FMark		FMark
examples		examples
.gitattributes		.gitattributes
.gitignore		.gitignore
.travis.yml		.travis.yml
LICENSE		LICENSE
README.md		README.md
build.sh		build.sh

Feature	Example	Exaplanation
depth	`%%TOC depth=3`	Build a TOC with headers level less than 3
exclude	`%%TOC excludes=[Appendix;Acknowledgement]`	Exclude headers with name `Appendix` and `Acknowledgement`

License

Zestylogic/FMark

Folders and files

Latest commit

History

Repository files navigation