Literate files

[dbaynard]

Hello,

I've been through the language addition guide, and would like some advice on implementing support for literate files (e.g. literate haskell, *.lhs).

In a literate file, the code is demarcated, whereas the comments are not. Traditionally literate haskell files used bird tracks.

This is a text description, that is stripped by the _literate preprocessor_.

> -- This is a normal comment
> example :: IO ()
> example = print "This is code"

More common now is the markdown style.

This is a text description, that is stripped by the _literate preprocessor_.

```haskell
-- This is a normal comment
example :: IO ()
example = print "This is code"
```

In either case, rather than demarcating the comments, it is the code which is demarcated, and therefore it does not appear that tokei can simply support such a file by extending the languages.json file.

How can tokei support such a syntax?

[XAMPPRocky]

@dbaynard Hello and thank you for this issue. This will require being added as a code path I think tokei can maybe support this syntax though I have a couple of questions about literate files. How many other types of styles are common? Can a *.lhs file have non haskell code fences and still be valid to the compiler?

[dbaynard]

The best resource on literate haskell files is the readme at https://github.com/wenkokke/unlit.

However, this applies beyond haskell.
For example, it may be useful to know how many lines of code are in tutorials written in markdown (or latex).

How many other types of styles are common?

unlit lists 6 types. In addition to the 'bird' style, there are two markdown styles (triple tildes or triple backticks) and three others:

LaTeX
\begin{code}
\end{code}

Org mode
#+BEGIN_SRC haskell
#+END_SRC

Jekyll
{% highlight haskell %}
{% endhighlight %}

Of these, I've only encountered the bird track, markdown and latex forms in the wild (and the latex form seems to be falling out of favour).

Can a *.lhs file have non haskell code fences and still be valid to the compiler?

With fences, haskell must be present.

[benjaminselfridge]

@XAMPPRocky Any progress on this? I'd love this feature, and might be up for implementing a patch for it.

[foxyseta]

However, this applies beyond haskell. For example, it may be useful to know how many lines of code are in tutorials written in markdown (or latex).

True! Linguist and go-enry detect "Literate Haskell/Agda/.." as their own language, though:

• https://github.com/github-linguist/linguist/blob/be39d09a3789d068afb33533f285df02d8e9c367/lib/linguist/;
• https://github.com/search?q=language%3A%22Literate+Agda%22&type=codelanguages.yml#L3972.

Non-literate code blocks and listings in typesetting systems, on the other hand, are just detected as code written in the typesetting system.