Skip to content

Commit 38ce99f

Browse files
committed
Improved include file filter
- Raw includes - Including/excluding on different output formats. - Variable substitution in include paths. - Documentation. - Tests.
1 parent 6813921 commit 38ce99f

File tree

8 files changed

+264
-32
lines changed

8 files changed

+264
-32
lines changed

include-files/Makefile

+9-4
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,20 @@
11
DIFF ?= diff --strip-trailing-cr -u
22

3+
.EXPORT_ALL_VARIABLES:
4+
5+
SUBDIR_NAME=subdir
6+
7+
38
test: sample.md file-a.md file-b.md file-c.md include-files.lua
4-
@pandoc --lua-filter=include-files.lua --to=native $< \
9+
@pandoc --lua-filter=include-files.lua -M subdir-name="subdir" --to=native $< \
510
| $(DIFF) expected.native -
6-
@pandoc --lua-filter=include-files.lua -M include-auto --to=native $< \
11+
@pandoc --lua-filter=include-files.lua -M subdir-name="subdir" -M include-auto --to=native $< \
712
| $(DIFF) expected-auto.native -
813

914
expected.native: sample.md file-a.md file-b.md file-c.md include-files.lua
10-
pandoc --lua-filter=include-files.lua --output $@ $<
15+
pandoc --lua-filter=include-files.lua -M subdir-name="subdir" --output $@ $<
1116

1217
expected-auto.native: sample.md file-a.md file-b.md file-c.md include-files.lua
13-
pandoc --lua-filter=include-files.lua -M include-auto --output $@ $<
18+
pandoc --lua-filter=include-files.lua -M subdir-name="subdir" -M include-auto --output $@ $<
1419

1520
.PHONY: test

include-files/README.md

+53-5
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,12 @@ be convenient to modify the level of headers; a top-level header
1818
in an included file should be a second or third-level header in
1919
the final document.
2020

21-
#### Manual shifting
21+
#### Manual Shifting
2222

2323
Use the `shift-heading-level-by` attribute to control header
2424
shifting.
2525

26-
#### Automatic shifting
26+
#### Automatic Shifting
2727

2828
1. Add metadata `-M include-auto` to enable automatic shifting.
2929
2. Do not specify `shift-heading-level-by`
@@ -52,14 +52,56 @@ file-a.md
5252
Comment lines can be added in the include block by beginning a
5353
line with two `//` characters.
5454

55-
### Different formats
55+
### Different Formats
5656

5757
Files are assumed to be written in Markdown, but sometimes one
5858
will want to include files written in a different format. An
5959
alternative format can be specified via the `format` attribute.
60-
Only plain-text formats are accepted.
60+
Only plain-text formats are accepted. The default format for all includes
61+
can be set by the meta data variable `include-format`, which is useful
62+
if you want all your files to be parsed in the same way as your main document,
63+
e.g. with extensions.
6164

62-
### Recursive transclusion
65+
### Filter by Formats
66+
67+
Files can be included and excluded on different formats by
68+
using attribute `include-if-format=formatA;formatB;...`
69+
and `exclude-if-format=formatA;formatB;...`, e.g.
70+
71+
````md
72+
```{.include include-if-format=native;html;latex}
73+
subdir/file-h.md
74+
```
75+
76+
```{.include exclude-if-format=native;commonmark}
77+
subdir/file-i.md
78+
```
79+
````
80+
81+
### Variable Substitution in Paths
82+
83+
If attribute `var-replace` is used, the patterns `${meta:<meta-var>}` or `${env:<env-var>}`
84+
will be replaced by the corresponding meta data variable `<meta-var>` in the document or the
85+
environment variable `<env-var>`, e.g.
86+
87+
````md
88+
```{.include .var-replace}
89+
${meta:subdir-name}/file-h.md
90+
${env:SUBDIR_NAME}/file-h.md
91+
```
92+
````
93+
94+
### Raw Includes
95+
96+
You can also include the files in a raw block by using `raw=true`, e.g.
97+
98+
````md
99+
```{.include raw=true format=latex include-if-format=latex}
100+
subdir/file-h-latex.md
101+
```
102+
````
103+
104+
### Recursive Transclusion
63105

64106
Included files can in turn include other files. Note that all
65107
filenames must be relative to the directory from which they are
@@ -70,6 +112,12 @@ this case `b/c.md`. The full relative path will be automatically
70112
generated in the final document. The same goes for image paths and
71113
codeblock file paths using the `include-code-files` filter.
72114

115+
### Missing Includes
116+
117+
You can set the meta data variable `include-fail-if-read-error` to `true`
118+
such that any not found include will fail the convertion and error out
119+
immediateley.
120+
73121
## Example
74122

75123
Let's assume we are writing a longer document, like a thesis.

include-files/expected-auto.native

+9
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,15 @@
2323
,Header 2 ("source-include",[],[]) [Str "Source",Space,Str "include"]
2424
,Para [Str "File",Space,Str "inclusion",Space,Str "codeblocks",Space,Str "for",Space,Str "use",Space,Str "with",Space,Str "include-code-files",Space,Str "will",Space,Str "be",SoftBreak,Str "updated",Space,Str "too."]
2525
,CodeBlock ("",["c"],[("include","subdir/somecode.c")]) ""
26+
,Header 1 ("includeexclude-if-format",[],[]) [Str "Include/exclude",Space,Str "if",Space,Str "format"]
27+
,Para [Str "The",Space,Str "next",Space,Str "document",Space,Str "should",Space,Str "be",Space,Str "included",Space,Str "in",Space,Str "formats",Space,Code ("",[],[]) "native, html, latex",Str "."]
28+
,Header 2 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
29+
,Para [Str "The",Space,Str "next",Space,Str "document",Space,Str "should",Space,Str "not",Space,Str "be",Space,Str "included",Space,Str "in",Space,Str "formats",Space,Code ("",[],[]) "native, commonmark",Str "."]
30+
,Header 1 ("meta-and-env-var-replacement",[],[]) [Str "Meta",Space,Str "and",Space,Str "env",Space,Str "var",Space,Str "replacement"]
31+
,Header 2 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
32+
,Header 2 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
33+
,Header 1 ("raw-include",[],[]) [Str "Raw",Space,Str "include"]
34+
,RawBlock (Format "latex") "\\section{Some other stuff}\n"
2635
,Header 1 ("appendix",[],[]) [Str "Appendix"]
2736
,Para [Str "More",Space,Str "info",Space,Str "goes",Space,Str "here."]
2837
,Header 2 ("questionaire",[],[]) [Str "Questionaire"]

include-files/expected.native

+9
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,15 @@
2323
,Header 1 ("source-include",[],[]) [Str "Source",Space,Str "include"]
2424
,Para [Str "File",Space,Str "inclusion",Space,Str "codeblocks",Space,Str "for",Space,Str "use",Space,Str "with",Space,Str "include-code-files",Space,Str "will",Space,Str "be",SoftBreak,Str "updated",Space,Str "too."]
2525
,CodeBlock ("",["c"],[("include","subdir/somecode.c")]) ""
26+
,Header 1 ("includeexclude-if-format",[],[]) [Str "Include/exclude",Space,Str "if",Space,Str "format"]
27+
,Para [Str "The",Space,Str "next",Space,Str "document",Space,Str "should",Space,Str "be",Space,Str "included",Space,Str "in",Space,Str "formats",Space,Code ("",[],[]) "native, html, latex",Str "."]
28+
,Header 1 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
29+
,Para [Str "The",Space,Str "next",Space,Str "document",Space,Str "should",Space,Str "not",Space,Str "be",Space,Str "included",Space,Str "in",Space,Str "formats",Space,Code ("",[],[]) "native, commonmark",Str "."]
30+
,Header 1 ("meta-and-env-var-replacement",[],[]) [Str "Meta",Space,Str "and",Space,Str "env",Space,Str "var",Space,Str "replacement"]
31+
,Header 1 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
32+
,Header 1 ("some-other-stuff",[],[]) [Str "Some",Space,Str "other",Space,Str "stuff"]
33+
,Header 1 ("raw-include",[],[]) [Str "Raw",Space,Str "include"]
34+
,RawBlock (Format "latex") "\\section{Some other stuff}\n"
2635
,Header 1 ("appendix",[],[]) [Str "Appendix"]
2736
,Para [Str "More",Space,Str "info",Space,Str "goes",Space,Str "here."]
2837
,Header 2 ("questionaire",[],[]) [Str "Questionaire"]

include-files/include-files.lua

+150-23
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,99 @@ PANDOC_VERSION:must_be_at_least '2.12'
99
local List = require 'pandoc.List'
1010
local path = require 'pandoc.path'
1111
local system = require 'pandoc.system'
12+
local cs = PANDOC_STATE
1213

13-
--- Get include auto mode
14+
-- This is the codeblock-var-replace
15+
-- filter directly copied, since we
16+
-- cannot run Lua filters inside this filter
17+
-- https://github.com/jgm/pandoc/issues/6830
18+
-- We replace variables in include blocks.
19+
20+
local sys = require 'pandoc.system'
21+
local utils = require 'pandoc.utils'
22+
-- local ut = require "module-lua.utils"
23+
24+
-- Save env. variables
25+
local env = sys.environment()
26+
27+
-- Save meta table and metadata
28+
local meta
29+
function save_meta (m)
30+
meta = m
31+
end
32+
33+
--- Replace variables in code blocks
34+
local metaMap
35+
local function var_replace_codeblocks (cb)
36+
--- Replace variable with values from environment
37+
--- and meta data (stringifing).
38+
local function replace(what, var)
39+
local repl = nil
40+
if what == "env" then
41+
repl = env[var]
42+
elseif what == "meta" then
43+
local v = metaMap[var]
44+
if v then
45+
repl = utils.stringify(v)
46+
end
47+
end
48+
49+
if repl == nil then
50+
io.stderr:write("Could not replace variable in codeblock: '".. var .."'\n")
51+
end
52+
53+
return repl
54+
end
55+
56+
-- ignore code blocks which are not of class "var-replace".
57+
if not cb.classes:includes 'var-replace' then
58+
return
59+
end
60+
61+
cb.text = cb.text:gsub("%${(%l+):([^}]+)}", replace)
62+
end
63+
64+
--- Include/exclude by attribute
65+
--- `exclude-if-format='formatA;formatB;...'
66+
--- `include-if-format='formatA;formatB;...`
67+
--- Default: true
68+
local function is_included(cb)
69+
local include = true
70+
local exclude = false
71+
72+
if cb.attributes['include-if-format'] then
73+
include = cb.attributes['include-if-format']:match(FORMAT) ~= nil
74+
end
75+
76+
if cb.attributes['exclude-if-format'] then
77+
exclude = cb.attributes['exclude-if-format']:match(FORMAT) ~= nil
78+
end
79+
80+
return include == true and exclude == false
81+
end
82+
83+
--- Get default settings
1484
local include_auto = false
85+
local default_format = nil
86+
local include_fail_if_read_error = false
87+
1588
function get_vars (meta)
1689
if meta['include-auto'] then
1790
include_auto = true
1891
end
92+
93+
if meta['include-fail-if-read-error'] then
94+
include_fail_if_read_error = true
95+
end
96+
97+
-- If this is nil, markdown is used as a default format.
98+
default_format = meta['include-format']
99+
100+
-- Save meta table for var_replace
101+
metaMap = meta
19102
end
20103

104+
21105
--- Keep last heading level found
22106
local last_heading_level = 0
23107
function update_last_level(header)
@@ -62,8 +146,23 @@ function transclude (cb)
62146
return
63147
end
64148

65-
-- Markdown is used if this is nil.
149+
-- Filter by includes and excludes
150+
if not is_included(cb) then
151+
return List{} -- remove block
152+
end
153+
154+
-- Variable substitution
155+
var_replace_codeblocks(cb)
156+
66157
local format = cb.attributes['format']
158+
if not format then
159+
-- Markdown is used if this is nil.
160+
format = default_format
161+
end
162+
163+
-- Check if we include the file as raw inline
164+
local raw = cb.attributes['raw']
165+
raw = raw == "true"
67166

68167
-- Attributes shift headings
69168
local shift_heading_level_by = 0
@@ -77,35 +176,63 @@ function transclude (cb)
77176
end
78177
end
79178

80-
--- keep track of level before recusion
179+
--- Keep track of level before recursion
81180
local buffer_last_heading_level = last_heading_level
82181

83182
local blocks = List:new()
84183
for line in cb.text:gmatch('[^\n]+') do
85-
if line:sub(1,2) ~= '//' then
86-
local fh = io.open(line)
87-
if not fh then
88-
io.stderr:write("Cannot open file " .. line .. " | Skipping includes\n")
184+
if line:sub(1,2) == '//' then
185+
goto skip_to_next
186+
end
187+
188+
if cs.verbosity == "INFO" then
189+
io.stderr:write(string.format("Including: [format: %s, raw: %s]\n - '%s'\n",
190+
format,
191+
tostring(raw), line))
192+
end
193+
194+
local fh = io.open(line)
195+
if not fh then
196+
local cwd = system.get_working_directory()
197+
local msg = "Cannot find include file: '" .. line .. "' in working dir: '" .. cwd .. "'"
198+
if include_fail_if_read_error then
199+
io.stderr:write(msg .. " | error\n")
200+
error("Abort due to include failure")
89201
else
90-
local contents = pandoc.read(fh:read '*a', format).blocks
91-
last_heading_level = 0
92-
-- recursive transclusion
93-
contents = system.with_working_directory(
94-
path.directory(line),
95-
function ()
96-
return pandoc.walk_block(
97-
pandoc.Div(contents),
98-
{ Header = update_last_level, CodeBlock = transclude }
99-
)
100-
end).content
101-
--- reset to level before recursion
102-
last_heading_level = buffer_last_heading_level
103-
blocks:extend(update_contents(contents, shift_heading_level_by,
104-
path.directory(line)))
105-
fh:close()
202+
io.stderr:write(msg .. " | skipping include\n")
203+
goto skip_to_next
106204
end
107205
end
206+
207+
-- Read the file
208+
local text = fh:read('*a')
209+
fh:close()
210+
211+
if raw then
212+
-- Include as raw inline element
213+
blocks:extend({pandoc.RawBlock(format, text)})
214+
else
215+
-- Inlcude as parsed AST
216+
local contents = pandoc.read(text, format).blocks
217+
last_heading_level = 0
218+
-- Recursive transclusion
219+
contents = system.with_working_directory(
220+
path.directory(line),
221+
function ()
222+
return pandoc.walk_block(
223+
pandoc.Div(contents),
224+
{ Header = update_last_level, CodeBlock = transclude }
225+
)
226+
end).content
227+
--- Reset to level before recursion
228+
last_heading_level = buffer_last_heading_level
229+
blocks:extend(update_contents(contents, shift_heading_level_by,
230+
path.directory(line)))
231+
end
232+
233+
::skip_to_next::
108234
end
235+
109236
return blocks
110237
end
111238

include-files/sample.md

+32
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,38 @@ file-f.md
3737
subdir/file-g.md
3838
```
3939

40+
# Include/exclude if format
41+
42+
The next document should be included in formats `native, html, latex`.
43+
44+
```{.include include-if-format=native;html;latex}
45+
subdir/file-h.md
46+
```
47+
48+
The next document should not be included in formats `native, commonmark`.
49+
50+
```{.include exclude-if-format=native;commonmark}
51+
subdir/file-i.md
52+
```
53+
54+
# Meta and env var replacement
55+
56+
```{.include .var-replace}
57+
// Replace meta data variable in path
58+
${meta:subdir-name}/file-h.md
59+
```
60+
61+
```{.include .var-replace}
62+
// Replace envrionment variable in path
63+
${env:SUBDIR_NAME}/file-h.md
64+
```
65+
66+
# Raw include
67+
68+
```{.include raw=true format=latex}
69+
subdir/file-h-latex.md
70+
```
71+
4072
# Appendix
4173

4274
More info goes here.

include-files/subdir/file-h-latex.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
\section{Some other stuff}

include-files/subdir/file-h.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Some other stuff

0 commit comments

Comments
 (0)