Weaver

1. Introduction

Here is an overview of the weave program. This file turns a literate source file into one or more html files.

{weaver.d 1}
{Imports, 16}

void weave(Program p) {
    {Parse use locations, 2}
    {Run weaveChapter, 2}
    if (isBook && !noOutput) {
{Create the table of contents, 3}
    }
}

{WeaveChapter, 5}
{LinkLocations function, 12}

2. Parsing Codeblocks

Now we parse the codeblocks across all chapters in the program. We have four arrays:

{Parse use locations 2}
string[string] defLocations;
string[][string] redefLocations;
string[][string] addLocations;
string[][string] useLocations;

foreach (chapter; p.chapters) {
    foreach (s; chapter.sections) {
        foreach (block; s.blocks) {
            if (block.isCodeblock) {
                if (block.modifiers.canFind(Modifier.noWeave)) {
                    defLocations[block.name] = "noWeave";
                    continue;
                }

                {Check if it's a root block, 4}

                if (block.modifiers.canFind(Modifier.additive)) {
                    if (block.name !in addLocations || !addLocations[block.name].canFind(to!string(s.num)))
                        addLocations[block.name] ~= chapter.num() ~ ":" ~ to!string(s.num);
                } else if (block.modifiers.canFind(Modifier.redef)) {
                    if (block.name !in redefLocations || !redefLocations[block.name].canFind(to!string(s.num)))
                        redefLocations[block.name] ~= chapter.num() ~ ":" ~ to!string(s.num);
                } else {
                    defLocations[block.name] = chapter.num() ~ ":" ~ to!string(s.num);
                }

                foreach (lineObj; block.lines) {
                    string line = strip(lineObj.text);
                    if (line.startsWith("@{") && line.endsWith("}")) {
                        useLocations[line[2..$ - 1]] ~= chapter.num() ~ ":" ~ to!string(s.num);
                    }
                }
            }
        }
    }
}

Used in section 1

Here we simply loop through all the chapters in the program and get the html for them. If noOutput is false, we generate html files in the outDir.

{Run weaveChapter 2}
foreach (c; p.chapters) {
    string output = weaveChapter(c, p, defLocations, redefLocations,
                                 addLocations, useLocations);
    if (!noOutput) {
        string dir = outDir;
        if (isBook) {
            dir = outDir ~ "/_book";
            if (!dir.exists()) {
                mkdir(dir);
            }
        }
        File f = File(dir ~ "/" ~ stripExtension(baseName(c.file)) ~ ".html", "w");
        f.write(output);
        f.close();
    }
}

Used in section 1

3. Table of contents

If the program being compiled is a book, we should also write a table of contents file.

{Create the table of contents 3}
string dir = outDir ~ "/_book";
File f = File(dir ~ "/" ~ p.title ~ "_contents.html", "w");
f.writeln(
q"DELIMITER
<!DOCTYPE html>
<html>
<head>
<style>
ul#contents {list-style-type: none;}a{color:#337ABF;}body{min-width:200px;max-width:850px;margin:0 auto;padding:30px;}.chapter-nav{font-size: 10pt;}.codeblock_name,code,pre.prettyprint{font-family:Monaco,"Lucida Console",monospace}body{font-size:14pt}.codeblock_name,.math,.seealso,code{font-size:10pt}.codeblock{page-break-inside:avoid;padding-bottom:15px}.math{text-indent:0}pre.prettyprint{font-size:10pt;padding:10px;border-radius:10px;border:none;white-space:pre-wrap}.codeblock_name{margin-top:1.25em;display:block}a:link{text-decoration:none}a:link:not(.lit):hover{text-decoration:underline}a:link:active{color:red}h4{padding-right:1.25em}h4.noheading{margin-bottom:0}h1{text-align:center}code{padding:2px}pre{-moz-tab-size:4;-o-tab-size:4;tab-size:4}.two-col{list-style-type:none}.two-col li:before{content:'-';padding:5px;margin-right:5px;color:orange;background-color:#fff;display:inline-block}@media print{body{font-size:10pt}pre.prettyprint{font-size:8pt}.seealso{font-size:9pt}.codeblock_name,.math,code{font-size:8pt}.math{text-indent:0}}
</style>
</head>
<body>
<div class="container">
DELIMITER"
);

f.writeln("<h1>" ~ p.title ~ "</h1>");

f.writeln(filterMarkdown(p.text));

f.writeln("<ul id=\"contents\">");
foreach (c; p.chapters) {
    f.writeln("<li>" ~ c.num() ~ ". <a href=\"" ~ stripExtension(baseName(c.file)) ~ ".html\">" ~ c.title ~ "</a></li>");
}

f.writeln("
</ul>
</div>
</body>
");

f.close();

Used in section 1

4. Root block check

We check if the block is a root code block. We check this using a regex that basically checks if it the name has an extension. Additionally, users can put the block name in quotes to force it to be a root block.

If the block name is in quotes, we have to make sure to remove those once we're done.

{Check if it's a root block 4}
auto fileMatch = matchAll(block.name, regex(".*\\.\\w+"));
auto quoteMatch = matchAll(block.name, regex("^\".*\"$"));
if (fileMatch || quoteMatch) {
    block.isRootBlock = true;
    if (quoteMatch) {
        block.name = block.name[1..$-1];
    }
}

Used in section 2

5. WeaveChapter

This function weaves a single chapter.

{WeaveChapter 5}
string weaveChapter(Chapter c, Program p, string[string] defLocations,
                    string[][string] redefLocations, string[][string] addLocations,
                    string[][string] useLocations) {
{prettify}
{css}
{katex}

    string output = "";

    {Write the head of the HTML, 6}
    {Write the body, 7}

    if (use_katex) {
{Write the katex source, 16}
    }

    if (isBook) {
        output ~= "<br>";
        int index = cast(int) p.chapters.countUntil(c);
        if (index - 1 >= 0) {
            Chapter lastChapter = p.chapters[p.chapters.countUntil(c)-1];
            output ~= "<a style=\"float:left;\" class=\"chapter-nav\" href=\"" ~ stripExtension(baseName(lastChapter.file)) ~ ".html\">Previous Chapter</a>";
        }
        if (index + 1 < p.chapters.length) {
            Chapter nextChapter = p.chapters[p.chapters.countUntil(c)+1];
            output ~= "<a style=\"float:right;\" class=\"chapter-nav\" href=\"" ~ stripExtension(baseName(nextChapter.file)) ~ ".html\">Next Chapter</a>";
        }
    }

    output ~= "</body>\n";

    return output;
}

Used in section 1

6. Head of the Document

This writes out the start of the document. Mainly the scripts (prettify.js) and the css (prettiy css, default css, and colorscheme css). It also adds the title of the document.

{Write the head of the HTML 6}
string prettifyExtension;
foreach (cmd; p.commands) {
    if (cmd.name == "@overwrite_css") {
        defaultCSS = readall(File(cmd.args));
    } else if (cmd.name == "@add_css") {
        defaultCSS ~= readall(File(cmd.args));
    } else if (cmd.name == "@colorscheme") {
        colorschemeCSS = readall(File(cmd.args));
    }

    if (cmd.name == "@code_type") {
        if (cmd.args.length > 1) {
            string ext = cmd.args.split()[1][1..$];
            if (ext in extensions) {
                prettifyExtension = "<script>\n" ~ extensions[ext] ~ "</script>\n";
            }
        }
    }
}
foreach (cmd; c.commands) {
    if (cmd.name == "@overwrite_css") {
        defaultCSS = readall(File(cmd.args));
    } else if (cmd.name == "@add_css") {
        defaultCSS ~= readall(File(cmd.args));
    } else if (cmd.name == "@colorscheme") {
        colorschemeCSS = readall(File(cmd.args));
    }

    if (cmd.name == "@code_type") {
        if (cmd.args.length > 1) {
            string ext = cmd.args.split()[1][1..$];
            if (ext in extensions) {
                prettifyExtension = "<script>\n" ~ extensions[ext] ~ "</script>\n";
            }
        }
    }
}

string css = colorschemeCSS ~ defaultCSS;
string scripts = "<script>\n" ~ prettify ~ "</script>\n";
scripts ~= prettifyExtension;

bool use_katex = false;

output ~= "<!DOCTYPE html>\n"
             "<html>\n"
             "<head>\n"
             "<meta charset=\"utf-8\">\n"
             "<title>" ~ c.title ~ "</title>\n" ~
             scripts ~
             "<style>\n" ~
             css ~
             "</style>\n"
             "</head>\n";

Used in section 5

7. Parse the Chapter

Now we write the body -- this is the meat of the weaver. First we write a couple things at the beginning: making sure the prettyprint function is called when the page loads, and writing out the title as an h1.

Then we loop through each section in the chapter. At the beginning of each section, we write the title, and an empty a link so that the section title can be linked to. We also have to determine if the section title should be a noheading class. If the section title is empty, then the class should be noheading which means that the prose will be moved up a bit towards it -- otherwise it looks like there is too much empty space between the title and the prose.

{Write the body 7}
output ~= "<body onload=\"prettyPrint()\">\n"
          "<section>\n"
          "<h1>" ~ c.title ~ "</h1>\n";

foreach (s; c.sections) {
    string noheading = s.title == "" ? " class=\"noheading\"" : "";
    output ~= "<a name=\"" ~ c.num() ~ ":" ~ to!string(s.num) ~ "\"><div class=\"section\"><h4" ~
              noheading ~ ">" ~ to!string(s.num) ~ ". " ~ s.title ~ "</h4></a>\n";

    foreach (block; s.blocks) {
        if (!block.modifiers.canFind(Modifier.noWeave)) {
            if (!block.isCodeblock) {
                {Weave a prose block, 8}
            } else {
                {Weave a code block, 9}
            }
        }
    }
    output ~= "</div>\n";
}

Used in section 5

8. Weave a prose block

Weaving a prose block is not very complicated. First we get the html by sending it through the markdown compiler by using the filterMarkdown function. We disable the use of underscores for italics because that interferes with the katex use of underscores for math.

Then use a giant regex to determine if any katex is being used. Really what that regex does is determine if there is something in dollar signs or double dollar signs which aren't escaped by backslashes. Most of the regex is negative look behind to make sure the dollar sign isn't escaped.

{Weave a prose block 8}
string html = filterMarkdown(block.text(), MarkdownFlags.disableUnderscoreEmphasis);

if (html.matchAll(regex(r"(?<!\\)[\$](?<!\\)[\$](.*?)(?<!\\)[\$](?<!\\)[\$]")) || html.matchAll(regex(r"(?<!\\)[\$](.*?)(?<!\\)[\$]"))) {
    use_katex = true;
}

Used in section 7

Here we use the same regex to actually perform the substitution. Double dollars mean a block math which means we have to use a div. For inline math (single dollars) we use a span. After that substitution we replace all backslash dollars to real dollar signs.

Finally we add this html to the output and add a newline for good measure.

{Weave a prose block 8} +=
html = html.replaceAll(regex(r"(?<!\\)[\$](?<!\\)[\$](.*?)(?<!\\)[\$](?<!\\)[\$]"), "<div class=\"math\">$1</div>");
html = html.replaceAll(regex(r"(?<!\\)[\$](.*?)(?<!\\)[\$]"), "<span class=\"math\">$1</span>");
html = html.replaceAll(regex(r"\\\$"), "$$");
output ~= html ~ "\n";

Used in section 7

9. Weave a code block

{Weave a code block 9}
output ~= "<div class=\"codeblock\">\n";

{Write the title out, 10}
{Write the actual code, 11}
{Write the 'added to' links, 13}
{Write the 'redefined in' links, 15}
{Write the 'used in' links, 14}

output ~= "</div>\n";

Used in section 7

10. The codeblock title

Here we create the title for the codeblock. For the title, we have to link to the definition (which is usually the current block, but sometimes not because of +=). We also need to make the title bold (<strong>) if it is a root code block.

{Write the title out 10}
{Find the definition location, 10}
{Make the title bold if necessary, 10}

output ~= "<span class=\"codeblock_name\">{" ~ name ~ 
          " <a href=\"" ~ htmlFile ~ "#" ~ def ~ "\">" ~ defLocation ~ "</a>}" ~ extra ~ "</span>\n";

Used in section 9

To find the definition location we use the handy defLocation array that we made earlier. The reason we have both the variables def and defLocation is because the definition location might be in another chapter, in which case it should be displayed as chapterNum:sectionNum but if it's in the current file, the chapterNum can be removed. def gives us the real definition location, and defLocation is the one that will be used -- it strips out the chapterNum if necessary.

{Find the definition location 10}
string chapterNum;
string def;
string defLocation;
string htmlFile = "";
if (block.name !in defLocations) {
    error(c.file, block.startLine, "{" ~ block.name ~ "} is never defined");
} else {
    def = defLocations[block.name];
    defLocation = def;
    auto index = def.indexOf(":");
    string chapter = def[0..index];
    htmlFile = getChapterHtmlFile(p.chapters, chapter);
    if (chapter == c.num()) {
        defLocation = def[index + 1..$];
    }
}

We also add the += or := if necessary. This needs to be the extra because it goes outside the {} and is not really part of the name anymore.

{Find the definition location 10} +=
string extra = "";
if (block.modifiers.canFind(Modifier.additive)) {
    extra = " +=";
} else if (block.modifiers.canFind(Modifier.redef)) {
    extra = " :=";
}

We simple put the title in in a strong tag if it is a root codeblock to make it bold.

{Make the title bold if necessary 10}
string name;
if (block.isRootBlock) {
    name = "<strong>" ~ block.name ~ "</strong>";
} else {
    name = block.name;
}

11. The actual code

At the beginning, we open the pre tag. If a codetype is defined, we tell the prettyprinter to use that, otherwise, the pretty printer will try to figure out how to syntax highlight on its own -- and it's pretty good at that.

{Write the actual code 11}
if (block.codeType.split().length > 1) {
    output ~= "<pre class=\"prettyprint lang-" ~ block.codeType.split()[1][1..$] ~ "\">\n";
} else {
    output ~= "<pre class=\"prettyprint\">\n";
}

foreach (lineObj; block.lines) {
    {Write the line, 11}
}
output ~= "</pre>\n";

Used in section 9

Now we loop through each line. The only complicated thing here is if the line is a codeblock use. Then we have to link to the correct definition location.

Also we escape all ampersands and greater than and less than signs before writing them.

{Write the line 11}
string line = lineObj.text;
string strippedLine = strip(line);
if (strippedLine.startsWith("@{") && strippedLine.endsWith("}")) {
    {Link a used codeblock, 11}
} else { 
    output ~= line.replace("&", "&amp;").replace(">", "&gt;").replace("<", "&lt;") ~ "\n";
}

For linking the used codeblock, it's pretty much the same deal as before. We reuse the def and defLocation variables. We also write the final html as a span with the nocode class, that way it won't be syntax highlighted by the pretty printer.

{Link a used codeblock 11}
def = "";
defLocation = "";
if (strip(strippedLine[2..$ - 1]) !in defLocations) {
    error(c.file, lineObj.lineNum, "{" ~ strip(strippedLine[2..$ - 1]) ~ "} is never defined");
} else if (defLocations[strip(strippedLine[2..$ - 1])] != "noWeave") {
    def = defLocations[strippedLine[2..$ - 1]];
    defLocation = def;
    auto index = def.indexOf(":");
    string chapter = def[0..index];
    htmlFile = getChapterHtmlFile(p.chapters, chapter);
    if (chapter == c.num()) {
        defLocation = def[index + 1..$];
    }
    def = ", <a href=\"" ~ htmlFile ~ "#" ~ def ~ "\">" ~ defLocation ~ "</a>";
}
output ~= "<span class=\"nocode pln\">" ~ leadingWS(line) ~ "{" ~ strippedLine[2..$ - 1] ~ def ~ "}</span>\n";

12. Add links to other sections

Writing the links is pretty similar to figuring out where a codeblock was defined because we have access to the sectionLocations array (which is addLocations, useLocations, or redefLocations). Then we just have a few if statements to figure out the grammar -- where to put the and and whether to have plurals and whatnot.

{LinkLocations function 12}
string linkLocations(string text, string[][string] sectionLocations, Program p, Chapter c, Section s, parser.Block block) {
    if (block.name in sectionLocations) {
        string[] locations = dup(sectionLocations[block.name]);

        if (locations.canFind(c.num() ~ ":" ~ to!string(s.num))) {
            locations = remove(locations, locations.countUntil(c.num() ~ ":" ~ to!string(s.num)));
        }

        if (locations.length > 0) {
            string seealso = "<p class=\"seealso\">" ~ text;

            if (locations.length > 1) {
                seealso ~= "s ";
            } else {
                seealso ~= " ";
            }

            foreach (i; 0 .. locations.length) {
                string loc = locations[i];
                string locName = loc;
                auto index = loc.indexOf(":");
                string chapter = loc[0..index];
                string htmlFile = getChapterHtmlFile(p.chapters, chapter);
                if (chapter == c.num()) {
                    locName = loc[index + 1..$];
                }
                loc = "<a href=\"" ~ htmlFile ~ "#" ~ loc ~ "\">" ~ locName ~ "</a>";
                if (i == 0) {
                    seealso ~= loc;
                } else if (i == locations.length - 1) {
                    seealso ~= " and " ~ loc;
                } else {
                    seealso ~= ", " ~ loc;
                }
            }
            seealso ~= "</p>";
            return seealso;
        }
    }
    return "";
}

Used in section 1

13. See also links

Writing the 'added to' links is pretty similar to figuring out where a codeblock was defined because we have access to the addLocations array. Then we just have a few if statements to figure out the grammar -- where to put the and and whether to have plurals and whatnot.

{Write the 'added to' links 13}
output ~= linkLocations("Added to in section", addLocations, p, c, s, block) ~ "\n";

Used in section 9

14. Also used in links

This is pretty much the same as the 'added to' links except we use the useLocations array.

{Write the 'used in' links 14}
output ~= linkLocations("Used in section", useLocations, p, c, s, block) ~ "\n";

Used in section 9

15. Redefined in links

{Write the 'redefined in' links 15}
output ~= linkLocations("Redefined in section", redefLocations, p, c, s, block) ~ "\n";

Used in section 9

16. Katex source

This is the source code for katex which should only be used if math is used in the literate file. We include a script which uses the cdn first because that will use better fonts, however it needs the user to be connected to the internet. In the case that the user is offline, we include the entire source for katex, but it will use worse fonts (still better than nothing though).

{Write the katex source 16}
output ~= "<link rel=\"stylesheet\" href=\"http://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.3.0/katex.min.css\">\n"
"<script src=\"http://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.3.0/katex.min.js\"></script>\n";
output ~= katex;

Used in section 5

Then we loop over all the math divs and spans and render the katex.

{Write the katex source 16} +=
output ~= q"DELIMITER
<script>
var mathDivs = document.getElementsByClassName("math")
for (var i = 0; i < mathDivs.length; i++) {
    var el = mathDivs[i];
    var texTxt = el.textContent;
    try {
        var displayMode = false;
        if (el.tagName == 'DIV') {
            displayMode = true;
        }
        katex.render(texTxt, el, {displayMode: displayMode});
    }
    catch(err) {
        el.innerHTML = "<span class='err'>"+err+"</span>";
    }
}
</script>
DELIMITER";

Used in section 5

{Imports 16}
import globals;
import std.file;
import std.conv;
import std.algorithm;
import std.regex;
import std.path;
import std.stdio;
import std.string;
import parser;
import util;
import dmarkdown;

Used in section 1