shell: hypertext processor
posted
modified
that name might ring a few alarm bells to anyone with a sense of morality. alas, necessity is the mother of invention, and I have willed this into existence
in normal speak, I have created probably the most cursed static site generator ever, by combining the inlining of php with the language of shell. I have spent too much time to refine this to a state where it's basically frictionless for me to use that I have looped back to being proud of it (I think this is called stockholm syndrome?)
but why
this site you're reading right now is hosted on unix.dog, a pubnix that doesn't really want users hosting dynamic pages. when I got a unix.dog account, there was no templating software such as php, hugo, etc. due to this, I was making most of my pages manually to start off, which I knew very well wasn't going to scale. php did actually get added to the pubnix at some point between then and my "first draft" for something like this, but I don't remember when. Either way, I thought there wasn't any templating software still. I had a few options options: create my own using resources already on the system, ask citali to add something new, or find/compile one
I chose to make my own. I've already had a bunch of experience writing shell, so I wanted to take a stab at another "big" thing in it, much like stuff akin to flacconv
the first draft fucking sucked in retrospect. how it'd work was that you'd have to make a single line comment following the form of <!--RES f=file;VAR=val;VAR2=val2 --> etc. to bring in a template in the BUILD_RESOURCES folder, replacing any strings matching var-VAR with val, var-VAR2 with val2, etc.
certainly, it was slow as shit as well. I can't remember the exact specifics, but it'd take almost 2 seconds to go through my (very small at the time, mind you) site on my hardware, let alone how long it took on unix.dog's hardware (I want to say around 5-6 seconds?). all those grep calls weren't really helping. clearly, I needed something better
again, I still thought no templating software was available, so I set out to make a different way to update my site with a shell script
and that's how I landed on shp
the process
from the get-go, I had the idea of making a php-like templating method but with shell. the initial draft had the bright idea of using the html comments <!--sh and --> to represent the blocks of shell that would be run. when it came to detecting the boundaries and substituting the output of shell commands, I used awk and temporary files to perform the actual replacement, as well as grep to see if I still needed to replace stuff
of course, this was slow as shit, as well as unreliable. It'd take over a second to build the site on my machine--which keep in mind, still wasn't using markdown for anything. this was optimized to just about a second by doing a single run of grep for each file, storing the counted lines, then iterating over that. even still, the unreliability persisted, because in the event that the shell block outputted <!--sh <stuff> -->, the awk would fuck up and flag it. of course, any reasonable person wouldn't do this, but reasonable people would put --> for arrows in their webpage, and managing that proved to be a much harder task than one would think in awk. another refactor had to be done
by this point, I dropped the idea of using normal html comments and went full-on using <?shp <shell> ?> syntax (well, <?sh here, but that'll change in the next commit). this would actually have made the current implementation handle --> properly and generally be nicer to parse, but by now, instead of using the super slow method of grep + awk for every single file, I made a while read -r loop for the processing function, which fed into a case glob parser for each line, setting a flag depending on if the line contained <?shp or not. if the flag was null, then echo the line. if it was just set, echo everything before the <?shp and set it. if it is active (including just set), eval the line. if the line contains ?>, then print everything after it and unset the shell flag. this significantly sped up the process, given how it wasn't iterating over the entire file multiple times--only once now--as well as not rely on external calls for each file, which induces quite a bit of overhead!
by this point, it was looking pretty good! my site was building much faster now--about 0.3 seconds on my machine. for this being in shell, that's pretty good! this wasn't without its own set of issues, though. for one, calling eval separately for each line was both slow and worked weird. if I needed multiline commands, such as a here-document or simply wrapping a long command across multiple lines, it'd break. the major issue though is that I didn't have a proper way of "including" other files. how I worked around this was by directly calling other shell scripts, where the global working directory would always be the include directory. this was a bit clunky, and something that nagged at me. sure, it worked, but it wasn't really "including", and I was moreso just wrangling shell scripts together without a proper "include" function. as such, I went back to the drawing board
the major thing I noticed with attempting to make a proper include function was shell variables. if you didn't know, variables in shell are always global unless set in a subshell/subprocess. this is actually why I made the variables that are used for the build script all caps and prefixed with two underscores--something to clearly indicate that these variables are only meant to be used by the build script, and you probably shouldn't be using variables that follow the same convention. anyway, I realized that making an include function that worked like php, where variables persist, would mean processing that file in the same shell environment. however, doing so would override crucial variables such as __LINE and __SH_FLAG, causing who knows what to break.
realizing this made me come to the most cursed part of the next iteration of the script. I wrapped the entire file processing part in an eval, single-quoted as to not double-evaluate any parts. whenever it came to the important variables like __LINE, __SH_FLAG, and __LINE_EXEC (a variable I made to work around the "eval every line" issue), I would go out of the single quotes briefly to add a number to the end, defined by the variable __SHLVL. this variable is incremented by 1 each time __process_shp is run, then decremented when the function ends. this causes includes to use a different variable for their own contexts without influencing the parent's special variables, while keeping the benefit of making included variables work like php.
by now, it was basically perfect. implementing markdown support was very simple--passing the markdown file through lowdown and sending that to the stdin of a special script in the include directory, _markdown-template.sh. as I was writing this, I changed this to be a proper shp file by making __process_shp() take an argument rather than consuming stdin for itself. personally, I never needed like, a bunch of html outside of include for markdown--just using shell's echo with commands was enough for me--but oh well! at least it's consistent now, and it doesn't incur a big speed detriment
an update from the future
this post has become a bit of a "living post", so to say. I thought I was finished with the script by now--by all means I went 4 ish months without touching it to a high degree, but recently I did a big refactor again, and now I want to document it. for reference, this was the state of the script right before the next big refactor. there's a little bit of change between it and the state when I wrote this blog post, but I'll only focus on one of them in particular. this part's about the nitty gritty
to start, I realized very quickly that I wanted an RSS feed for my blog. additionally, there was a lot of duplicated html between stuff like credits and blog posts. as such, I needed metadata for each markdown file on my site. my initial idea for this was leaving the markdown files as-is and creating a new file, mdmeta, inside the same directory. this actually worked quite well, and I was pleasantly surprised with the result… but it was very fickly. filenames had to match and the friction between "writing" and "modifying metadata" was way too high. I got tired after a few months, and set out for something better
initially, I wanted to keep markdown files pure and unencumbered, left for the text alone with metadata delegated elsewhere. however, after realizing that mdmeta wasn't doing the trick, I realized that I basically had to put it in the markdown files themselves. as such, I abused the idea of stdin to require only one pass through each file, changing set_md_metadata to work based on a header like so:
--- # any amount of - in a row
variable = value
var = foo
--- # any amount of - in a row
then sending the rest of the stdin to the markdown template. works surprisingly well for what is is
it was at this point that I realized the sheer magnitude of the overhead for simply calling external programs in some situations. when I was looking over the script some, the way I always had __process_shp nagged me. it's such an eyesore. surely there has to be a better way? I revisited my good friend awk and whipped up this script:
eval "$(awk -v RS='<\\?shp|\\?>' \
'{n=!n; gsub(/'\''/,"'\''\\'\'''\''")}
!n{print "eval '\''" $0 "'\''"}
n{print "printf %s '\''" $0 "'\''"}' "${1:--}"
)"
please mind the massive amount of escaping quotes. essentially, what the awk does is have two states, represented by n. for each record--where the record separator is <?shp or ?>--it first replaces all single quotes ' with '\'', then either prints printf %s '<text>' or eval '<text>'. the quote replacement was to make sure all quotes are escaped correctly. then, since that's wrapped in one big eval, it either prints data or executes shell. this is quite clever! I felt good about making this! but it ran slower. about 3x slower, in fact. ugh. I even tried experimenting with using only shell but having it work with the entire block of data rather than a line of a time, but this went into issues with making sure stuff was truncated right, which even still, made it slower than the while read loop. what I have in the repo right now is the fastest method I have found, which is crazy for me to think about
anyway, after realizing that simply calling these external tools incurs overhead, I turned my head towards another bottleneck--escape. it was just a simple 5-statement sed call for escaping html, but because it was an external program, calling it incurred a surprising amount of overhead! I refactored it specifically for the purposes of the build script… and it was abysmally slow at building my RSS feed content field. I realized then that the pure-shell solution is faster for small input, but slower for large input. as such, I split the functions! escape and escapepipe. the former only accepts input via its argument, which is suitable for small inputs, while the latter only accepts input through, well, a pipe, which is usually the case for larger inputs (notably content for rss feeds)
the site was building quickly. 0.35 seconds on my machine! quite impressive for what it's worth. but of course, there's almost always possibility for improvement. in all my optimizations for single-threaded execution, I sorely forgot the most powerful optimization of them all: multithreading! well, less of an optimization and more of "squeezing as much as possible out of the shell". wanna know how to multithread natively in shell--no special parallel program or anything? here you go:
# make a parallel fifo/file descriptor we need
__fifo_para="$(mktemp -u -t "shpbuild.XXXXXX")"
mkfifo "$__fifo_para"
exec 9<>"$__fifo_para"
rm -f "$__fifo_para"
while [ "$__THREADS" -gt 0 ]; do
printf "\n" >&9 # start with THREADS amount of lines in fd 9 for later
__THREADS="$((__THREADS - 1))"
done
# read each line from fd 9, launch new program for each line
# print a line after program finished such that another one can take its place
__run_in_parallel() {
read -r __ <&9
{
"$@"
printf '\n' >&9
} &
}
a clever hack for sure, and one that came in quite handy. I set __THREADS to nproc, which seems to be a good balance for speed both on my machine and unix.dog. setting it higher has a chance of not being able to fork one of the files, which, well, isn't good
conclusion
with that, we have hit the end of the road. I don't know of any more optimizations to be had here. I've pulled out all the tricks I can--little external calls, avoiding big data wrangling with shell, multithreading for crying out loud, and now I believe I have spent it all. the results have proved fruitful:
# on my machine
real 0m0.065s
user 0m0.026s
sys 0m0.037s
# on unix.dog:
real 0m0.302s
user 0m0.065s
sys 0m0.051s
pretty damn impressive for shell. the bottlenecks now lie mostly on the files rather than the shell, which means I can finally rest easy. I'm not touching this anymore--it's perfect. an absolutely insane journey through squeezing as much as I can out of the shell for the sake of this site. at least I can move on to writing now
where to find it
the script is bundled with my website repo. if you want a good idea of using it effectively, poke around my site some--and take a look at those feeds especially. in any case, it's a shell script. I don't care enough to give it the whole 9 yards with a separate repo and license and etc. I don't think anyone other than me is even crazy enough to use it. be reasonable