---- title: Beastie – an experimental processor for BibTeX and friends SPDX-FileCopyrightText: 2023 Norman Gray SPDX-License-Identifier: BSD-2-Clause Beastie is an experiment in parsing and processing BibTeX files. Home page: Beastie is: * A parser for `.bib` files, amongst others. * A framework for processing the parsed bibliographic data, written in a dialect of Scheme, extended with suitable parsing and processing primitives. * A implementation of the `.bst` BibTeX Style Language, using that framework. The goals of this project are: * to explore ways of processing `.bib` files which _aren't_ exclusively based on `.bst`; * to provide a second implementation of the BST language; and * to initiate some sort of conversation about BibTeX standards, and their documentation, broadly construed. The first and second of those points aim to get away from BibTeX being a single-implementation system, where the only thing that can reliably process a `.bib` file is the `bibtex` program. This isn't _quite_ the case at present, but combined with the third documentation point, it's quite hard to process BibTeX input in a confident way. The `.bst` language counts as rather _niche_. It has substantial amusement value, but expertise could hardly be said to be widespread (of course, one could complain that [Scheme][] has some niche aspects, but it also has its _numerous_ compensating virtues). The implementation of the `.bst` language is intended to be fairly complete, in the sense that it is nearly a drop-in replacement for the `bibtex` program (as described in the ‘Writing BibTeX styles’ document, ‘btxhak.pdf’, on the [bibtex][] package page). Some of the differences from BibTeX will be bugs, but others are deliberate mild deviations, as discussed below. Beastie also provides a framework for writing alternative bibliography tools. The ‘experimental’ status means that it is not yet fully-featured in that respect. The fact that it implements the `.bst` language confirms its scope, and acts as some indicator of ‘correctness’, and the `examples/` directory in the distribution illustrates how one would go about writing a from-scratch style file, but there isn't (yet) a drop-in Scheme-based alternative to, for example, `plain.bst`. It is a goal of Beastie to go slightly beyond BibTeX, and to make it a little easier to use `.bib` files to generate HTML or other formats, or for the output of beastie to be ingested into other processing pipelines which are _not_ based on (La)TeX. This has informed some of the choices made. Another limitation of \BibTeX, and its ecosystem, is that, although it provides fairly comprehensive support for STEM-style bibliographies, it starts to struggle with citations much outside this area. For humanities or legal bibliographies, the most active support currently comes from [BibLaTeX](https://mirror.ctan.org/macros/latex/contrib/biblatex-contrib/), possibly because the intricacy of the implied support requires a more approachable language than `.bst`. It's also a goal of this project to produce a v1.0 at some point. Features -------- * Programmatic manipulation of BibTeX bibliographies in a high-level language. That language is Scheme, with [notes on the Beastie dialect](scheme.xhtml). And using that... * An implementation of the `.bst` BibTeX style language. * `.bib`, `.bst`, `.aux`. and Markdown parser/converters (and `.json` because why not?). * A few minor extensions to the `.bst` language. * Slightly more informative error handling. * Parsing of `.bib` files promptly expands a selection of `{\"u}`-style escapes into the corresponding characters (this makes it easier to use this tool for generating HTML and other non-LaTeX bibliographies). * Unicode-aware throughout, in terms of case-folding and word-boundary recognition (with some qualifications). Limitations ----------- * There are one or two edge-cases where the `btxhak` document doesn't make it fully clear what counts as ‘correct behaviour’. There are also some places where the implementation deliberately extends or mildly deviates from BibTeX behaviour – see below. * The `.bib` parsing doesn't attempt to round-trip the file contents: inter-entry text is discarded when parsing, and `{\"u}` escapes are expanded promptly. This isn't so much a limitation, as a restriction of scope. * The Markdown support is fairly basic – it's a side-project here. It implements most of the [Gruber spec][gruber], but not all, and in particular it doesn't know anything about HTML content. * Although the project's goal is to make it easier to write bibliography processors, the provision within Beastie is still at the level of adding parsing primitives, and not much at a higher level. That is, the [distributed example scripts][examples] contain much that would probably become ‘library’ code in a mature state. As far as the `.bst` language is concerned, it aims to support it quite fully, in the sense that it will produce the same results as BibTeX, for the standard styles. However it doesn't support it slavishly, and there are some deviations noted below. It also aims to slightly improve `.bst` error reporting, and adds a couple of extra `.bst` builtin functions, in both cases with the aim of being useful for debugging or developing `.bst` files (eg, `show.stack$$` and `printf$$`). Some of these points are discussed more extensively below. Future plans (tentative) ------------------------ Beastie is still a development project (though there is a firm determination to get to a version 1.0 before too long!). Some acknowledged gaps are: * Beastie contains support for Unicode both using the Unicode consortium's [ICU][] library, and also using a local library for parsing and serialising Unicode. The main limitation of the latter is that is has only a very naive collation algorithm. This is unlikely to change, however, since it's mostly regarded as a fallback. The support for locales has not been thoroughly tested; bug reports are welcome. * The documentation is still somewhat ragged. There's quite a lot of it, but it's grown rather organically and could do with being rationalised. Longer-term developments (that is, things which might wait until after a v1.0): * An implicit goal of Beastie is that it provide the sort of foundation upon which one might experiment with implementing [Citation Style Language][csl]. But that’s definitely a longer-term project. That's the sort of project which would use the ‘library code’ mentioned above. * Develop a fuller replacement for (eg) `plain.bst` in Scheme, as a proof-of-concept? Acknowledgements ----------------- Most of this is implemented in an embedded copy of s7, from [ccrma.stanford.edu][s7]. Thanks to Bill Schottstaedt for creating s7, and for advice on some of its intricacies. Thanks to Karl Berry and Paulo Ney for comments, and for functionality suggestions. Usage ===== There is a [manpage](beastie.1.txt). Processing `.aux` files using `.bst` --------------------- If you call beastie with % beastie foo then it will parse the aux file `foo.aux`, looking for `\citation`, `\bibdata` and `\bibstyle` commands, just like BibTeX, and process the bibliography with the indicated `.bst` program. That is, this is (still tentatively) intended to be a drop-in alternative to `bibtex`, though I say this with some diffidence, since it hasn't really been pushed around much. There are some deviations from the authoritative BibTeX behaviour, noted below. Processing using Scheme – programming ----------------------- If you call beastie with % beastie foo.scm [args...] then the program `foo.scm` is interpreted as Scheme, extended with the functions described below. My assertion is that this is a more enjoyable way of generating bibliographies than the standard `.bst` language. That is documented [elsewhere](scheme.xhtml). Any `args...` are available in the program in the global variable `*command-line*`; there is a `getopt` function which may be useful if you are writing a script. If you invoke beastie with the `-r` option, or if you make `repl` and invoke the program `repl`, then you're dropped into a basic REPL. While beastie still has a version number less than 1.0, everything here should be regarded as provisional. Converting BibTeX ----------------- If invoked with the name of a `.bib` file, % beastie foo.bib then the given file will be parsed and re-written as a `.bib` file. If you supply the `-O` option, then you can select an output format; if this format is ` bib`, `json`, or `sexp`, the output will be in the corresponding format. You can do the same thing with `beastie -I bib