----
title: UNICODE module – basic Unicode support
SPDX-FileCopyrightText: 2025 Norman Gray <https://nxg.me.uk>
SPDX-License-Identifier: BSD-2-Clause


Beastie does not have complicated Unicode-related requirements, but it
does need to be able to read and write Unicode files, and to do
language-sensitive sorting and case-switching.

### Character classes

The character class functions, in the `utils` module, are _partially_
Unicode aware, in the sense that they will report the classes of
characters in the BMP.  These functions are `char-alnum?`, `char-alpha?`,
`char-upper?`, `char-lower?`, `char-other-letter?`, `char-digit?`,
`char-space?`, `char-punct?`, `char-cntrl?`, `char-symbol?` and
`char-mark?`.  There are also case-changing functions
`uchar-upcase`, `uchar-downcase`, `uchar-titlecase`, and
classification functions such as `uchar-alphabetic?`.
See the documentation in the [utils
module](utils.xhtml) for details of their behaviour, and see the
[Unicode Character Database][ucd] (and its [file
collection][ucd-files]) for detailed discussion of the properties in question.


Ustrings
--------

Various functions within beastie produce and handle ‘ustrings’, which
should generally be equivalent to normal strings in use, but which
have various other unicode-related operations defined.  These
principally consist of the unicode-related functions described just below.

Construct an empty ustring with `(make-ustring)`, or add content with
`(make-ustring codepoint/string/ustring ...)`.  The various
`ustring-*` functions below are intended to broadly match the
corresponding ordinary string ones.  Since `ustring-append` creates a
new ustring, that is an alternative way of creating a ustring from scratch.

Ustrings have write-form `#"abç"`, and the reader will parse this back
into a ustring.  Within the quotes, the _only_
escapes recognised are `\\`, `\"`, and `\n`, with the last denoting a
newline.  Any other escaped characters are ignored, with a warning.

A ustring is an ‘applicable object’ in s7 terms: if `us` is a ustring,
then `(us 2)` is the same as `(ustring-ref us 2)`, evaluating to the
codepoint of `us` at index 2.  Similarly `map` or `for-each` applied
to a ustring will call the mapped function for each codepoint in the string.


Reading and writing
-------------------

A `unicode-reader?` is an object which turns a string or file into a
sequence of Unicode codepoints.  Associated functions are

  * unicode-reader?
  * make-unicode-reader/file
  * make-unicode-reader/string
  * unicode-reader-read
  * unicode-reader-source
  * unicode-reader-location

A `unicode-reader?` object is a `lexeme-source?` in the sense of the
[klipspringer](klipspringer.xhtml) module.

It may occasionally be useful to note that two `unicode-reader?`
objects are deemed to be `equivalent?` if they refer to the same
underlying object, string or file; they are `equal?` only if they are
additionally at the same offset into the string or file.

Separately, we can encode lists of codepoints to UTF-8, and decode a
UTF-8 string.

  * unicode-decode/utf8
  * unicode-encode/utf8
  * unicode-encode1/utf8

Unicode support in beastie
--------------------------

Beastie aims to be Unicode-aware by design rather than by accident.
That is, input is parsed and handled as Unicode codepoints, rather
than being bundled into UTF-8 strings and hoping for the best.

This means, for example, that the rules for allowed strings in
`.bib` key and field names are expressed in terms of Unicode
alphabetic characters (thus `नाम` is a valid key or field name, since
it's a list of letters, but `a¶` isn't, since it includes a
punctuation character).  This obviously isn't a key beastie use-case,
but it serves as a check on the ‘unicodeness’ of the code.

The conformance to Unicode is reasonably extensive, but not complete.
There are two cases here.

If Beastie is built using the [ICU][] library, then it has (some)
locale support, and in particular it has language-sensitive sorting.

  * unicode-get-locale
  * unicode-get-locales
  * unicode-set-locale!

If Beastie is build _without_ the ICU library, either because it is
not available or because it was suppressed at build time, then there
is still Unicode support, but less sophisticated.

Further notes, applicable to the with- and without-ICU cases:

  1. We properly support only the [Basic Multilingual
Plane](https://en.wikipedia.org/wiki/Plane_(Unicode%29#Basic_Multilingual_Plane)
(BMP) – characters outside of that, in the ‘astral planes’, are
perfectly welcome in strings, but are not regarded as being included
in any character class (so yes, you can happily have emojis in field
_values_, but no, you may _not_ have them in database keys).

  2. There are various edge-cases, such as around title casing,
  that the code avoids (yes, the eszett!).

  3. It's doubtless possible to confuse the
string-classification algorithm by using one or other Unicode normal
forms.

[ucd]: https://www.unicode.org/reports/tr44/
[ucd-files]: https://www.unicode.org/Public/UCD/latest/ucd/
[ICU]: https://icu.unicode.org