list of delimiters
tokenizer.cls line 187 "self~delims = array[2]"
tokenizer.cls line 206 "if self~delims~hasIndex(lah)"
tokenizer.cls line 214 "tkl~tokens~queue(.token~new(lah, self~delims[lah], index))"

list of lookahead lengths, descending
tokenizer.cls line 186 "self~lookaheads = array[1]"
tokenizer.cls line 204 "do i over self~lookaheads"

process a list of arrays of delimiter/space pairs
into a list of delimiters sorted into descending
order of length, and a directory of delimiters
indexed from delimiter to name
tokelist - list of two-element arrays
two element array of
sorted queue of lookaheads
directory from delimiter to name
documents.cls line 1775 "array = .tokenizer~setup(ken)"

initialize tokenizer
tokelist - list of two-element arrays

tokenize a string
line - string to tokenize
tokenlist
documents.cls line 588 "self~tokenized = tokenizer~toke(self~flatclause~flat)"