Index & bib commands

The programs that build indexes and bibliographies — makeindex, xindy, bibtex, biber — are not LaTeX macros but separate executables. Each therefore decides for itself how arguments are written, what exit code it returns and where its log goes. And here is the fact that quietly breaks continuous integration: makeindex returns exit code 0 even when it has thrown entries away, and bibtex returns 0 even when it warns that a cited work is missing. The failure is not recorded in the build status but in the .ilg and .blg files. This page looks at all four from the command side rather than the LaTeX side; writing index entries and designing a bibliography database are covered elsewhere.

bibtex doc versus makeindex doc.idx: which one takes the extension

There is nothing to derive here; you simply learn it. The bibliography programs take no extension, the index programs do. bibtex and biber receive the document’s job name and open .aux or .bcf themselves. Get that wrong and the message is surprisingly unhelpful: bibtex doc.tex answers I couldn t open file name doc.tex.aux and exits 1, while biber doc.tex answers ERROR - Cannot find 'doc.tex.bcf'! Both have merely appended an extension to the name you gave; neither will tell you that the .tex was the mistake. makeindex, upmendex and texindy, by contrast, take the input file itself, so you write doc.idx. Use -o to change the output name and -s to name a style.

bibtex   doc          # job name, no extension  -> reads doc.aux, writes doc.bbl
biber    doc          # job name, no extension  -> reads doc.bcf, writes doc.bbl
makeindex doc.idx     # the file itself         -> writes doc.ind and doc.ilg
upmendex -o doc.ind doc.idx
texindy  -C utf8 -L german-din -o doc.ind doc.idx

The exit codes, and where CI misses a failure

The table below records values actually observed on this machine under TeX Live 2024. What it shows is that the line between a warning and a failure is drawn differently by each program. bibtex returns 2 only when it has printed error messages; a citation key missing from the .bib counts as a warning, so the status is 0. makeindex returns 1 when the input file is absent, but stays at 0 however many entries it throws away inside it. Watch only the status of latexmk or of your CI job, therefore, and you can get a green tick on a build where entries vanished from the index and a reference came out blank. For indexes and bibliographies, the correct defence is to inspect the .ilg and .blg files rather than the exit code.

Reading makeindex’s .ilg: this is where the discarded entries went

Every run of makeindex writes both a .ind — the index that gets typeset — and a .ilg, a record of the work. A healthy run is terse: Scanning input file doc.idx....done (6 entries accepted, 0 rejected). then Sorting entries....done (19 comparisons). then Generating output file doc.ind....done (20 lines written, 0 warnings). It is barely an exaggeration to say the only things worth reading are the numbers in parentheses. Feed it a damaged .idx and the accepted count drops while the reasons line up: !! Input index error (file = bad.idx, line = 4): followed by -- Incomplete first argument (premature LFD). The exit code is still 0. Checking that the accepted count matches the number of \index commands you wrote heads off most accidents on its own. The version in TeX Live 2024, incidentally, is makeindex 2.17, which introduces itself as (kpathsea + Thai support).

makeindex doc.idx
# This is makeindex, version 2.17 [TeX Live 2024] (kpathsea + Thai support).
# Scanning input file doc.idx....done (6 entries accepted, 0 rejected).
# Sorting entries....done (19 comparisons).
# Generating output file doc.ind....done (20 lines written, 0 warnings).

grep -c "^\\\\indexentry" doc.idx   # compare this with "entries accepted"
grep "rejected"          doc.ilg   # the number CI should be watching

This unassuming program has a surprising pedigree. It was written by Pehong Chen, but the acknowledgments in its man page record that “Leslie Lamport contributed significantly to the design of MakeIndex.” The author of LaTeX was closely involved in designing the index program — which is why the syntax of \index feels continuous with the rest of LaTeX rather than bolted on. The use of @ in \index{key@printed}, and the business of giving accented words a sort key, are covered in detail on the page about indexes themselves.

Choosing an index program: the same four words, sorted by all three

There is only one criterion: does the language you are indexing need collation? Put the four words Zeta, Ähre, Apfel and Öl into an .idx unchanged and hand them to all three programs, and the difference is immediate. makeindex produced Apfel, Zeta, Ähre, Öl — Ä and Ö have UTF-8 byte values greater than Z, so they fall past the end of the alphabet. texindy -C utf8 -L german-din and upmendex both produced Ähre, Apfel, Öl, Zeta, treating Ä as A and Ö as O exactly as the German DIN rule prescribes. xindy (by Joachim Schrod, release 2.5.1) reaches that answer through language modules; upmendex (version 1.08) reaches it through the collation algorithm of ICU 74.2.

Two practical cautions. First, the .ind that texindy writes is structured differently from the makeindex one: it uses \lettergroup to head each alphabetical group and writes the definitions itself with \providecommand. If your document redefines those, they will collide, so look at the output once when you switch. Second, xindy runs on top of Common Lisp — CLISP 2.49.93 in this release — which makes startup heavy and can be noticeably slow on a large index. When Japanese is in the mix, upmendex is both faster and less fussy.

The strange census at the end of a bibtex .blg

Open doc.blg after running bibtex doc and, past the warnings, an unfamiliar table appears: if$ -- 47, while$ -- 2, swap$ -- 1, substring$ -- 6, and on it goes. That is a count of how many times each instruction of BibTeX’s internal stack machine was executed — 237 calls in all on this run. The reason such a thing exists is that a .bst style file is not a configuration file at all but a program for that virtual machine. Near the top of the same .blg sits Capacity: max_strings=200000, hash_size=200000, hash_prime=170003, numbers preserved from the memory budgets of the 1980s; they are the ceiling behind the cry that begins Sorry---you ve exceeded BibTeX s on a very large bibliography. The author is Oren Patashnik of Stanford, and the version in TeX Live 2024 is BibTeX 0.99d.

BibTeX has eight-bit and Unicode-aware descendants, and TeX Live 2024 ships them all. bibtex8 calls itself “8-bit Big BibTeX version 0.99d-x4.02”; bibtexu is the “UTF-8 Big BibTeX” and is built with ICU 74.2. For Japanese there are pbibtex (pTeX family) and upbibtex (upTeX family, announcing itself as upBibTeX 0.99d-j0.36-u1.30 (utf8.uptex)). All of them are the same 0.99d lineage underneath; only the handling of characters and collation differs. Two errors dominate in practice. A malformed .bib produces Illegal end of database file---line 14 of file broken.bib and I m skipping whatever remains of this entry, with exit code 2; a \bibliography pointing at a file that does not exist produces I found no database files---while reading file doc.aux.

biber: a log that labels every line INFO, WARN or ERROR

biber is a newer design written in Perl, and even the shape of its log is different. It opens with INFO - This is Biber 2.19 and then reports, one labelled line at a time, the .bcf it read, how many citation keys it found, the locale it applied and the .bbl it wrote — every line prefixed INFO -. When something is wrong the label changes, as in WARN - I didn t find a database entry for 'missingkey' (section 0), and a tally follows at the end: INFO - WARNINGS: 1. That machine-readability is the biggest practical difference from bibtex: a CI check can be written by counting WARN - and ERROR - with grep. The common accident is a mistyped input name; biber doc.tex prints ERROR - Cannot find 'doc.tex.bcf'! and exits 2. Note also that biber and bibtex both write a file with the extension .blg, so if you have tried both, read the first line to be sure which one you are looking at.

biber doc
# INFO - This is Biber 2.19
# INFO - Found 2 citekeys in bib section 0
# INFO - Output to doc.bbl
# WARN - I didn t find a database entry for 'missingkey' (section 0)
# INFO - WARNINGS: 1

# a CI check that the exit code will not give you
grep -c "^WARN -\|^ERROR -" doc.blg
grep "rejected" doc.ilg

The order to run them in, and who counts the passes for you

The order is: typeset, then index and bibliography, then typeset, then typeset again. The first pass has LaTeX write .idx and .aux — or .bcf under biblatex; you then run these programs over them to produce .ind and .bbl; you typeset again to pull those in; and if numbering has shifted, once more. The awkward part is that the number of passes is not fixed, which is precisely why build tools such as latexmk exist. Running the commands by hand is worth it only when you need to isolate which stage failed — and even that has a fixed order. First check whether .idx or .bcf was produced at all: if not, the problem is on the LaTeX side. Then read .ilg and .blg: if they exist, the problem is in the program. Finally typeset once more and see whether .ind and .bbl really reached the body. Those three steps narrow the cause to almost a single point.

# the classic Japanese sequence, written out
uplatex   paper          # writes paper.aux and paper.idx
upbibtex  paper          # reads paper.aux -> paper.bbl
upmendex  paper.idx      # reads paper.idx -> paper.ind
uplatex   paper          # pulls both in
uplatex   paper          # settles the numbering

# and the same thing delegated
latexmk paper.tex