Encoding & newlines

In Shift_JIS the character 本 is the two bytes 96 7B, and 7B is ASCII {; the character 表 is 95 5C, and 5C is a backslash. Get the character encoding wrong and Japanese prose turns, byte by byte, into LaTeX control sequences and braces. That is why mojibake in TeX is never merely "unreadable glyphs", and why a .tex file today should be saved as UTF-8 with LF line endings. This page shows what actually happens when you meet the remains of Shift_JIS, EUC-JP or ISO-2022-JP, how to convert them, and how the range of characters differs between platex and uplatex — with the real error messages quoted.

What should a .tex file be saved as? UTF-8 and LF

UTF-8 — with or without a BOM — and LF. On TeX Live 2024, uplatex, lualatex and xelatex all read UTF-8 by default, and Git assumes UTF-8 and LF when it builds a diff. What makes this awkward in Japanese is that before Unicode spread there were three mutually incompatible encodings in simultaneous use: Shift_JIS on Windows, EUC-JP on Unix, and ISO-2022-JP — the "JIS code" — for email, which had to survive seven-bit transport. The same kanji had three different byte sequences, and no amount of staring at a file will tell you which one you have with certainty. That is why encoding is the first thing to suspect when an old lab directory is opened.

What happens when a Shift_JIS file meets a UTF-8 toolchain

The screen does not fill with unreadable glyphs. A cascade of errors appears and the run falls over on ! Undefined control sequence. Feed a Shift_JIS file straight to uplatex on TeX Live 2024 and the first message is ! LaTeX Error: Invalid UTF-8 byte "93., followed by ! LaTeX Error: Invalid UTF-8 byte sequence (^^ea^^82̕). So far so helpful — these are honest reports of illegal bytes. The trouble comes next: the echoed line reads l.3 ^^93^^fa^^96{^^8c^^ea^^82̕\^^8e, and you can see it plainly — the trailing byte of 本 has become a { and the trailing byte of 表 has become a \, and both are now live TeX syntax. The symptom is therefore not "the Japanese looks wrong" but "LaTeX says some command I never wrote is undefined", which is exactly why the encoding is the last thing people suspect.

$ uplatex sjis.tex
! LaTeX Error: Invalid UTF-8 byte "93.
l.3 ^^93
! LaTeX Error: Invalid UTF-8 byte sequence (^^ea^^82̕).
! Undefined control sequence.
l.3 ^^93^^fa^^96{^^8c^^ea^^82̕\^^8e

$ uplatex -kanji=sjis sjis.tex        # tell the engine what it is reading
Output written on sjis.dvi (1 page, 320 bytes).

The accident follows inevitably from how Shift_JIS was designed. Because the second byte of a two-byte character is allowed to fall inside the ASCII range 0x40–0x7E, there are 52 characters whose second byte is exactly 0x5C (a backslash) and 50 whose second byte is 0x7B (an opening brace). The first group includes 表, 十, ソ, 能, 貼, 暴, 申 and 構; the second includes 本, 宮, 施, 旬, 養 and 鶏 — all perfectly ordinary characters in running Japanese. Japanese developers called these the "bad characters" and fought them for years, because the same accident struck shell scripts and configuration files, not only TeX. EUC-JP does not have the problem — its second bytes are always 0x80 or above — which is why EUC-JP was for a while the preferred encoding in TeX circles.

Converting Shift_JIS to UTF-8: iconv and nkf

The classic Japanese tool is nkf (Network Kanji Filter), but it has to be installed — it ships with neither macOS nor most Linux distributions. Reach for iconv first: it is a POSIX standard utility and lives at /usr/bin/iconv on macOS and Linux alike. iconv -f CP932 -t UTF-8 old.tex > new.tex does the Shift_JIS to UTF-8 conversion. If nkf is available, nkf -w -Lu --overwrite *.tex rewrites many files in place in a single line, which is worth installing for when you inherit a whole directory. Whichever you use, take a copy or commit to Git before you run it: --overwrite destroys the original exactly as advertised.

# iconv -- always present; safest one file at a time
iconv -f CP932  -t UTF-8 old.tex > new.tex     # Shift_JIS -> UTF-8
iconv -f EUC-JP -t UTF-8 old.tex > new.tex     # EUC-JP    -> UTF-8

# whole tree, keeping a backup of every original
for f in *.tex; do cp "$f" "$f.bak"; iconv -f CP932 -t UTF-8 "$f.bak" > "$f"; done

# nkf, if installed: detect first, then convert in place to UTF-8 + LF
nkf -g old.tex
nkf -w -Lu --overwrite *.tex

When you use iconv, name the encoding CP932, not SHIFT_JIS. The two are widely assumed to be the same and are not. Ask macOS's iconv to convert text containing ～ (the wave dash) or ① (a circled digit) to SHIFT_JIS and it stops with iconv: iconv(): Illegal byte sequence; ask for CP932 and it succeeds. SHIFT_JIS is the narrow definition faithful to JIS X 0208 and excludes the NEC and IBM extension characters — circled digits, Roman numerals and the rest. Files that came from Windows are CP932 in practice, so CP932 is the right default. The same reasoning applies in reverse, when you must convert UTF-8 back to Shift_JIS for an old tool.

-kanji=: telling the engine how to read, without converting

When you do not want to rewrite the file — or are not allowed to — tell the engine instead. (u)platex accepts -kanji=, with -kanji=sjis, -kanji=euc, -kanji=jis and -kanji=utf8. The Shift_JIS file that produced a wall of errors when read as UTF-8 compiles under uplatex -kanji=sjis sjis.tex with a plain Output written on sjis.dvi (1 page, 320 bytes). But treat this as first aid. Your editor, Git, grep and any file pulled in with \input still assume UTF-8, so the moment encodings mix you get a different accident. Use it to compile a manuscript someone sent you once, just far enough to read it — then convert. Note that lualatex and xelatex have no -kanji= option at all: they always read UTF-8.

platex versus uplatex: the same binary, two different character worlds

The difference is the range of characters, not the program. Run platex --version and uplatex --version on TeX Live 2024 and both announce themselves as e-upTeX 3.141592653-p4.1.1-u1.30-230214-2.6 — the same binary. Only the parenthesis differs: (utf8.euc) for platex, (utf8.uptex) for uplatex. They load different formats, and the format decides how characters are held internally. The consequence is concrete. Put 髙 (U+9AD9, a variant of 高 common in Japanese surnames) in a document and platex stops with ! LaTeX Error: Unicode character ^^e9^^ab^^99 (U+9AD9), while uplatex sets it without a word. Plain platex is closed inside the JIS X 0208 repertoire, and anything outside it is refused at the door. There is no longer a reason to choose platex for a new document. Make uplatex the default and 髙, 𠮟 and the variant forms in a membership list all go through.

$ platex --version | head -1
e-upTeX 3.141592653-p4.1.1-u1.30-230214-2.6 (utf8.euc) (TeX Live 2024)
$ uplatex --version | head -1
e-upTeX 3.141592653-p4.1.1-u1.30-230214-2.6 (utf8.uptex) (TeX Live 2024)

$ platex takashima.tex          # the document contains 髙 (U+9AD9)
! LaTeX Error: Unicode character ^^e9^^ab^^99 (U+9AD9)
$ uplatex takashima.tex
Output written on takashima.dvi (1 page, 304 bytes).

Newlines and the BOM: do LF, CRLF and CR actually break anything?

LaTeX itself accepts all of them without complaint. Hand uplatex a .tex file written with \r\n (CRLF, Windows), or one that begins with a UTF-8 BOM (EF BB BF), and on TeX Live 2024 both uplatex and lualatex compile it without a single warning. The folklore that a BOM leaves an invisible character at the start of the document does not hold for these two engines today. What suffers is not LaTeX but the tools around it. A file with mixed CRLF and LF shows up in Git as every line changed, which makes review impossible. A .sty with stray \r at line ends can defeat an end-of-line anchor in grep. Standardising on LF is therefore a collaboration decision, not a typesetting one. Under Git, one line in .gitattributes is the most reliable fix, and it absorbs the differences between contributors' machines at checkout.

# .gitattributes -- normalise on checkin, hand out LF on checkout
*.tex text eol=lf
*.sty text eol=lf
*.bib text eol=lf
*.pdf binary

# one-off cleanup of a file that arrived with CRLF
sed -i.bak $'s/\r$//' old.tex

• Standardise everything, old and new, on UTF-8 + LF. It is the default of all three TeX Live 2024 engines and it matches what Git expects.
• Start conversions with iconv -f CP932 -t UTF-8. Say CP932, not SHIFT_JIS, or circled digits and the wave dash will fail to convert.
• If nkf is installed, nkf -w -Lu --overwrite *.tex is fastest — but it ships with neither macOS nor most Linux distributions.
• Copy the file or commit to Git before converting. --overwrite cannot be undone.
• Start new documents on uplatex (or lualatex). Plain platex errors on characters outside JIS X 0208 such as 髙 (U+9AD9).
• -kanji=sjis is first aid. Once you can read the document, convert the file itself to UTF-8.