Chinese characters shows as # in generated PDF

[WolframKahl]

Problem description

Just starting on DocBook, via DAPS, and using Unicode in awareness that XML processing is Unicode-based, expecting Unicode characters in documents to work “out of the box” given the general appearence of DAPS, I was quite disappointed to see
Chinese characters (which I need) rendered as # in PDF. (I have ample fonts installed (on Ubuntu), and have Chinese working in XeLaTeX.)

Expected behavior

Chinese characters should display properly in PDF.

If this is beyond the scope of DAPS, instructions and/or warnings should be included prominently in the DAPS documentation.

Steps to reproduce problem

daps-init -d TestDoc
sed -e 's/First item/序号/' TestDoc/xml/MAIN-daps-example.xml > TestDoc/xml/MAIN-daps-example.xml_tmp
mv TestDoc/xml/MAIN-daps-example.xml_tmp TestDoc/xml/MAIN-daps-example.xml
daps -d TestDoc/DC-daps-example pdf
mupdf TestDoc/build/daps-example/daps-example_color_en.pdf

Notice the ## in the “Example ordered list”.

[ghost]

Browsers (and most OS's) can use a fallback font for individual characters: You define a custom font and if one of the characters you need is not part of that font, the browser/OS will just pick that specific character out of a fallback font.

Unlike browsers, FOP (which DAPS currently uses for generating PDFs), cannot pick individual characters out of another font automatically. It only ever uses one font at a time. This means if you need Chinese output, you must define a font that supports Chinese characters up front and that font must be the first one available for FOP to pick (in practical terms, the Chinese font should be the first font listed in your font stack).

All of this makes this a stylesheet question. In the SUSE stylesheets we solved the issue by switching out the FO font stack depending on the xml:lang attribute of the current document (cf. https://github.com/openSUSE/suse-xsl/blob/main/suse2013/fo/l10n.properties.xml) -- However, this not a behavior found in the upstream DocBook stylesheets.

[WolframKahl]

A new DAPS user may not even know what XSL is, and even after a day of trying to hunt down where the "Times-Roman" displayed in

 [WARN] FOUserAgent - Glyph "序" (0x5e8f) not available in font "Times-Roman".

(which I get from daps --verbosity=3 pdf) is coming from, I found neither where it is coming from, nor a way to override it.

It only ever uses one font at a time.

I found font-selection-strategy=character-by-character mentioned somewhere --- is

  --param="font-selection-strategy=character-by-character"

the right way to communicate that to DAPS?

Is there a difference between --param and --stringparam? Section 9.1 of the DAPS manual https://opensuse.github.io/daps/doc/cha.daps.user.layout.html#sec.daps.user.layout.xsltparams appears to suggest that there is no difference...

This means if you need Chinese output, you must define a font that supports Chinese characters up front and that font must be the first one available for FOP to pick (in practical terms, it should be the first font listed in your font stack).

How do I do that?

[ghost]

A new DAPS user may not even know what XSL is, and even after a day of trying to hunt down where the "Times-Roman" displayed in

The stylesheets are a bit of an area for experts. You can do a lot with just parameters but bigger layout changes do need XSLT knowledge. As for how to override this value see at the bottom.

Usually, the DocBook XSLT stylesheets are installed into /usr/share/xml/docbook/stylesheet/nwalsh which is where fonts should be defined. In this case, the font is defined as just "serif", leaving FOP to make a choice, and apparently it picks "Times-Roman" as the default.

I found font-selection-strategy=character-by-character mentioned somewhere --- is

  --param="font-selection-strategy=character-by-character"

the right way to communicate that to DAPS?

Excellent pointer & I didn't know it yet, but:

1. No, this is a property that would need to be set in XSL-FO as an attribute.
   What the --param you suggested does. is to set an XSLT parameter, which is a totally different thing.
2. The DocBook stylesheets already include this parameter by default.
3. The real issue again is that FOP does not support it, even in its newest version: https://xmlgraphics.apache.org/fop/compliance.html#fo-property-font-selection-strategy

The way DAPS creates PDFs is as follows:

1. DAPS takes the DocBook file and generates an XSL-FO file (a layout-oriented XML file format) using xsltproc (which applies an XSLT stylesheet).
2. DAPS takes the XSL-FO file and generates PDF from it using FOP.

This property needs to be generated into the XSL-FO file using the XSLT stylesheet (again, already done by default).
The behavior of a stylesheet can be modified by using --param options.

Is there a difference between --param and --stringparam? Section 9.1 of the DAPS manual https://opensuse.github.io/daps/doc/cha.daps.user.layout.html#sec.daps.user.layout.xsltparams appears to suggest that there is no difference...

• At the DAPS level, DAPS 3.1 will handle --param and --stringparam exactly the same, all will be converted to --stringparam. (This is not yet the case for DAPS 3.0 which still handles them separately like xsltproc.)
• Deeper down in xsltproc, using --param only supports numbers as values and may enable a few memory optimizations. But you can just always use --stringparam and everything will work correctly.

This means if you need Chinese output, you must define a font that supports Chinese characters up front and that font must be the first one available for FOP to pick (in practical terms, it should be the first font listed in your font stack).

How do I do that?

Depends on the stylesheets. If you're using the upstream DocBook stylesheets, you can define font choices in the DC file. The following should give you what you need for Simplified Chinese:

MAIN=mymain.xml
XSLTPARAM="--stringparam body.font.family='AR PL UMing TW' --stringparam title.font.family='AR PL UMing TW' --stringparam monospace.font.family='WenQuanYi Micro Hei Mono'"

---

I hope that helps/makes sense.

[WolframKahl]

Usually, the DocBook XSLT stylesheets are installed into /usr/share/xml/docbook/stylesheet/nwalsh which is where fonts should be defined. In this case, the font is defined as just "serif", leaving FOP to make a choice, and apparently it picks "Times-Roman" as the default.

Found it:

 $ grep '<xsl:param name=.*font\.family' $(find  /usr/share/xml/docbook/stylesheet/nwalsh/ -type f)
/usr/share/xml/docbook/stylesheet/nwalsh/slides/fo/param.xsl:<xsl:param name="slide.font.family">Helvetica</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/slides/fo/param.xsl:<xsl:param name="slide.title.font.family">Helvetica</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="body.font.family">serif</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="dingbat.font.family">serif</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="monospace.font.family">monospace</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="sans.font.family">sans-serif</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="symbol.font.family">Symbol,ZapfDingbats</xsl:param>
/usr/share/xml/docbook/stylesheet/nwalsh/fo/param.xsl:<xsl:param name="title.font.family">sans-serif</xsl:param>

XSLTPARAM="--stringparam body.font.family='AR PL UMing TW' --stringparam title.font.family='AR PL UMing TW' --stringparam monospace.font.family='WenQuanYi Micro Hei Mono'"

That leads to fonts not found...

At https://lists.oasis-open.org/archives/docbook-apps/201007/msg00047.html , I found:

If you do want to load OS fonts such as "Arial Unicode MS", in addition to specifying the font name in the FO, you need to create a FOP configuration file and add the <auto-detect /> instruction, as documented here: http://xmlgraphics.apache.org/fop/0.95/fonts.html#basics

On my system, /usr/share/doc/fop/fop.xconf contains <auto-detect/> and claims:

This is an example configuration file for FOP.
This file contains the same settings as the default values
and will have no effect if used unchanged.

However, if I execute

CLASSPATH="/usr/share/java/fop.jar:/usr/share/java/batik-libs.jar:/usr/share/java/avalon-framework.jar:/usr/share/java/xmlgraphics-commons.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-io.jar" java org.apache.fop.tools.fontlist.FontListMain

, I get only Courier, Helvetica, Times, ZapfDingbats, while if I add -c /usr/share/doc/fop/fop.xconf at the end of that command, I get a more plausible-looking very long list of fonts (including the Chinese fonts you mentioned).

Does DAPS override the default <auto-detect/> or is /usr/share/doc/fop/fop.xconf lying about corresponding to the defaults?

Either way, how do I tell DAPS to call FOP with a particular fop.xconf, or how do I otherwise inject <auto-detect/> into the way that DAPS calls FOP?

The output of --verbose=3 mentions /usr/share/daps/libexec/daps-fop, and calling that with --help gives:

  -c cfg.xml        use additional configuration file cfg.xml

Manually repeating the daps-fop call from the log with -c /usr/share/doc/fop/fop.xconf inserted actually finds the fonts and produces Chinese characters in the PDF!

So how can I inject -c /usr/share/doc/fop/fop.xconf into that daps-fop call via proper DAPS usage?

[ghost]

Usually, the DocBook XSLT stylesheets are installed into /usr/share/xml/docbook/stylesheet/nwalsh which is where fonts should be defined. In this case, the font is defined as just "serif", leaving FOP to make a choice, and apparently it picks "Times-Roman" as the default.

Found it:

Good! The parameter names you find there can be changed from your DC file as described below.

XSLTPARAM="--stringparam body.font.family='AR PL UMing TW' --stringparam title.font.family='AR PL UMing TW' --stringparam monospace.font.family='WenQuanYi Micro Hei Mono'"

That leads to fonts not found...

The fonts need to be installed. I picked open-source fonts, so there definitely should be a system package for them.

Does DAPS override the default <auto-detect/> or is /usr/share/doc/fop/fop.xconf lying about corresponding to the defaults?

DAPS ships with/uses its own FOP configuration file: https://github.com/openSUSE/daps/blob/main/etc/fop/fop-daps.generic

I wonder if it defines the wrong directory to search fonts in there? The ~same file is used on openSUSE and works fine there.
If the directory set up in that file needs updating for modern Ubuntu systems, please tell us.

Either way, how do I tell DAPS to call FOP with a particular fop.xconf, or how do I otherwise inject <auto-detect/> into the way that DAPS calls FOP?

I guess by putting the line FOP_CONFIG_FILE='/path/to/your/custom/config' in either ~/.config/daps/dapsrc or into individual DC files. (Haven't tried, but /etc/daps/config includes this parameter, so it should be overridable.)

Manually repeating the daps-fop call from the log with -c /usr/share/doc/fop/fop.xconf inserted actually finds the fonts and produces Chinese characters in the PDF!

Woo!