Cleartype with Japanese fonts, will it finally work in Avalon Text? | The Coffeehouse

MarkPerris

Hello All,

I am a user of a Japanese edition of Windows XP (I live in Japan), and as those others who have used Japanese XP will probrobly have noticed, cleartype is almost non-functional in these builds.

The Japanese font in XP, (By Xerox Corp), is a huge collection containing not only all the kanji, japanese scripts, extended Shift-JIS and EUC character sets, but also the western roman text fonts too. Since, as i mentioned above, cleartype fails on this font containing the entire western & asian sets, as does any other form of anti-aliasing, it essentially means that any font rendering in Japanese XP is horribly aliased.

This frustrated me for a very long time, being the owner of an LCD laptop. So, i decided to make my own version of the Kanji fonts (MsGothic and MSMincho) that would have cleartype enabled, after following the instructions from a Japanese friend and a bit of intuition.

For those of you wishing to do this, here are the steps from the top of my head, you will need to download the command line font handling utilities from MS's Typography site for it to work.

1. First, extract the font from the windows\fonts folder.
2. Split the TTC collection into its TTF files.
3. For each TTF file, strip out the bitmap font data which is used for rendering at the most common sizes.

This is the key to getting Cleartype working, it forces the system to render as glyphs right down to the smallest size

4. For each TTF file, edit the headers to change the name, subtype, and postscript definition for each header entry (Windows, Mac, and again in Japanese). This gives it a new name and ID so it can be installed alongside the original. You can do this with a batch file as its a long task.

5. Rebuild the TTF's back into the original TTC collection, and install it.

I cannot provide the edited font as it is copyright, but this procedure should be perfectly legal for windows owners who already have the bits needed.

Here are the results, both before and after, (Captured on an English Language XP with the new Kanji font installed)

Cleartype On - Edited Font http://www.geocities.com/darkangelxx2001/ClearTypeOn.png

Cleartype On - Unedited Font
http://www.geocities.com/darkangelxx2001/ClearTypeOff.png

Hopefully you can see the difference, and also perhaps why it was disabled from the start. The bold, or complex kanji characters almost become black splots. This is probrobly because not only was the font not originally designed with cleartype in mind, but also cleatrype only seems to work in the horizontal axis, not the vertical. So while horizonal features are nicely smoothed, vertical ones are not, and that is kinda of essential for asian text systems.

So, the big question now is, will cleartype
a: be improved into both horizontal and vertical directions for Avalon Text *and* enabled on Japanese fonts? or...
b: will a new Kanji font be provided, if, even with H&V Cleartype, the Xerox-produced font is still unreadable?

This might not be the place to say, but i must say i was something disappointed to see cleartype advertised in XP Japanese, and yet have it fail miserably to work. Surely system builders in Japan must have brought this up with the Text engineers at MS? Out here in the land of Fasionable PC's, everything is LCD, and all the text all looks equally bad.

Thanks for reading,

Mark

scobleizer
I'm the video guy

Wait until you see the new Japanese fonts that the Cleartype team is working on.

They are STUNNING.

I'll ask Bill Hill how you can get them. They'll definitely be included in Longhorn.

We just won a major award for our new Russian font too. Lots of fun stuff is coming.

By the way, you'll meet the Cleartype and Cleartype font team soon. It's part of our birthday celebration on April 6.

MarkPerris

Thanks for your reply, that's very good news for me and all other Japanese users, and of course, congratulations on winning the award for the Russian font, may i ask which award it was?

I'll also be looking forward to seeing the ClearType team interviews on Channel 9 too, that will be interesting. Another reason to be looking forward to Longhorn.

scobleizer
I'm the video guy

I forget the award's name at the moment. Sorry, I'll find out from Bill Hill and get back to you. Actually, if I remember right, it won several awards.

Sven Groot
My name has 9 letters. Coincidence? I think not...

I can't believe I never noticed that before... but you're 100% right. 私、バカだな。

I'm looking forward to the new Japanese font, that sounds like cool stuff.

Xenon101

Wow I really like what you've done with the japanese fonts! I can't understand any of it but it does look like a big improvement on my LCD screen Smiley

Minh
WOOH! WOOH!

MarkPerris wrote:
This is probrobly because not only was the font not originally designed with cleartype in mind, but also cleatrype only seems to work in the horizontal axis, not the vertical. So while horizonal features are nicely smoothed, vertical ones are not, and that is kinda of essential for asian text systems.

Have you tried to see how it would look under "Standard" font smoothing? "ClearType" is optimized for the LCD of laptops, where the red/green/blue elements are arranged horizontally. "Standard" looks to do a better job vertically, to me.

MarkPerris

Good point. Standard, as you say, does filter in both directions, however it is not functional at common web page or OS text font sizes. Using standard font smoothing on my "glyph only" resulted in the picture below, which looks worse than bitmap fonts. I think if i turned the size down, it would probrobly come on, but at the standard IE 'medium' setting, it doesn't work.

Standard Font Smoothing On, Edited Font.
http://www.geocities.com/darkangelxx2001/StandardOn.png

I think i'll make do with original bitmap font for now, and look forward to Longhorn.

Edit: Minh, i guess from your icon you are from China? What is the situation with the chinese fonts?

MarkPerris

Thanks for your reply, interesting to hear about the Vietnamese writing and fonts, i had no idea it was written with an adapted roman script.

As for the conversion, it's not so much conversion of the font from bitmap to glyph, its the removal of the fixed-size bitmap font from the whole font resource.

This is what i think is happening, (and i may not be entirely right)
The font contains 3 main sections, the kanji, the symbols, and the roman alphabet(s), and many more data sections with very small sizes, i have no idea what characters they represent. Each section has glyph data which is used for larger font sizes over about 14 points, and then a bitmap resource for the very common 10 and 12 point sizes. It seems that when used at 10 or 12 points, windows picks the bitmap font resource over the glyphs, for readability purposes when the only font smoothing type available was 'standard'.
If it had used standard smoothing in Win98 days on the glyphs, the font would have been unreadable (as you can see from my picture), hence the need for the inclusion of hand drawn bitmaps.

Hence, it is impossible to apply cleartype to the bitmaps, it will only work with glyph data. I removed all the bitmap resources, leaving only the glyphs, which forces windows to use them, and as with all glyphs, cleartype is then applied.

Sven Groot
My name has 9 letters. Coincidence? I think not...

That sounds a lot like Japanese, which has three writing systems: kanji (or Chinese characters) for words, hiragana for indicating the pronounciation of kanji words, writing words where there is no kanji spelling or the kanji isn't very common, and writing particles and verb inflexions, and lastly there's katakana, which is used for foreign words and names.

MarkPerris: where in Japan are you?

僕は卒業の後日本の東大でPhDしたいと思います。でも、今は日本語は残念ながらまだ下手です。

Minh
WOOH! WOOH!

MarkPerris wrote:
Edit: Minh, i guess from your icon you are from China? What is the situation with the chinese fonts?

Actually, I'm Vietnamese. We've switched to a roman-like alphabet since early 1900's, so adding TrueType support was easy even before Unicode codification. Adding TrueType means adding a few marks to the existing TrueType font.

BTW, something I didn't understand about your process. How did you convert a bitmap font to TrueType (which is define with curves & such)?

Addendum: Prior to the big switch, we were using a hybrid system of writing. A worse-of system. Each word is represented by a Chinese character for meaning, AND a Vietnamese/Chinese character for pronounciation. In the 17th century, a Dutch missionary arrived & adapted writing system for translating biblical texts. The rest as they say, is history...

MarkPerris

I ought to give some credit here, this is the original article in Japanese, but the pictures are universal.

http://tomtia.plala.jp/PC/ttfont/

Sven, i'm in Nara, the old capital near Osaka. If you want to chat about Japan and postgrad studies, drop me a IM on Messenger at markpschool@hotmail.com

(that email address gets enough spam anyway, posting it here won't make much difference! Smiley

reinux

Thanks Mark, I was looking for that.

By the way, if Scoble or anyone's still watching, will there be a way to have different ClearType modes on different monitors? I have an LCD and a CRT; I want ClearType on the LCD and no ClearType on the CRT.

I'm pretty sure it's doable technically since ClearType can be enabled/disabled with only a redraw of all the visible windows.

CannotResolveSymbol
{insert caption here}

I'm not Japanese, but Japanese (like in the article) has ClearType enabled on my machine. And it's an English version of XP, even. I suppose it doesn't have all the characters a full Japanese font would, though.

reinux

Actually, I'm Japanese, running English Windows and English Office 2003, and I can tell you that both are fully functional and fully featured in terms of Japanese support. Office even has Japanese "sounds-like" search and all the text functionality to East-Asian languages, like vertical text and phonetic guides etc. I was pretty amazed.

I can't figure out how to get ClearType to work with Japanese fonts though... I don't have a font editor Sad

CannotResolveSymbol
{insert caption here}

English (en_us) XP uses Arial Unicode MS for its Japanese text by default (I don't have Japanese language support specifically enabled.) And it is ClearTyped at all font sizes:

So were Japanese users just ignored in Windows XP? Or did they just think it would be better at the time if they defaulted to having it look nice under standard antialiasing?

reinux

Arial Unicode MS does work with Japanese, but I hardly see any programs using it. I think most programs are choosing MS UI Gothic. Changing everything under Appearances from Tahoma to Arial Unicode MS didn't change anything other than make it a tad uglier (I like Tahoma) for things menu bars and dialog boxes and stuff.

IE is only letting me choose certain fonts for Plain text; Arial Unicode MS isn't one of them. It seems to be listing only fonts that are exclusively Japanese... is there a setting or registry entry or anything I can change?

All the text on pages with Japanese encoding have no ClearType.

I think the reason Japanese was left out is probably because Japanese doesn't look as good (at least to my eyes) with ClearType on on small fonts. Maybe with a lighter ClearType setting, but then English doesn't look so good.

Windows Media Player seems to be the only app that does it "right" -- Japanese characters don't have ClearType; English text is ClearTyped.

SimonDa

The Windows XP Japanese fonts contain "embedded bitmaps" that implement a form of stroke reduction so that complex glyphs are readable at small sizes on screen - for this reason on XP the old fonts are special cased so that the embedded bitmaps are used.

In Windows Vista we've had time to make new fonts for Japanese, CHT, CHS and Korean designed with ClearType in mind. These were required because Avalon does not support embedded bitmaps in TrueType fonts (or old-style .FON bitmap fonts for that matter)

Further Arial Unicode has never shipped as part of Windows. It comes with Office and is rarely installed these days.

Also no there’s no current way to have different rendering paths on different displays attached to a Windows machine. We realize that this makes life difficult for some users especially those that may have a Laptop and an external LCD panel that happens to have BGR oriented sub-pixels.

Finally I should state the obvious in that hacking our fonts violates our EULA, and care should be taken not to let such modified fonts out into the wild.

Cheers, Si
Lead PM for fonts, Microsoft Typography, Windows International

reinux

Ahh, okay.

Thanks for replying Smiley

Aircraft

can someone tell me how exactly to split the TTC collection into TTF files and what exactly i need to have to do that? i went to MS Topography but i dont know what to download.

PS: i heard you can use a batch file for this. does anyone have a batch file i can use? im really lost on what to do

leighsword
LeighSword

Minh wrote:

MarkPerris wrote: Edit: Minh, i guess from your icon you are from China? What is the situation with the chinese fonts?
Actually, I'm Vietnamese. We've switched to a roman-like alphabet since early 1900's, so adding TrueType support was easy even before Unicode codification. Adding TrueType means adding a few marks to the existing TrueType font.

BTW, something I didn't understand about your process. How did you convert a bitmap font to TrueType (which is define with curves & such)?

Addendum: Prior to the big switch, we were using a hybrid system of writing. A worse-of system. Each word is represented by a Chinese character for meaning, AND a Vietnamese/Chinese character for pronounciation. In the 17th century, a Dutch missionary arrived & adapted writing system for translating biblical texts. The rest as they say, is history...

About the history of Viet(越)nam(南), the "Viet" in "Vietnam" (literally, "south of Viet") is a cognate of the "Yue". Click Yue for more information.
After visited this Vietnamese forum, I think the roman-like alphabet writing system is more like PingYing, so I can read a little bit Vietnamese without extra learning. Smiley