Ro-Zofola problem :: The cause and solution

Posted by Mehdi Hasan on September 18, 2010

After releasing Avro Keyboard 4.5.3 on 21 August 2010, we’ve received several “bug report” that R-Zo fola (র‍্য) isn’t being rendered correctly. People facing this problem are using Windows XP or earlier OS. We’ve mentioned in the change log that we’ve fixed R-Zo fola, they were wondering how could a fix can break anything?

After Khandata (ৎ), R-Zo fola seems to be the most discussed issue in Bangla Unicode. Here goes the disclaimer that the problem lies in your Windows and this is NOT a bug of Avro Keyboard.

History and insight:

Previously, Paul Nelson (Microsoft Corporation) suggested Ra (র) + ZWNJ + Virama (্) + Ya (য) to produce R-Zo fola (র‍্য) in Public Review Issue 9 (screenshot below)

Later, Peter Constable (again from Microsoft) revised the specification in Public Review Issue 37 and suggested that ZWJ should be used instead of ZWNJ to produce R-Zo fola (screenshot below)

Which was adopted in Unicode Standard 5.0 and is continuing to current Unicode Standard 5.2.0 (screenshot below)

Though, Indic Scripts and Languages FAQ page on Unicode.org is still showing outdated information, using Ra (র) + ZWJ + Virama (্) + Ya (য) is the most updated way to produce R-Zo fola (র‍্য) (screenshot below)

The Problem:

Windows has uniscribe engine (usp10.dll) which contains all these instructions. We’ve tested that a fresh installed Windows XP service pack 3 has still retained outdated Ra (র) + ZWNJ + Virama (্) + Ya (য) specification to produce R-Zo fola (again some reported that SP3 is updated with ZWJ combination, a perfect dll hell!). Whenever you write Bangla on applications that depends on this usp10.dll (Notepad for example), it shows broken R-Zo fola. Some application has its own rendering engine which is up to date regarding R-Zo fola issue(Office 2007 or Firefox 3.6 for example), they seems to be working fine.

Windows Vista, Windows 7, Office 2007, Office 2010 kept backward compatibility with ZWNJ format and also support the newer ZWJ combination. However, Firefox 3.6 is strictly using ZWJ with no compatibility for ZWNJ, even if you use it on Windows 7, quite normal as it is using its own rendering engine.

So whenever someone tries to write Ro-Zofola in Windows Xp using Avro Keyboard 4.5.3, it fails on Notepad, succeeds on Firefox, may or may not fail in MS Word (depending on the version). There is only one uniform solution for this problem. Read below.

The solution:

True, OS update is a tedious process for many of us. Google analytic shows that still Windows XP users are 3 times more than Windows Vista/7 users. Nevertheless, I’m afraid that we have no other options than to ask you to update your XP to Vista or Windows 7. We know many of you even don’t have required hardware to run this OS, then sticking to Avro Keyboard 4.5.1 would be your best choice. If you have lost the backup copy, it is still here.

We supported ZWNJ for may days just like Windows or Office, but since applications like Firefox are being restricted to only ZWJ now, it is our time to move forward.

Upcoming Avro Keyboard 5.x will also use ZWJ just like version 4.5.3.

(If you don’t mind playing a bit with system files, there is still a way to update usp10.dll and continue using Windows Xp. But be aware, the responsibility is yours only!)

Joy says:

September 18, 2010 at 7:29 pm

Thanks for the solution.

But I might have to continue with XP for few more years. For now, I would like to get Rifat’s solution regarding system files. Where can I get that? Can anyone tell me?

Thanks!

Mehdi Hasan says:

September 18, 2010 at 8:46 pm

http://www.omicronlab.com/blog/support/replace-usp10-dll/

sarim khan says:

September 18, 2010 at 8:34 pm

well, Here all concern is going for Windows. But as a Linux guy let me say something.
To write ro-zofola in these way,
র + \u200D + ্ + য
produces, র‍্য

AND

র + \u200D + ্ + য
produces, র‌্য

Both are unreadable in a whole brunch of programs in linux (and windows also).
It seems that it will take whole lot of time to adopt the fix in all software in all platforms. So Let me show another alternative way.

After searching for a solution for a quite long time, and after testing some ways i decided that bo + nukta way is the best.
Let me show,

\u09ac + \u09BC + ্ + য

produces, ব়্য
which is readable in both Linux and windows in all programs (as far as i found).
And i have also edited the m17ndb for linux to adopt the patch.

now let me show some screenshot of this post. Taken from Google Chrome 5 on a OpenSuse 11.3 KDE machine.
http://173.236.78.240/snapshot1.jpeg

Attach: details of all my work will be found in the following links,
http://forum.projanmo.com/post174942.html
http://forum.projanmo.com/topic16493.html
http://forum.amaderprojukti.com/viewtopic.php?f=44&t=5558

for live testing with the patched probhat and unijoy:
http://sarimbd.cz.cc/probhat/
http://sarimbd.cz.cc/unijoy/

Mehdi Hasan says:

September 19, 2010 at 1:15 am

Sarim bhai,

Thank you for your comment.

I visit those forums occasionally. Somehow I missed those threads, otherwise I’d definitely mention that it is totally a wrong approach. Not only that, it is just NOT recommended.

Unicode specifications are not just for DTP or something like What-You-See-Is-What-You-Get purpose. It involves a whole range of text processing. Your human readable Ro-Zofola doesn’t necessary produce a machine readable one. Simply put, someone searching web for র‍্যাম (Ram) will never reach your page mentioning ব়্যাম (written in your method).

I think you have heard about Unicode Normalization. The above method would be valid for ড়/ঢ়/য় (breaking ড় to ড and ়), but never for র. For details, please refer to Unicode normalization chart for Bangla:
http://www.unicode.org/charts/normalization/chart_Bengali.html

For more on Unicode normalization, please read:
1) http://unicode.org/faq/normalization.html
2) http://www.macchiato.com/unicode/nfc-faq

If you continue distributing those patches, you should mention they don’t follow Unicode standard for Ra-Zofola.

BTW, last time I used Ubuntu 9.04, Ra (র) + ZWJ + Virama (্) + Ya (য) did work. It is strange what you have mentioned “Both are unreadable in a whole brunch of programs in linux”, and of course the later part is wrong – “(and windows also)”. I’m using Windows 7 right now and undoubtedly I know what I’m seeing

- sarim khan says:
  
  September 19, 2010 at 11:52 am
  
  Sorry, i wasn’t write that is details. Actually it will be good when all programs (and OS) adopt the unicode specification. I know this my method is totally not recommend. And i’m not quite sure about the situation in windows, bcz i dont use that, but people reported it.
  
  There are other disto based on linux, not only ubuntu. And firefox people do not mention many hazards because it shows many thing readable. But I can classify the program’s in two types, one type is like firefox another is like google chrome. I can’t recall the names now, i wrote them in a forum post quite long ago.
  
  All we can do is wait and send bug fix request to the program vendors.
  
  Another thing about ঢ় ড় য় , some how they are treat differently in firefox and chrome. So a people who create a account in any site and his username contains those chars in firefox, could not login in google chrome, and vise versa. Its another bug, The programs has not fully adopted the unicode specification yet
  
  Thank you very much for wasting time for me, and sorry for my bad english. And plz test your software’s output in linux if you can manage.(At least major distros, like Ubuntu,fedora,Opensuse)