PtokaX forum

PtokaX => Feature Proposals => Topic started by: PPK on 03 September, 2011, 00:36:15

Poll
Question: Do you want Unicode (UTF-8) support in PtokaX
Option 1: Yes! votes: 10
Option 2: Optional. votes: 4
Option 3: NO. votes: 0
Title: Unicode (UTF-8) support...
Post by: PPK on 03 September, 2011, 00:36:15
One of things i'm missing in PtokaX (and whole nmdc protocol) is full Unicode (http://en.wikipedia.org/wiki/Unicode) (using UTF-8 (http://en.wikipedia.org/wiki/UTF-8) encoding) support. Many users actually use that because many clients already support it, but PtokaX missing it and will be nice to get proper support for that.
Advantage is of course that everyone will see characters correctly, when now for example users with central european encoding don't see correctly russian characters.
Disadvantage is that users without UTF-8 client don't see special characters correctly (in similar way as CE users don't see correctly RUS characters now). They will see correctly standard low ascii characters (characters 0 - 127 in ascii table (http://www.asciitable.com/)).
NMDC protocol is UTF-8 compatible, it will not break anything (even original old neo-modus hub and client version 1 don't have problem with UTF-8 encoded protocol messages, only don't show correctly special characters).

Please vote if that change should happen ::)
Title: Re: Unicode (UTF-8) support...
Post by: Fox_home on 03 September, 2011, 04:46:45
I agree, also request to add support for UTF8 in scripts.
Title: Re: Unicode (UTF-8) support...
Post by: WAJIM on 04 September, 2011, 12:10:16
Optional, of course!  :-\
Title: Re: Unicode (UTF-8) support...
Post by: PPK on 04 September, 2011, 19:05:13
Example image (Click to enlarge):
(http://www.ptokax.org/images/encoding-test.png)

On top smart client with UTF-8 support. Correctly show UTF-8 text and it is smart enough to detect non-UTF-8 strings and show them correctly using actual system locale (in my case central european - win-1250 encoding).

On bottom PtokaX UDP-Debug Receiver, serve here as client without UTF-8 support ::)

Quote from: Fox_home on 03 September, 2011, 04:46:45
I agree, also request to add support for UTF8 in scripts.
Question is what you mean by UTF-8 support in scripts, you can actually use UTF-8 in scripts (as is shown on my image) but you need to avoid BOM (http://en.wikipedia.org/wiki/Byte_order_mark) (ie you can't save scripts in UTF-8 in windows notepad, but you can for example in PSPad where is possible to disable BOM). Problem with BOM is (or should be, because i seen code related to that in source) fixed in Lua 5.2 :P

Quote from: WAJIM on 04 September, 2011, 12:10:16
Optional, of course!  :-\
:'(
Title: Re: Unicode (UTF-8) support...
Post by: Fox_home on 05 September, 2011, 03:28:12
Quote from: PPK on 04 September, 2011, 19:05:13
Question is what you mean by UTF-8 support in scripts, you can actually use UTF-8 in scripts (as is shown on my image) but you need to avoid BOM (http://en.wikipedia.org/wiki/Byte_order_mark) (ie you can't save scripts in UTF-8 in windows notepad, but you can for example in PSPad where is possible to disable BOM). Problem with BOM is (or should be, because i seen code related to that in source) fixed in Lua 5.2 :P
:'(
thanks
Title: Re: Unicode (UTF-8) support...
Post by: dmvn on 26 December, 2013, 22:16:50
It's interesting to know, how much code (estimated value, of course!) should be revised to support UTF-8. Basic functions like strlen() works fine with UTF, so where is the root of all evil in PX sources?
And what about your plans on this refactoring?
Title: Re: Unicode (UTF-8) support...
Post by: PPK on 27 December, 2013, 01:39:32
In PtokaX core that should be really simple, because core is working with most text data as with data of some size in bytes and not care about encoding. Here will be only all input data (from users, scripts, language files, settings) checked if they are utf-8 and if not then converted.
In gui it is more complicated, as gui on windoze don't use utf-8 or ascii (one byte per character) encoding to display texts. So here all text data will be converted from/to unicode encoding used by winapi.

I don't know when i have time to make bigger change as this one is  :(
Title: Re: Unicode (UTF-8) support...
Post by: RPGamer on 11 March, 2016, 20:23:30
That's great, you must be like god level programmer.  :angel:
Is it on Github? How can one contribute to it?