I just added Unicode functionality to the server that our MOO runs on, and I was wondering if there was any way that the full unicode set could be made to work with the MUSHclient input line. There are quite a few characters (???????????????????) that show up as question marks.
Full Unicode support
Posted by daniel perry on Sun 11 May 2003 07:13 PM — 66 posts, 310,683 views.
Hmmm, I was wondering when someone would ask that. :)
I have been thinking about Unicode for a while, however it isn't as simple as making it work "with the MUSHclient input line".
Here are some of the problems:
My preliminary research indicates the probably the simplest thing would be to expect text in UTF-8 format, which would at least support existing MUDs that only use 7-bit character encoding, however it would need fiddling if a MUD used 8-bit encoding, but not Unicode.
I am curious to know how far you have got with this server project, can you tell me ...
I have been thinking about Unicode for a while, however it isn't as simple as making it work "with the MUSHclient input line".
Here are some of the problems:
- It is not just a case of changing the input line, since input can be echoed in the output window, the output window would need to support Unicode as well.
- Plus, the whole point presumably of entering Unicode is to send it to the MUD and get it back again, so the output window would definitely need to display Unicode.
- For this to work, the send/receive routines would need to send/receive Unicode rather than single-byte characters
- All sorts of internal things (eg. command history, saved output buffer) would need to store text as Unicode
- Comparisons (eg. for searching) would need to be Unicode-aware
- Regular expression matching (eg. for triggers, aliases) would need to support Unicode
- The XML parser which reads in world files, and the writer which writes them out, would need to support Unicode (in case you had a Unicode string in a trigger, for instance)
- It would need to still support non-Unicode MUDs
- It would need to know whether or not the MUD was Unicode somehow
- There are about 6800 internal strings in MUSHclient (eg. "you cannot connect to the world") some, but not all, of those would need to be converted to Unicode, if MUSHclient became Unicode-aware
My preliminary research indicates the probably the simplest thing would be to expect text in UTF-8 format, which would at least support existing MUDs that only use 7-bit character encoding, however it would need fiddling if a MUD used 8-bit encoding, but not Unicode.
I am curious to know how far you have got with this server project, can you tell me ...
- Do you in fact send 2 bytes for each character to the client, and expect 2 bytes back?; or
- Do you use UTF-8 encoding?
- What client are you using for testing? Is it an existing MUD client? What is its name?
- Do you propose to support 8-bit (non-Unicode) clients as well?
- If so, in what way will you tell if the client is 8-bit or Unicode-aware? Also, how will you handle Unicode text being sent to a non-Unicode client?
One would hope that unicode based muds would encode it in special sequences and allow the client to translate it, if needed. For the input/output windows, I figured that a 'major' redesign would be needed and that since it didn't appear to already exist, that no one had yet added something like a <unicode>--string--</unicode> tag to anything that a client would be expected to use. Full unicode only works if you 'know' that the server is going to use it, otherwise it would attempt to translate normal single byte sequences into unicode letters. An effect not unlike switching and old dos client into 7 bit encoding, when the BBS was using 8-1 (8 bits + 1 bit checksum).
I briefly considered bringing up this issue myself, though mostly as a question about the infobar, since some fonts, including Lucida Console have imbedded unicode for special characters, including the blocks used in gauges, which currently require the use of a seperate font to produce. Not to mention any special characters in dingbats and other such fonts that are 'missing' because they only exist in the unicode sections.
I briefly considered bringing up this issue myself, though mostly as a question about the infobar, since some fonts, including Lucida Console have imbedded unicode for special characters, including the blocks used in gauges, which currently require the use of a seperate font to produce. Not to mention any special characters in dingbats and other such fonts that are 'missing' because they only exist in the unicode sections.
I am actually not the one that is writing the code, it is a freely available patch for the LambdaMOO server code, written by Lao Tzu, so I do not know the exact details of it :/
The link to his software site is: http://stompstompstomp.com/software/#unicode_moo
The link to his software site is: http://stompstompstomp.com/software/#unicode_moo
Hmm - the UTF-8 route eh?
I have been doing a bit more research, and it mightn't be quite as bad as I thought, for example the regexp routine (PCRE) has an update that supports UTF-8.
If I understand the spec correctly (RFC 2279) the encoding for UTF-8 means that:
What this means is that the client could probably pass UTF-8 strings through it without really realising it, excepting that they wouldn't be displayed properly, of course.
Thus, the minimal change necessary to make Unicode work might be ...
It is a reasonably big job, but interestingly, I think you might find that many MUDs would also handle UTF-8 (since it would just look like a character string to them), however I cannot test that right now because I can't find a terminal program that will actually let me send UTF-8.
I have been doing a bit more research, and it mightn't be quite as bad as I thought, for example the regexp routine (PCRE) has an update that supports UTF-8.
If I understand the spec correctly (RFC 2279) the encoding for UTF-8 means that:
- The first 128 characters in the ASCII set encode to themselves, basically supporting all existing messages that MUSHclient is likely to use (eg. error messages, XML sequences and so on).
- The encoding for the remaining characters get progressively longer depending on what you are trying to encode (up to a maximum of 6 bytes), however each intermediate byte is guaranteed to have a bit set. In other words, things like strlen will continue to work.
What this means is that the client could probably pass UTF-8 strings through it without really realising it, excepting that they wouldn't be displayed properly, of course.
Thus, the minimal change necessary to make Unicode work might be ...
- Have a "MUD uses UTF-8" flag which would distinguish between MUDs using Unicode and others that simply use characters in the range 0x80 to 0xFF
- Leave most of the client alone (eg. just store the UTF-8 in the command history, and output buffer in the usual way)
- For Unicode MUDs, switch to the UTF-8 version of the regular expression parser
- Where necessary (eg. doing Finds) decompose UTF-8 into Unicode for comparison purposes
- Change the screen output routine to display UTF-8 as Unicode where required
- Change the command window to handle Unicode where required, and encode into UTF-8.
- Look for places that might display Unicode (eg. you are about to replace "X" with "Y" in the command window) and handle it appropriately.
- Handle UTF-8 in the XML parser.
- Look for places where bytes will no longer equal characters (eg. wrapping output at column 80) and fix them appropriately
It is a reasonably big job, but interestingly, I think you might find that many MUDs would also handle UTF-8 (since it would just look like a character string to them), however I cannot test that right now because I can't find a terminal program that will actually let me send UTF-8.
For the RFC, see: http://www.faqs.org/rfcs/rfc2279.html
However I see what you mean about using the command window. Even pasting in Unicode shows as ??? even before MUSHclient "gets at it" so-to-speak.
Clearly there must be some change to the input window (maybe make it a Rich Edit control) to even allow the Unicode characters to be displayed.
If anyone knows more about this than me I would be pleased to hear from them. :)
Clearly there must be some change to the input window (maybe make it a Rich Edit control) to even allow the Unicode characters to be displayed.
If anyone knows more about this than me I would be pleased to hear from them. :)
My mud has support for a variety of character sets including Unicode/Latin1/CP1252/ASCII. It lets the user choose which one they want or otherwise autodetect based on ttype. (latin1 usually except in a couple of other cases.)
There is a telnet CHARSET negotiation option (see RFC 2066), but it's crap and unimplemented.
The ISO-2022 escape code for indicating UTF-8 is '\033%G', to indicate the end of UTF-8 is \033%@. Mushclient could detect these and switch.
Your analysis of UTF-8 is correct. There are a couple of other useful features about it. The initial byte of a multibyte-sequences is always in the range 0x80-0xbf, and the continuation bytes are always 0xc0-0xff. This means you can find the character boundaries really easily. Also, sort order is preserved.
Also, you'll not find sequences longer than 3, as the codespace >= 0x110000 has been abandoned.
There is a telnet CHARSET negotiation option (see RFC 2066), but it's crap and unimplemented.
The ISO-2022 escape code for indicating UTF-8 is '\033%G', to indicate the end of UTF-8 is \033%@. Mushclient could detect these and switch.
Your analysis of UTF-8 is correct. There are a couple of other useful features about it. The initial byte of a multibyte-sequences is always in the range 0x80-0xbf, and the continuation bytes are always 0xc0-0xff. This means you can find the character boundaries really easily. Also, sort order is preserved.
Also, you'll not find sequences longer than 3, as the codespace >= 0x110000 has been abandoned.
I have been experimenting with Unicode, and as far as I can see it is quite tedious to implement it. The problem is not with the UTF-8 part, that seems simple enough, but the thing I haven't got to work yet is the seemingly-simple task of showing Unicode in the command window (or any window for that matter).
It seems that if you have a non-Unicode application, however that is defined exactly, then the text windows (eg. dialog boxes, edit windows) are just "straight text" windows, with one byte per character.
To enable Unicode means trawling through the code converting hundreds and possibly thousands of strings (and code that uses them) to Unicode strings, where applicable. It isn't just a case of making everything Unicode, which itself isn't all that simple, because some stuff (eg. disk files, data from the MUD, data to the MUD, the chat system) still uses one byte per character, or possibly UTF-8. Thus it needs to be converted to/from Unicode at the appropriate point.
I'll keep experimenting, if my brain doesn't fuse first. ;)
If anyone knows how to mix Unicode windows (eg. an edit window) with a so-called non-Unicode application, please let me know.
It seems that if you have a non-Unicode application, however that is defined exactly, then the text windows (eg. dialog boxes, edit windows) are just "straight text" windows, with one byte per character.
To enable Unicode means trawling through the code converting hundreds and possibly thousands of strings (and code that uses them) to Unicode strings, where applicable. It isn't just a case of making everything Unicode, which itself isn't all that simple, because some stuff (eg. disk files, data from the MUD, data to the MUD, the chat system) still uses one byte per character, or possibly UTF-8. Thus it needs to be converted to/from Unicode at the appropriate point.
I'll keep experimenting, if my brain doesn't fuse first. ;)
If anyone knows how to mix Unicode windows (eg. an edit window) with a so-called non-Unicode application, please let me know.
I have investigated adding Unicode support for quite some time now, and am going to abandon it for a while. There are some unanswered questions about the process, that - strangely enough - seem to be very hard to work out.
Below I will describe the problem and what I have found so far, mainly to remind myself later on what I did, so I don't spend weeks re-researching it all. Maybe someone reading this will be able to suggest the solution too. :)
The problem
Without using Unicode, the main problem is that many applications, including MUSHclient, encode text data as 8-bit character strings, like this:
The problem is that 8 bits will only hold 256 different characters, and some of those (the first 32) are already "lost" as they are used for "control" characters, like carriage-return, newline, form-feed, bell, page-feed, text-terminator (0x00) and so on. Also the last one (0xFF) is used for Telnet negotiation (the IAC character), although some programs work around that by sending it twice.
Thus only 256 - 32 (224) different characters are available. This is fine for normal English text, because the normal letters (A-Z, a-z), numbers (0-9) and punctuation fit nicely into the first 128, even including the control characters. Thus to write ordinary English text you can get away with the character range 0x00 to 0x7F.
However other languages (eg. Greek, Cyrillic, Arabic, Indic, Japanese, Chinese) have so many different characters in them they simply can't be represented in the 224 characters available.
Unicode
Unicode solves this problem by encoding characters in 2 bytes each rather than one. In Windows the character type is WCHAR which defined as unsigned short. eg.
The "L" in front of the character string says to compile it as Unicode characters.
Because an unsigned short can contain 65,536 character that gives plenty of scope for encoding various languages.
Unicode is not Windows-specific, for more details see:
There is quite a good article about Unicode on MSDN at:
UTF-8
In order to have a "mixed" environment of Unicode and ordinary text (eg. on web pages) you can use UTF-8 which uses single bytes to store the first 128 characters (so most text can be the same as usual) but uses the high-order bit to signal the start of extra bytes. Also, as the high-order bit is always set in further bytes the text string can be processed in C programs (which use 0x00 as a text terminator) without any problems. The general encoding scheme is:
You can see from this scheme that if any byte has the high-order bit clear it must be in the range 00-7F, any byte with the first two bits being '11' must be the start of a Unicode sequence (where the next bit(s) tell you how many bytes follow) and any byte with the first two bits being '10' is the middle of a Unicode sequence, the start of which can be found by scanning backwards a maximum of 4 bytes.
Converting to/from Unicode
In Windows you can convert to and from Unicode using WideCharToMultiByte and MultiByteToWideChar.
eg. (eg. sInput is input string, sOutput is output string)
Byte-order marks
To identify what sort of text file you are dealing with (if data is on disk) the first 2 or 3 bytes can be used for this purpose. Since these characters won't normally occur in ordinary text this should be safe enough:
Notepad uses this scheme to identify Unicode files.
Writing a Unicode application in Windows
OK, so much for the background. :)
Windows NT (and thus 2000 and XP) support Unicode internally, and thus you can write a Unicode application for those platforms.
Many operating system calls have an Ansi version (the A version) and a Wide (Unicode) version (the W version), eg. TextOutA to draw Ansi text and TextOutW to draw Unicode text.
When using MFC (Microsoft Foundation Class) libraries you simply have to define UNICODE and the compiler selects the appropriate routine for you from the "generic" version (in this case, TextOut), like this:
There is one more trick if you want to make a Unicode application, you need to set the Link -> Output -> Entry point symbol to be "wWinMainCRTStartup" otherwise you get a link error.
Once you decide to compile with Unicode (or to make an application that can be compiled both ways) you need to use various generic typedefs, such as:
Also, literals should be enclosed with _T("blah") which expands to either "blah" or L"blah" as appropriate.
Thus a portable Unicode/Non-Unicode application might say:
Also various MFC classes (like CString) automatically become the "wide" versions when compiled with Unicode.
However in the case of MUSHclient, it is extremely tedious to convert it to Unicode after it is written. For one thing there are around 6,500 text strings (like "You can't do that") which need to be inspected and have _T() put around them.
However there are some calls (like inet_addr) which do not have a wide version, and in those cases the strings being passed to them have to be downgraded to Ansi strings before they can be used.
For another, MUSHclient has to handle non-Unicode in places like disk files, chat sessions, and normal TCP/IP to a MUD. I have attempted it, and gave up, after a couple of days of fixing one compiler error, only to find the fix caused four more.
Thus, I want to make an app that is basically non-Unicode (in other words, staying much the same as it is) but to optionally (at user request) output Unicode to the output window, and accept Unicode in the command window.
The "at user request" part is because some people may want to use characters with the high-order bit set (eg. German characters with umlats) which are not UTF-8 but simply use the characters in the range 0x80 to 0xFF.
How does Windows know whether it is a Unicode app or not?
After some research, I gather that Windows (NT) does not treat a whole application as Unicode or not, but treats individual calls on their merits. For instance, if you to TextOutW to a particular window, then you are outputting Unicode text to it. Thus, it ought to be possible to mix Unicode and non-Unicode windows in a particular application, which is what I want to do.
For example, in a test application this successfully drew Unicode in a window (once I had created the font "Lucida Sans Unicode" in the view, because that font will draw the Unicode characters):
Note the use of wcslen to find the length of a "wide" string.
In the middle of the text is an ordinary space (0x0020) demonstrating that the normal Ansi characters are in the first 128 bytes of the Unicode character space.
This particular application was not compiled with UNICODE defined, I was trying to mix Unicode and non-Unicode.
Window Procedures
There is more complexity than that in writing Unicode applications because some Windows messages handle text (eg. WM_SETTEXT) which involves text being passed around internally by Windows. Also other messages (like WM_CHAR) involve text from the user being passed to the application.
The specific problem I am trying to solve here is to create an "edit" window (in fact, the MUSHclient command window) which accepts Unicode, without having to write an edit window from scratch. Currently in a non-Unicode app (compiled without UNICODE defined) such windows just show question marks if you try to put Unicode text into them.
It appears that each Window belongs to a window "class" - which has to be pre-registered with Windows before a window of that class can be created. Amongst other things, a window class defined a window procedure (WNDPROC) which handles messages for that window.
If you register a class with RegisterClassW then Windows thinks the window is a Unicode window, otherwise if you register it with RegisterClassA it becomes an Ansi window.
Here is an example of registering a Unicode window:
Note the use of WNDCLASSW to get the Wide WNDCLASS version and using RegisterClassW to register it.
Then in the MFC PreCreateWindow function you can tell it to use a different class, like this:
However, that appears to not work, as MFC doesn't seem to like you switching to a window class it doesn't know about.
A bit more research shows you can "subclass" a window, which means that you indicate you want to have "first stab" at the messages for that window, which then get passed on to the real window procedure if you don't want to handle them. It seems that you use SetWindowLong to do that, and indeed if you use SetWindowLongW (note the W) then it registers that window (or at least, that window procedure) as one that wants Unicode. Here is an example:
For more information on subclassing, see:
However despite doing this, Unicode still appears as question marks, even though Spy shows that the window is now considered to be Unicode.
Another approach seems to be to "superclass" the window, which involves finding out about the original class (in this case "Edit") and then registering a new class based on it, like this:
However whilst this works better than making a new class from scratch, Unicode still shows up as question marks.
What I think is happening is this: first, the documentation for CallWindowProc indicates that it can handle a mix of Unicode and non-Unicode in the chain of window procedures. If you change from one to the other it converts the messages (eg. WM_SETTEXT) to/from Unicode as appropriate.
Second, MFC does its own subclassing of the windows (as part of the application framework) and thus installs Ansi subclasses in the chain (because it is not a UNICODE build).
Thus what is happening is:
Unicode message --> MFC Ansi message --> Unicode message --> display
The MFC Ansi subclass in the middle there causes the Unicode to be thrown away, and adding Unicode at either end of the chain does not really help.
What seems to be needed is to somehow stop MFC from subclassing the window at all, or to make the command window one that is independent of MFC, which I am not sure how to do.
Any constructive suggestions appreciated.
Below I will describe the problem and what I have found so far, mainly to remind myself later on what I did, so I don't spend weeks re-researching it all. Maybe someone reading this will be able to suggest the solution too. :)
The problem
Without using Unicode, the main problem is that many applications, including MUSHclient, encode text data as 8-bit character strings, like this:
char myString [] = "Nick Gammon";
The problem is that 8 bits will only hold 256 different characters, and some of those (the first 32) are already "lost" as they are used for "control" characters, like carriage-return, newline, form-feed, bell, page-feed, text-terminator (0x00) and so on. Also the last one (0xFF) is used for Telnet negotiation (the IAC character), although some programs work around that by sending it twice.
Thus only 256 - 32 (224) different characters are available. This is fine for normal English text, because the normal letters (A-Z, a-z), numbers (0-9) and punctuation fit nicely into the first 128, even including the control characters. Thus to write ordinary English text you can get away with the character range 0x00 to 0x7F.
However other languages (eg. Greek, Cyrillic, Arabic, Indic, Japanese, Chinese) have so many different characters in them they simply can't be represented in the 224 characters available.
Unicode
Unicode solves this problem by encoding characters in 2 bytes each rather than one. In Windows the character type is WCHAR which defined as unsigned short. eg.
typedef unsigned short WCHAR;
WCHAR myWideString [] = L"Nick Gammon";
The "L" in front of the character string says to compile it as Unicode characters.
Because an unsigned short can contain 65,536 character that gives plenty of scope for encoding various languages.
Unicode is not Windows-specific, for more details see:
http://www.unicode.org
There is quite a good article about Unicode on MSDN at:
http://www.microsoft.com/globaldev/getwr/steps/wrg_unicode.mspx
UTF-8
In order to have a "mixed" environment of Unicode and ordinary text (eg. on web pages) you can use UTF-8 which uses single bytes to store the first 128 characters (so most text can be the same as usual) but uses the high-order bit to signal the start of extra bytes. Also, as the high-order bit is always set in further bytes the text string can be processed in C programs (which use 0x00 as a text terminator) without any problems. The general encoding scheme is:
Unicode range UTF-8 bytes
0x00000000 - 0x0000007F 0 xxxxxxx
0x00000080 - 0x000007FF 110 xxxxx 10 xxxxxx
0x00000800 - 0x0000FFFF 1110 xxxx 10 xxxxxx 10 xxxxxx
0x00010000 - 0x001FFFFF 11110 xxx 10 xxxxxx 10 xxxxxx 10 xxxxxx
You can see from this scheme that if any byte has the high-order bit clear it must be in the range 00-7F, any byte with the first two bits being '11' must be the start of a Unicode sequence (where the next bit(s) tell you how many bytes follow) and any byte with the first two bits being '10' is the middle of a Unicode sequence, the start of which can be found by scanning backwards a maximum of 4 bytes.
Converting to/from Unicode
In Windows you can convert to and from Unicode using WideCharToMultiByte and MultiByteToWideChar.
eg. (eg. sInput is input string, sOutput is output string)
// convert Unicode to ANSI:
char sOutput [100];
WideCharToMultiByte (CP_ACP, 0, sInput, -1, sOutput, sizeof sOutput, NULL, NULL);
// convert Unicode to UTF-8:
char sOutput [100];
WideCharToMultiByte (CP_UTF8, 0, sInput, -1, sOutput, sizeof sOutput, NULL, NULL);
// convert ANSI to Unicode:
WCHAR sOutput [100];
MultiByteToWideChar (CP_ACP, MB_PRECOMPOSED, sInput, -1, sOutput,
sizeof sOutput / sizeof WCHAR);
// convert UTF-8 to Unicode:
WCHAR sOutput [100];
MultiByteToWideChar (CP_UTF8, 0, sInput, -1, sOutput,
sizeof sOutput / sizeof WCHAR);
// find the length of a UTF-8 string in characters:
int iLength = MultiByteToWideChar (CP_UTF8, 0, sInput, -1, NULL, NULL);
Byte-order marks
To identify what sort of text file you are dealing with (if data is on disk) the first 2 or 3 bytes can be used for this purpose. Since these characters won't normally occur in ordinary text this should be safe enough:
Encoding Encoded BOM
UTF-16 big-endian FE FF
UTF-16 little-endian FF FE (Windows)
UTF-8 EF BB BF
Notepad uses this scheme to identify Unicode files.
Writing a Unicode application in Windows
OK, so much for the background. :)
Windows NT (and thus 2000 and XP) support Unicode internally, and thus you can write a Unicode application for those platforms.
Many operating system calls have an Ansi version (the A version) and a Wide (Unicode) version (the W version), eg. TextOutA to draw Ansi text and TextOutW to draw Unicode text.
When using MFC (Microsoft Foundation Class) libraries you simply have to define UNICODE and the compiler selects the appropriate routine for you from the "generic" version (in this case, TextOut), like this:
#ifdef UNICODE
#define TextOut TextOutW
#else
#define TextOut TextOutA
#endif // !UNICODE
There is one more trick if you want to make a Unicode application, you need to set the Link -> Output -> Entry point symbol to be "wWinMainCRTStartup" otherwise you get a link error.
Once you decide to compile with Unicode (or to make an application that can be compiled both ways) you need to use various generic typedefs, such as:
// non-generic typedefs
CHAR = char
WCHAR = unsigned short
// generic ones
TCHAR = CHAR or WCHAR
LPTSTR = CHAR * or WCHAR *
LPCTSTR = const CHAR * or const WCHAR *
Also, literals should be enclosed with _T("blah") which expands to either "blah" or L"blah" as appropriate.
Thus a portable Unicode/Non-Unicode application might say:
TCHAR myString [] = _T("Nick Gammon");
Also various MFC classes (like CString) automatically become the "wide" versions when compiled with Unicode.
However in the case of MUSHclient, it is extremely tedious to convert it to Unicode after it is written. For one thing there are around 6,500 text strings (like "You can't do that") which need to be inspected and have _T() put around them.
However there are some calls (like inet_addr) which do not have a wide version, and in those cases the strings being passed to them have to be downgraded to Ansi strings before they can be used.
For another, MUSHclient has to handle non-Unicode in places like disk files, chat sessions, and normal TCP/IP to a MUD. I have attempted it, and gave up, after a couple of days of fixing one compiler error, only to find the fix caused four more.
Thus, I want to make an app that is basically non-Unicode (in other words, staying much the same as it is) but to optionally (at user request) output Unicode to the output window, and accept Unicode in the command window.
The "at user request" part is because some people may want to use characters with the high-order bit set (eg. German characters with umlats) which are not UTF-8 but simply use the characters in the range 0x80 to 0xFF.
How does Windows know whether it is a Unicode app or not?
After some research, I gather that Windows (NT) does not treat a whole application as Unicode or not, but treats individual calls on their merits. For instance, if you to TextOutW to a particular window, then you are outputting Unicode text to it. Thus, it ought to be possible to mix Unicode and non-Unicode windows in a particular application, which is what I want to do.
For example, in a test application this successfully drew Unicode in a window (once I had created the font "Lucida Sans Unicode" in the view, because that font will draw the Unicode characters):
WCHAR sMsg [] = { 0x0443, 0x0433, 0x043e,
0x043c, 0x0420, 0x0020,
0x0448, 0x0443, 0x043c,
0x0435, 0x043b, 0x0442,
0 };
pDC->SelectObject(m_font); // select Unicode font
TextOutW (pDC->m_hDC, 150, 150, sMsg, wcslen (sMsg));
Note the use of wcslen to find the length of a "wide" string.
In the middle of the text is an ordinary space (0x0020) demonstrating that the normal Ansi characters are in the first 128 bytes of the Unicode character space.
This particular application was not compiled with UNICODE defined, I was trying to mix Unicode and non-Unicode.
Window Procedures
There is more complexity than that in writing Unicode applications because some Windows messages handle text (eg. WM_SETTEXT) which involves text being passed around internally by Windows. Also other messages (like WM_CHAR) involve text from the user being passed to the application.
The specific problem I am trying to solve here is to create an "edit" window (in fact, the MUSHclient command window) which accepts Unicode, without having to write an edit window from scratch. Currently in a non-Unicode app (compiled without UNICODE defined) such windows just show question marks if you try to put Unicode text into them.
It appears that each Window belongs to a window "class" - which has to be pre-registered with Windows before a window of that class can be created. Amongst other things, a window class defined a window procedure (WNDPROC) which handles messages for that window.
If you register a class with RegisterClassW then Windows thinks the window is a Unicode window, otherwise if you register it with RegisterClassA it becomes an Ansi window.
Here is an example of registering a Unicode window:
HINSTANCE hInst = AfxGetResourceHandle();
WNDCLASSW WndClass;
WndClass.style = CS_DBLCLKS;
WndClass.lpfnWndProc = (WNDPROC) MainWndProc;
WndClass.cbClsExtra = (INT) NULL;
WndClass.cbWndExtra = (INT) NULL;
WndClass.hInstance = hInst;
WndClass.hIcon = LoadIcon( IDR_MAINFRAME );
WndClass.hCursor = LoadCursor( (LPTSTR) IDC_ARROW );
WndClass.hbrBackground = (HBRUSH) (COLOR_APPWORKSPACE+1);
WndClass.lpszMenuName = L"";
WndClass.lpszClassName = L"MUSHclientWindow";
if( !RegisterClassW (&WndClass) )
::AfxMessageBox ("Could not register the class");
Note the use of WNDCLASSW to get the Wide WNDCLASS version and using RegisterClassW to register it.
Then in the MFC PreCreateWindow function you can tell it to use a different class, like this:
cs.lpszClass = "MUSHclientWindow";
However, that appears to not work, as MFC doesn't seem to like you switching to a window class it doesn't know about.
A bit more research shows you can "subclass" a window, which means that you indicate you want to have "first stab" at the messages for that window, which then get passed on to the real window procedure if you don't want to handle them. It seems that you use SetWindowLong to do that, and indeed if you use SetWindowLongW (note the W) then it registers that window (or at least, that window procedure) as one that wants Unicode. Here is an example:
// store previous window procedure here
WNDPROC oldproc = NULL;
// define our own window procedure
LRESULT CALLBACK MainWndProc ( HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam )
{
switch( uMsg )
{
case WM_SETTEXT:
// handle WM_SETTEXT here ...
break;
} // end of switch
// send others to the original one
return CallWindowProcW (oldproc, hWnd, uMsg, wParam, lParam);
} // end of MainWndProc
// now install it - note use of SetWindowLongW
oldproc = (WNDPROC) SetWindowLongW (m_hWnd, GWL_WNDPROC, (long) MainWndProc);
For more information on subclassing, see:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwui/html/msdn_subclas3.asp
However despite doing this, Unicode still appears as question marks, even though Spy shows that the window is now considered to be Unicode.
Another approach seems to be to "superclass" the window, which involves finding out about the original class (in this case "Edit") and then registering a new class based on it, like this:
WNDCLASSW WndClass;
if (!GetClassInfoW(hInst, L"Edit", &WndClass))
::AfxMessageBox ("Could not GetClassInfo");
HINSTANCE hInst = AfxGetResourceHandle();
WndClass.hInstance = hInst;
WndClass.lpszClassName = L"MUSHclientWindow";
oldproc = WndClass.lpfnWndProc;
WndClass.lpfnWndProc = MainWndProc;
if( !RegisterClassW (&WndClass) )
::AfxMessageBox ("Could not RegisterClassW");
However whilst this works better than making a new class from scratch, Unicode still shows up as question marks.
What I think is happening is this: first, the documentation for CallWindowProc indicates that it can handle a mix of Unicode and non-Unicode in the chain of window procedures. If you change from one to the other it converts the messages (eg. WM_SETTEXT) to/from Unicode as appropriate.
Second, MFC does its own subclassing of the windows (as part of the application framework) and thus installs Ansi subclasses in the chain (because it is not a UNICODE build).
Thus what is happening is:
Unicode message --> MFC Ansi message --> Unicode message --> display
The MFC Ansi subclass in the middle there causes the Unicode to be thrown away, and adding Unicode at either end of the chain does not really help.
What seems to be needed is to somehow stop MFC from subclassing the window at all, or to make the command window one that is independent of MFC, which I am not sure how to do.
Any constructive suggestions appreciated.
Quote:
The initial byte of a multibyte-sequences is always in the range 0x80-0xbf, and the continuation bytes are always 0xc0-0xff. This means you can find the character boundaries really easily.
The initial byte of a multibyte-sequences is always in the range 0x80-0xbf, and the continuation bytes are always 0xc0-0xff. This means you can find the character boundaries really easily.
Looking at the bit patterns in my earlier post, I think you have that backwards. The initial byte (11 xxxxxx) will be in the range 0xC0 to 0xFF and the continuation bytes (10 xxxxxx) will be in the range 0x80 to 0xBF.
This also makes more sense for preserving the sort order, as the initial byte (which is higher, being 0xC0 to 0xFF) will sort before the continuation bytes.
This is a silly question, but do you have IME installed? (The Windows component that supports non-Europian characters. IE, Japanese Kanji. It can be found in Control Panel under Regional Settings.)
I've spent some time in a channel on IRC where Japanese characters were used quite a bit, and so I broke down and installed the language support so that they would stop displaying as broken two-byte characters in mIRC.
Once IME is installed, Windows itself handles the input of the characters. This includes installing multiple-language support for new fonts. (I have to be careful when choosing fonts, however.)
Without IME installed, UNICODE characters often appear as ? symbols, or are interpretted one byte per character.
---
I just tested using MushClient to send a few Japanese characters, with interesting results.
I have it currently connected to an IRC server where I have mIRC connected. I sent Japanese characters in both directions with the following results:
Sent from MushClient: Displayed correctly as Japanese characters in the command line. Displayed correctly at the other end in mIRC. (The IRC protocol does not doesn't echo back to the client that sent it) This worked correctly.
Sent from mIRC: Known to work fine on mIRC's end, however, the characters are obvious char, not wchar, and therefore did not display correctly.
This is using Windows XP, which admittedly, handles UNICODE better than earlier versions of Windows. (And much better than 95/98/ME).
I've spent some time in a channel on IRC where Japanese characters were used quite a bit, and so I broke down and installed the language support so that they would stop displaying as broken two-byte characters in mIRC.
Once IME is installed, Windows itself handles the input of the characters. This includes installing multiple-language support for new fonts. (I have to be careful when choosing fonts, however.)
Without IME installed, UNICODE characters often appear as ? symbols, or are interpretted one byte per character.
---
I just tested using MushClient to send a few Japanese characters, with interesting results.
I have it currently connected to an IRC server where I have mIRC connected. I sent Japanese characters in both directions with the following results:
Sent from MushClient: Displayed correctly as Japanese characters in the command line. Displayed correctly at the other end in mIRC. (The IRC protocol does not doesn't echo back to the client that sent it) This worked correctly.
Sent from mIRC: Known to work fine on mIRC's end, however, the characters are obvious char, not wchar, and therefore did not display correctly.
This is using Windows XP, which admittedly, handles UNICODE better than earlier versions of Windows. (And much better than 95/98/ME).
Erg, that should read "did not display currently in MushClient". My fault for posting at 2 AM.
Hmm - I'm using NT 4, and don't see IME as such under regional settings, however I'm installing different input locales in the hope that will do it.
However you are right, if I can solve the problem of inputting in UTF-8 the rest should be pretty trivial.
BTW - how do I type (in German, say, or Japanese) on my ordinary US keyboard?
However you are right, if I can solve the problem of inputting in UTF-8 the rest should be pretty trivial.
BTW - how do I type (in German, say, or Japanese) on my ordinary US keyboard?
A bit of testing seems to indicate that installing different input locales merely makes my keyboard behave strangely. For instance, if I type "say hello" I see "saz hello" on the screen.
However if I copy some Unicode from a test file, and paste it into the input window, it still comes out as question marks.
Still, if this problem can be solved, a Unicode version should be achievable.
Are you saying that the only problem at present is that you can type (and paste?) Unicode into the command window, but it simply doesn't display properly in the upper (output) window?
However if I copy some Unicode from a test file, and paste it into the input window, it still comes out as question marks.
Still, if this problem can be solved, a Unicode version should be achievable.
Are you saying that the only problem at present is that you can type (and paste?) Unicode into the command window, but it simply doesn't display properly in the upper (output) window?
The issue is confused here because I am not sure whether my lack of results is due to:
A bit of experimentation has found some interesting results ...
On a UTF-8 sampler page (http://www.columbia.edu/kermit/utf8.html) I found, amongst other things, this phrase in Greek:
"Ôç ãëþóóá " (this may not display in Greek due to your web browser or other problems)
Now opening that web page in Ultra-Edit appears to indicate that it is in UTF-8 ...
Looking at these characters, we see:
... and so on ...
Now, take out the 110xxxxx bits from the first byte, and the 11xxxxxx bits from the second byte, and combine the remaining ones, you get this:
(I got these names and numbers from the page: http://www.york.ac.uk/depts/maths/greekutf.htm ).
This seems to follow the UTF-8 scheme described above, the space was a single byte, the other characters were double bytes where the first three bits of the first byte were always 110 and the first two bits of the other byte were always 10. You combine the bits other than the marker bits, to get a number higher than 255, and look that up.
However if I copy that word onto the clipboard and paste it into MUSHclient's command window I just see "?? ???ssa" which appears to indicate it didn't paste properly (or didn't display properly).
I can copy and paste into Notepad, so the copying itself cannot be the problem.
I am using the same font in MUSHclient (Lucida Sans Unicode) that I successfully used in Notepad, so it would seem the font isn't the problem.
Interestingly, if I paste into UltraEdit I get the same results as in MUSHclient, so they have the same problem, whatever it is, that I do.
More tests show that pasting into Word works, but pasting into Wordpad doesn't. It certainly isn't consistent!
If I bring up the "find" dialog box, and paste the word into them, the following programs work: Word, Internet Explorer, however these don't: Notepad, Wordpad, UltraEdit, MUSHclient.
It is hard to know what to conclude from all this. I know the MUSHclient output routine is not designed to handle UTF-8, and changing that is not too hard. However to test it I need to get some UTF-8 into the command window (or displayed somehow) and that seems pretty tricky.
- Not knowing how to input (say) Japanese - I gather from my research that IME is an Input Method Engine, perhaps that would help
- Not having a Japanese font installed anyway
- If copying and pasting not knowing if the problem is:
- Unicode (or UTF-8) not being copied
- Unicode not being pasted correctly
- Unicode not being displayed
A bit of experimentation has found some interesting results ...
On a UTF-8 sampler page (http://www.columbia.edu/kermit/utf8.html) I found, amongst other things, this phrase in Greek:
"Ôç ãëþóóá " (this may not display in Greek due to your web browser or other problems)
Now opening that web page in Ultra-Edit appears to indicate that it is in UTF-8 ...
00000000 ce a4 ce b7 20 ce b3 ce bb cf 8e cf 83 cf 83 ce |.... ...........|
00000010 b1 20 ce bc ce bf cf 85 |. ......|
Looking at these characters, we see:
-
ce a4 - 11001110 10100100 -
ce b7 - 11001110 10110111 -
20 - a normal space -
ce b3 - 11001110 10110011 -
ce bb - 11001110 10111011 -
cf 8e - 11001111 10001110 -
cf 83 - 11001111 10000011
... and so on ...
Now, take out the 110xxxxx bits from the first byte, and the 11xxxxxx bits from the second byte, and combine the remaining ones, you get this:
- 1110100100 (decimal 932) - Greek capital "tau"
- 1110110111 (decimal 951) - Greek "eta"
- (space)
- 1110110011 (decimal 947) - Greek "gamma"
- 1110111011 (decimal 955) - Greek "lambda"
- 1111001110 (decimal 974) - Greek "omega with tonos"
- 1111000011 (decimal 963) - Greek "sigma"
(I got these names and numbers from the page: http://www.york.ac.uk/depts/maths/greekutf.htm ).
This seems to follow the UTF-8 scheme described above, the space was a single byte, the other characters were double bytes where the first three bits of the first byte were always 110 and the first two bits of the other byte were always 10. You combine the bits other than the marker bits, to get a number higher than 255, and look that up.
However if I copy that word onto the clipboard and paste it into MUSHclient's command window I just see "?? ???ssa" which appears to indicate it didn't paste properly (or didn't display properly).
I can copy and paste into Notepad, so the copying itself cannot be the problem.
I am using the same font in MUSHclient (Lucida Sans Unicode) that I successfully used in Notepad, so it would seem the font isn't the problem.
Interestingly, if I paste into UltraEdit I get the same results as in MUSHclient, so they have the same problem, whatever it is, that I do.
More tests show that pasting into Word works, but pasting into Wordpad doesn't. It certainly isn't consistent!
If I bring up the "find" dialog box, and paste the word into them, the following programs work: Word, Internet Explorer, however these don't: Notepad, Wordpad, UltraEdit, MUSHclient.
It is hard to know what to conclude from all this. I know the MUSHclient output routine is not designed to handle UTF-8, and changing that is not too hard. However to test it I need to get some UTF-8 into the command window (or displayed somehow) and that seems pretty tricky.
Nick, the german thing (the saz) is based on a german keyboard, because of letter frequencies, y and z are switched. Also, youll notice the shift number characters are different (!@#$%^&*()_+) etc. As well as ' ; [ being umlauts. Other keys are mapped differently, its just the way a german keyboard is setup. Windows automatically maps the keys to the language you have installed, it isnt a bug.
So, I would assume Japanese... or whatever else would do something similar, Im sure Cyrillic would be the same as well. (Cyrrilic is non unicode, I believe?)
So, I would assume Japanese... or whatever else would do something similar, Im sure Cyrillic would be the same as well. (Cyrrilic is non unicode, I believe?)
I didn't think it was a bug, it was just that what I typed didn't match the keyboard I had to hand.
What I really want is to know how to input Unicode characters using my existing setup. :)
What I really want is to know how to input Unicode characters using my existing setup. :)
I am pleased to say that MUSHclient version 3.43 will support, at least partly, Unicode in the form of UTF-8 characters arriving from the MUD. This is an option, as UTF-8 uses the high-order bit of a character which some people will currently be using for simple ASCII accented letters.
Here is a screen dump showing it in action ...
I am having trouble testing it properly as my version of Windows does not allow Unicode to be typed into the command window (in any case, I don't know where I would find, say, Russian letters on my keyboard), however by using the Debug window to push UTF-8 characters onto the screen, it seems to work OK.
Umm.. I hope that this can be turned off? The mud I play at uses the high bit for its normal purpose and a few things I use myself emply such letters. It may seriously screw up a lot more people than you think if it is on by default and can't be disabled... In fact this is bound to be even worse on mine, since I use a non-TTF font, which doesn't even have the unicode ranges in it. I am not sure how Windows handles that situation, but...
Quoting myself:
Yes, it can be turned off, and it is off by default.
Quote:
This is an option, as UTF-8 uses the high-order bit of a character which some people will currently be using for simple ASCII accented letters.
This is an option, as UTF-8 uses the high-order bit of a character which some people will currently be using for simple ASCII accented letters.
Yes, it can be turned off, and it is off by default.
Nice greek text :p
As for unicode, I haven't tried, because the MUD I play on won't accept it... I can try to send/receive greek but I'm not sure if it will be Unicode or just chars on the upper ASCII range... How can I send and receive Unicode?
Nick, I'm scared :p
Quote:
How can I send and receive Unicode?
How can I send and receive Unicode?
I was hoping *you* would know that. :)
First, go to Output configuration and check the UTF-8 (Unicode) box.
Then you could go to the debug window (Shift+Ctrl+F12) and enter:
Test: \ce\a4\ce\b7\20\ce\b3\ce\bb\cf\8e\0a
That (with a suitable font) should at least show some Greek letters on the screen (as per the screen dump above).
Now that you have some Unicode text on the screen, you could select it (shift-double-click to select a line), copy and paste down into the command window.
Then try saying or think it, eg.
think blah blah (whatever you copied)
Now, depending on the MUD server, you may or may not see it all echoed back. For example, SMAUG (stock) does not recognise the 8-bit set, and will not necessarily echo back the Unicode text, nor does PennMUSH.
The patch for SMAUG, if you have your own server, is on this site in a recent forum message. I'll investigate PennMUSH.
Now if your keyboard is setup for Greek letters (and don't ask me how to do that) you should in theory be able to type something like:
think <some Greek text>
and have it sent to the MUD, and echoed back, all correctly. Well, that is the theory.
As for testing if it is really Unicode or just high-order ASCII, turn on packet debug for a couple of packets and inspect the text.
Send something short like:
think (one letter)
You should see something like this in the packet debug:
Sent packet: 9 (10 bytes)
think .... 74 68 69 6e 6b 20 ce a4 0d 0a
The UTF-8 (Unicode) letter is in bold, but you see it takes two bytes. One has the high-order two bits on (Cx), the other doesn't.
The response back from the MUD should look similar (two bytes per Greek character).
OK, I entered the text in the debug window, came out fine, so Unicode works... When I just entered Greek in it, nothing came out (because it was ASCII, not Unicode), and when I switched off UTF-8 the ASCII text got displayed. As for sending Greek to SMAUG and getting it back, the MUD I play just omits the high-order ASCII for greek to what it echoes back, and I don't have a unicode server I could try it on... Hm, let me make a small program that echoes back what i send...
OK, I made a small greek textfile and had it echoed to MC, at first I saved it as unicode and garbage came out, then I saved it as UTF-8 and it showed up correctly. It is unicode, I turned on the packet debugger and it's 2 bytes per character... Well done :)
Oh, and, of course my keyboard is set up for greek :P Even though it's easy to do that too, just go to the language panel and add Greek to your keyboard layout, so you can test it yourself... I'm using the English edition of WinXP and everything shows up in Greek correctly. The keyboard is a standard 104-key one, have fun typing weird letters :P
Oh, and, of course my keyboard is set up for greek :P Even though it's easy to do that too, just go to the language panel and add Greek to your keyboard layout, so you can test it yourself... I'm using the English edition of WinXP and everything shows up in Greek correctly. The keyboard is a standard 104-key one, have fun typing weird letters :P
The trick will be to find a MUD that supports UTF-8.
I posted the patch for SMAUG, and am in negotiations with the PennMUSH developers. At this stage, PennMUSH doesn't support UTF-8, because of a test (isprint) in the input routine, and concerns about length-testing of strings.
I posted the patch for SMAUG, and am in negotiations with the PennMUSH developers. At this stage, PennMUSH doesn't support UTF-8, because of a test (isprint) in the input routine, and concerns about length-testing of strings.
"I posted the patch for SMAUG"
Could you tell me where that is? I've had quite a thorough Google around and failed to find it.
Could you tell me where that is? I've had quite a thorough Google around and failed to find it.
My earlier message on this page referred to "a recent forum post". See this:
http://www.gammon.com.au/forum/bbshowpost.php?bbsubject_id=3757
http://www.gammon.com.au/forum/bbshowpost.php?bbsubject_id=3757
Ah, thanks a lot. I'll try it out right away.
I'd like to have a Japanese supporting MUD running - so far I only know of one MOO (MOOsaico) which (sort of) supports Japanese text display / entry.
[EDIT]
Hmm, it compiled but it didn't actually change matters.
I guess I'll have to have a look at isprint
[EDIT2]
[...]
[EDIT3]
Ah, I think it needs
setlocale
to be used if it's going to work properly.
[EDIT4]
And if it's a multi-byte locale then you can't call printf to check a single byte.
I'd like to have a Japanese supporting MUD running - so far I only know of one MOO (MOOsaico) which (sort of) supports Japanese text display / entry.
[EDIT]
Hmm, it compiled but it didn't actually change matters.
I guess I'll have to have a look at isprint
[EDIT2]
[...]
[EDIT3]
Ah, I think it needs
setlocale
to be used if it's going to work properly.
[EDIT4]
And if it's a multi-byte locale then you can't call printf to check a single byte.
Hello everyone.
I am not even a newbie in Unicode support knowledge, so I ask for your patience while reading this message.
I have a mush game running. It is coded in my native language that uses diacritic signs from ISO-8859-02 (latin2) charset. The game is very automated; it is coded in storyteller's style, the player involves in different plots by just visiting different places. We never really cared for the input of special characters, but we wanted all possible descriptions to appear to a user in most accurate form with correct syntax and grammar, etc. We required our users to use a program called puTTY (no advertisement intended). It has an option that allows to translate character set ("Character set translation on received data"). We can set there a charset for option called "Received data assumed to be in which character set:". Thus, all descriptions from the game appear with our native diacritic characters. However, that program is not a very good solution for gaming purposes. Nevertheless, it was the only one we could find which worked for us in a way we wanted it.
You may ask how diacritic signs appeared in game's database. Well, everything is coded without special characters, since no client supports the input in this way. We just opened the database file in a simple editor which highligthed words with errors and we made needed corrections. It may sound as a lot of work, but frankly it is not. Everyone can try it by himself/herself.
My questions is, how much it differs from the Unicode support you want to implement? Can it happen the MUSHclient will have a similar function before having full Unicode support?
Thank you for your time.
--
Mandor
I am not even a newbie in Unicode support knowledge, so I ask for your patience while reading this message.
I have a mush game running. It is coded in my native language that uses diacritic signs from ISO-8859-02 (latin2) charset. The game is very automated; it is coded in storyteller's style, the player involves in different plots by just visiting different places. We never really cared for the input of special characters, but we wanted all possible descriptions to appear to a user in most accurate form with correct syntax and grammar, etc. We required our users to use a program called puTTY (no advertisement intended). It has an option that allows to translate character set ("Character set translation on received data"). We can set there a charset for option called "Received data assumed to be in which character set:". Thus, all descriptions from the game appear with our native diacritic characters. However, that program is not a very good solution for gaming purposes. Nevertheless, it was the only one we could find which worked for us in a way we wanted it.
You may ask how diacritic signs appeared in game's database. Well, everything is coded without special characters, since no client supports the input in this way. We just opened the database file in a simple editor which highligthed words with errors and we made needed corrections. It may sound as a lot of work, but frankly it is not. Everyone can try it by himself/herself.
My questions is, how much it differs from the Unicode support you want to implement? Can it happen the MUSHclient will have a similar function before having full Unicode support?
Thank you for your time.
--
Mandor
I don't mind your mentioning puTTY - I use it myself for connecting to other hosts. It is very good, but not a good MUD client.
I'm not sure what the option you refer to does exactly. I don't have access to my copy of puTTY right now. What is the problem with just using the correct font?
At present MUSHclient supports UTF-8 encoding for Unicode, which is described earlier in this thread. You should be able to adapt your server to output the correct UTF-8 codes without too much trouble.
I'm not sure what the option you refer to does exactly. I don't have access to my copy of puTTY right now. What is the problem with just using the correct font?
At present MUSHclient supports UTF-8 encoding for Unicode, which is described earlier in this thread. You should be able to adapt your server to output the correct UTF-8 codes without too much trouble.
As far as I noticed, in MUSHclient every character is displayed as being in latin1 charset.
I set the font for the output window (Courier New). I set it to display Central European characters. However, I still see latin1 (not latin2) characters in output window. I do enabled Unicode support in Settings.
It's a middle of the night for me. I will check everything in the morning and explain in detail what I have done and what the result was.
Thank you for your answer.
--
Mandor
I set the font for the output window (Courier New). I set it to display Central European characters. However, I still see latin1 (not latin2) characters in output window. I do enabled Unicode support in Settings.
It's a middle of the night for me. I will check everything in the morning and explain in detail what I have done and what the result was.
Thank you for your answer.
--
Mandor
I must have been doing something wrong. Everything I set was in World properties -> Appearance -> Output. I set a font with fixed width and chose Central European charset for it. I matched UTF-8 (Unicode) option. When I restart the world, all my native characters were not shown. They were omitted in the output.
Apologies for the delay in reply, I was off-world.
--
Mandor
Apologies for the delay in reply, I was off-world.
--
Mandor
I think you are mixing two methods here.
There are two ways of displaying non-standard (from my point of view) characters, ie. non-Latin ones.
The first way is to simply use a different code set, so that values like 0x80 to 0xFF (in hex) map to special characters for your language. This it not Unicode, and you should not check the UTF-8 box in this case. However you should choose an appropriate font that shows the correct characters that match what the MUD is sending.
The second way is to use UTF-8 encoding, which (as described earlier in this thread) maps every character in various languages to different Unicode characters. This is done by multi-byte sequences, so a single character on the screen may be sent from the MUD as two or three consecutive bytes, all with the high-order bit set. In this case you should check the UTF-8 box, and choose a suitable font that supports Unicode (eg. Lucida Sans Unicode).
However that will only work if the MUD itself sends out UTF-8 character sequences. Sounds like it doesn't.
There are two ways of displaying non-standard (from my point of view) characters, ie. non-Latin ones.
The first way is to simply use a different code set, so that values like 0x80 to 0xFF (in hex) map to special characters for your language. This it not Unicode, and you should not check the UTF-8 box in this case. However you should choose an appropriate font that shows the correct characters that match what the MUD is sending.
The second way is to use UTF-8 encoding, which (as described earlier in this thread) maps every character in various languages to different Unicode characters. This is done by multi-byte sequences, so a single character on the screen may be sent from the MUD as two or three consecutive bytes, all with the high-order bit set. In this case you should check the UTF-8 box, and choose a suitable font that supports Unicode (eg. Lucida Sans Unicode).
However that will only work if the MUD itself sends out UTF-8 character sequences. Sounds like it doesn't.
You are right. As I said before, I do not have required knowledge here.
I tried doing it using the first way you described. I have received national characters in the output window. However, they are shown in CP 1250, and not in ISO-8859-2 (it shows character #165 instead of #161, etc.). Is there any way to change it? Or would I need to change everything one more time in mush database? I do not like that idea...
--
Mandor
I tried doing it using the first way you described. I have received national characters in the output window. However, they are shown in CP 1250, and not in ISO-8859-2 (it shows character #165 instead of #161, etc.). Is there any way to change it? Or would I need to change everything one more time in mush database? I do not like that idea...
--
Mandor
I'm not sure what the problem here is exactly, but it might need changes to the way MUSHclient selects fonts. I have added that as suggestion #531.
I've probably missed the boat with answering this question, but I think any character in windows can be accessed via holding down the alt key, and typing a number on the number pad.
I'm using linux right now, but I know that þ can be done in windows buy holding down alt and typing "0222" (though I don't know exactly what number character þ is).
-CT
I'm using linux right now, but I know that þ can be done in windows buy holding down alt and typing "0222" (though I don't know exactly what number character þ is).
-CT
You can generate characters in the range 0 to 255 in that way, certainly.
But that isn't really Unicode, it is just another way of getting non-typeable characters to appear.
You can't go higher than that (eg. to get Chinese characters) because they are stored internally in a single byte.
But that isn't really Unicode, it is just another way of getting non-typeable characters to appear.
You can't go higher than that (eg. to get Chinese characters) because they are stored internally in a single byte.
This is one of the most frustrating things about various cheats to get unicode (or just characters). There is no certainty that #165 *is* the same character in font A and in font B, so you might end up thinking it is showing #161, simply because its mapped differently.
But, to get a bit technical about the real bloody mess this makes of things, Microsoft, when confronted with the need to use their OEM (the font used by DOS and built into all graphics cards for the PC, rather than making a True Type for it when in there console, cheated and used a trick like this:
Or something roughly like that. I suspect ones like puTTY work the same way, and translate the "normal" characters into there local versions, by substitution of them. The problem with both UTF-8 and MS' goofy solution is that it only works if "either" the sender and the reciever know what is going on and both methods prevent you from accessing parts of the unicode. There is a lot of stuff in the code spaces for, for example, x01NN, x02NN, x03NN, etc., where x0n is the section to display and NN is the character in that section. UTF-8 can't get at those, simply because it relies of values of x8000-xFFFF to designate a "unicode" character, and anything in x00-x7F as a "normal" letter. The letter N in true unicode is x004E, because something like 4E4F5051 would be *not* be NOPQ, but something totally different. Specifically, 4E is the CJK Unified Ideograph section, while 50 is in a no mans land, where there is no alternate set (at least that my viewer shows).
BTW, check out http://www.jhlabs.com/java/unicodeviewer.html for a good look at how this mess is arranged in just one common font.
---
Now for the "good" news. It is probably possible to use the OnPacketRecieved functions to intercept incoming text, then perform substitutions on it. There are examples of this around, though I am not sure what the links are. But even with UTF-8 turned on, there "may" be cases where the character you need is some place in the no-mans land of stuff that UTF-8 won't let you get to.
This, http://www.lischke-online.de/FontViewer.php should help anyone that wants to try to figure out a) which fonts will even work, and b) what to map the changes to in order to actually get those characters.
But, to get a bit technical about the real bloody mess this makes of things, Microsoft, when confronted with the need to use their OEM (the font used by DOS and built into all graphics cards for the PC, rather than making a True Type for it when in there console, cheated and used a trick like this:
if FONT = "System" then
display char
else
select case char
case <blah>
display box_drawing_x25
...
end select
end ifOr something roughly like that. I suspect ones like puTTY work the same way, and translate the "normal" characters into there local versions, by substitution of them. The problem with both UTF-8 and MS' goofy solution is that it only works if "either" the sender and the reciever know what is going on and both methods prevent you from accessing parts of the unicode. There is a lot of stuff in the code spaces for, for example, x01NN, x02NN, x03NN, etc., where x0n is the section to display and NN is the character in that section. UTF-8 can't get at those, simply because it relies of values of x8000-xFFFF to designate a "unicode" character, and anything in x00-x7F as a "normal" letter. The letter N in true unicode is x004E, because something like 4E4F5051 would be *not* be NOPQ, but something totally different. Specifically, 4E is the CJK Unified Ideograph section, while 50 is in a no mans land, where there is no alternate set (at least that my viewer shows).
BTW, check out http://www.jhlabs.com/java/unicodeviewer.html for a good look at how this mess is arranged in just one common font.
---
Now for the "good" news. It is probably possible to use the OnPacketRecieved functions to intercept incoming text, then perform substitutions on it. There are examples of this around, though I am not sure what the links are. But even with UTF-8 turned on, there "may" be cases where the character you need is some place in the no-mans land of stuff that UTF-8 won't let you get to.
This, http://www.lischke-online.de/FontViewer.php should help anyone that wants to try to figure out a) which fonts will even work, and b) what to map the changes to in order to actually get those characters.
So, what's the current status of this whole Unicode thing as of right now? Fully implemented? Partially?
When I type Unicode chars it shows question marks usually, but sometimes a different character all together (when I type 'ř' it shows as 'ø'). When I try to copy and paste 'ř' it shows as 'r(' Is the input field compatible with unicode?
PS: My friend reported that "řščďťňäë߀おやすみなさい" showed correcly when sent from PuTTY, but on trying to copy and paste it and send it back to me it showed as question marks. So I'm assuming the input box is plain-ASCII only?
When I type Unicode chars it shows question marks usually, but sometimes a different character all together (when I type 'ř' it shows as 'ø'). When I try to copy and paste 'ř' it shows as 'r(' Is the input field compatible with unicode?
PS: My friend reported that "řščďťňäë߀おやすみなさい" showed correcly when sent from PuTTY, but on trying to copy and paste it and send it back to me it showed as question marks. So I'm assuming the input box is plain-ASCII only?
I think the answer is "partially implemented", because there is some support in the output window for it. This is a checkbox (in the output window configuration) that will convert UTF-8 characters received from the MUD into Unicode, and then display as Unicode.
However, after extensive investigation that is documented earlier in this thread, I think, I cannot see an easy way of making the input (command) window handle Unicode in any form.
This is because when the client was first written it was made as an 8-bit application (straight ASCII), not Unicode. I attempted a while back to convert to Unicode, but it was a very hard job. There are thousands of strings internally that would need conversion, plus the issue of integrating with certain operating system calls (eg. that load/save files, or receive data from the Internet) which may need conversion to/from Unicode to 8-bit.
The conversion of the output window to optional Unicode display was easy enough, because the output window is "hand-coded" by me, and I know what it is doing. The command window on the other hand is a simple "edit control", and the details of its implementation are hidden away in the operating system.
The frustrating thing from a testing point of view is that it is hard for me to say how the client would behave if it was running under (say) a Chinese version of Windows. I am reluctant to install such a thing because, for one thing, I wouldn't be able to read anything on the screen, and thus would be uncertain whether it was working properly or not.
I think some people have reported success with MUSHclient under various other languages, but I might be wrong.
However, after extensive investigation that is documented earlier in this thread, I think, I cannot see an easy way of making the input (command) window handle Unicode in any form.
This is because when the client was first written it was made as an 8-bit application (straight ASCII), not Unicode. I attempted a while back to convert to Unicode, but it was a very hard job. There are thousands of strings internally that would need conversion, plus the issue of integrating with certain operating system calls (eg. that load/save files, or receive data from the Internet) which may need conversion to/from Unicode to 8-bit.
The conversion of the output window to optional Unicode display was easy enough, because the output window is "hand-coded" by me, and I know what it is doing. The command window on the other hand is a simple "edit control", and the details of its implementation are hidden away in the operating system.
The frustrating thing from a testing point of view is that it is hard for me to say how the client would behave if it was running under (say) a Chinese version of Windows. I am reluctant to install such a thing because, for one thing, I wouldn't be able to read anything on the screen, and thus would be uncertain whether it was working properly or not.
I think some people have reported success with MUSHclient under various other languages, but I might be wrong.
Sorry for the necropost, but I'd really like UTF8 input, so I'm bumping this. :)
In any case, if you want to test MUSHClient's behaviour in a non-UTF8 language, I suggest installing Microsoft's Applocale (from http://www.microsoft.com/globaldev/tools/apploc.mspx) or by manually changing your locale in regional and language settings.
However, regardless, if you use UTF8 encoded characters in Windows, it -should- act the same no matter what your locale is set to. If you want to test using a non-IME language such as Russian, you can pull up the onscreen keyboard to make sure that the key you press on your keyboard corresponds with the text that appears. With an IME language, however, you're stuck with either looking at the character before it's sent to the program, or using a lookup table.
Some fonts in Windows of course don't have the full gamut of characters, but I believe MS Gothic does. (At the least, I know it has Japanese and Russian)
In any case, if you want to test MUSHClient's behaviour in a non-UTF8 language, I suggest installing Microsoft's Applocale (from http://www.microsoft.com/globaldev/tools/apploc.mspx) or by manually changing your locale in regional and language settings.
However, regardless, if you use UTF8 encoded characters in Windows, it -should- act the same no matter what your locale is set to. If you want to test using a non-IME language such as Russian, you can pull up the onscreen keyboard to make sure that the key you press on your keyboard corresponds with the text that appears. With an IME language, however, you're stuck with either looking at the character before it's sent to the program, or using a lookup table.
Some fonts in Windows of course don't have the full gamut of characters, but I believe MS Gothic does. (At the least, I know it has Japanese and Russian)
Well, after a bit of experimentation I installed the Russian keyboard, which didn't seem to make much difference (perhaps because I don't have one).
Then I switched to the Russian code page (I think), which gave these results:
That is one character each for "abcd" namely F4 E8 F1 E2 - which I presume is the code page for Russian.
My first question to people who use another language is, how do you type stuff like "say"?
I couldn't because if I typed "say hello" while the code page was Russian it just sent a whole lot of Russian characters, none of which was the word "say".
I couldn't because if I typed "say hello" while the code page was Russian it just sent a whole lot of Russian characters, none of which was the word "say".
I just use alt+left shift to switch between languages.
Ah OK I get the picture now. You type "say" in EN mode, switch to RL or whatever and type what you want to say in Russian? It all becomes clearer now. :)
I think this plugin below might help. The basic problem is to get the input window to send Unicode, which it isn't designed to do.
What this plugin does is take the code-page characters, and turn them into UTF-8 for sending. It builds up a table on-the-fly from the conversion downloaded from www.unicode.org. In this particular case I used the 1251 code page (Cyrillic) however the general idea could be used for any code page, as you just access the correct table from www.unicode.org.
For example, taking the entry for:
The script parses the line, and extracts the 0xC0 and 0x0410. The 0xC0 is turned into a single byte which is the key of a table entry (the entry if you type "CYRILLIC CAPITAL LETTER A" on the keyboard). Then the 0x0410 is converted into a UTF-8 sequence by calling utils.utf8encode. This is the value that 0xC0 "maps to". In this case it is 0xD0 0x90.
Now we are ready to roll. The plugin then intercepts all text sent to the MUD by using OnPluginSend. It does a table lookup to convert the code-page values into UTF-8. The original text is dropped (by returning false) and the new text is sent instead.
To make this work, you need to configure Windows to display the correct code page, by using Control Panel -> Regional and Language Options -> Advanced. Set the "Language for non-Unicode programs" (such as MUSHclient) to the appropriate language (I used Russian for my test).
Now, when you enable the keyboard to be Russian mode (Alt+Left-Shift), as you type you see Russian characters in the input box. With the plugin installed they are converted to UTF-8 on their way out to the MUD.
To see them correctly displayed on the way back (eg. when you say something and the MUD echoes the said text), you need to check the UTF-8 (Unicode) check box in the Output window configuration.
Copy between the lines below and save this text as Translate_Unicode.xml - then use File -> Plugins to install it as a MUSHclient plugin. For a different language than Russian simply find the correct table from the Unicode web site.
I think this plugin below might help. The basic problem is to get the input window to send Unicode, which it isn't designed to do.
What this plugin does is take the code-page characters, and turn them into UTF-8 for sending. It builds up a table on-the-fly from the conversion downloaded from www.unicode.org. In this particular case I used the 1251 code page (Cyrillic) however the general idea could be used for any code page, as you just access the correct table from www.unicode.org.
For example, taking the entry for:
0xC0 0x0410 #CYRILLIC CAPITAL LETTER A
The script parses the line, and extracts the 0xC0 and 0x0410. The 0xC0 is turned into a single byte which is the key of a table entry (the entry if you type "CYRILLIC CAPITAL LETTER A" on the keyboard). Then the 0x0410 is converted into a UTF-8 sequence by calling utils.utf8encode. This is the value that 0xC0 "maps to". In this case it is 0xD0 0x90.
Now we are ready to roll. The plugin then intercepts all text sent to the MUD by using OnPluginSend. It does a table lookup to convert the code-page values into UTF-8. The original text is dropped (by returning false) and the new text is sent instead.
To make this work, you need to configure Windows to display the correct code page, by using Control Panel -> Regional and Language Options -> Advanced. Set the "Language for non-Unicode programs" (such as MUSHclient) to the appropriate language (I used Russian for my test).
Now, when you enable the keyboard to be Russian mode (Alt+Left-Shift), as you type you see Russian characters in the input box. With the plugin installed they are converted to UTF-8 on their way out to the MUD.
To see them correctly displayed on the way back (eg. when you say something and the MUD echoes the said text), you need to check the UTF-8 (Unicode) check box in the Output window configuration.
Copy between the lines below and save this text as Translate_Unicode.xml - then use File -> Plugins to install it as a MUSHclient plugin. For a different language than Russian simply find the correct table from the Unicode web site.
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE muclient>
<!-- Saved on Tuesday, June 03, 2008, 10:16 AM -->
<!-- MuClient version 4.25 -->
<!-- Plugin "Translate_Unicode" generated by Plugin Wizard -->
<muclient>
<plugin
name="Translate_Unicode_RU"
author="Nick Gammon"
id="bb1c8d004c596b19748fc66c"
language="Lua"
purpose="Translate sent text into UTF-8 (for Russian)"
date_written="2008-06-03 10:11:10"
date_modified="2008-06-04 13:20:00"
requires="4.25"
version="1.1"
>
</plugin>
<!-- Script -->
<script>
<![CDATA[
-- see http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT
-- <------------- replace here for other languages ------------->
conversion = [[
0x80 0x0402 #CYRILLIC CAPITAL LETTER DJE
0x81 0x0403 #CYRILLIC CAPITAL LETTER GJE
0x82 0x201A #SINGLE LOW-9 QUOTATION MARK
0x83 0x0453 #CYRILLIC SMALL LETTER GJE
0x84 0x201E #DOUBLE LOW-9 QUOTATION MARK
0x85 0x2026 #HORIZONTAL ELLIPSIS
0x86 0x2020 #DAGGER
0x87 0x2021 #DOUBLE DAGGER
0x88 0x20AC #EURO SIGN
0x89 0x2030 #PER MILLE SIGN
0x8A 0x0409 #CYRILLIC CAPITAL LETTER LJE
0x8B 0x2039 #SINGLE LEFT-POINTING ANGLE QUOTATION MARK
0x8C 0x040A #CYRILLIC CAPITAL LETTER NJE
0x8D 0x040C #CYRILLIC CAPITAL LETTER KJE
0x8E 0x040B #CYRILLIC CAPITAL LETTER TSHE
0x8F 0x040F #CYRILLIC CAPITAL LETTER DZHE
0x90 0x0452 #CYRILLIC SMALL LETTER DJE
0x91 0x2018 #LEFT SINGLE QUOTATION MARK
0x92 0x2019 #RIGHT SINGLE QUOTATION MARK
0x93 0x201C #LEFT DOUBLE QUOTATION MARK
0x94 0x201D #RIGHT DOUBLE QUOTATION MARK
0x95 0x2022 #BULLET
0x96 0x2013 #EN DASH
0x97 0x2014 #EM DASH
0x98 #UNDEFINED
0x99 0x2122 #TRADE MARK SIGN
0x9A 0x0459 #CYRILLIC SMALL LETTER LJE
0x9B 0x203A #SINGLE RIGHT-POINTING ANGLE QUOTATION MARK
0x9C 0x045A #CYRILLIC SMALL LETTER NJE
0x9D 0x045C #CYRILLIC SMALL LETTER KJE
0x9E 0x045B #CYRILLIC SMALL LETTER TSHE
0x9F 0x045F #CYRILLIC SMALL LETTER DZHE
0xA0 0x00A0 #NO-BREAK SPACE
0xA1 0x040E #CYRILLIC CAPITAL LETTER SHORT U
0xA2 0x045E #CYRILLIC SMALL LETTER SHORT U
0xA3 0x0408 #CYRILLIC CAPITAL LETTER JE
0xA4 0x00A4 #CURRENCY SIGN
0xA5 0x0490 #CYRILLIC CAPITAL LETTER GHE WITH UPTURN
0xA6 0x00A6 #BROKEN BAR
0xA7 0x00A7 #SECTION SIGN
0xA8 0x0401 #CYRILLIC CAPITAL LETTER IO
0xA9 0x00A9 #COPYRIGHT SIGN
0xAA 0x0404 #CYRILLIC CAPITAL LETTER UKRAINIAN IE
0xAB 0x00AB #LEFT-POINTING DOUBLE ANGLE QUOTATION MARK
0xAC 0x00AC #NOT SIGN
0xAD 0x00AD #SOFT HYPHEN
0xAE 0x00AE #REGISTERED SIGN
0xAF 0x0407 #CYRILLIC CAPITAL LETTER YI
0xB0 0x00B0 #DEGREE SIGN
0xB1 0x00B1 #PLUS-MINUS SIGN
0xB2 0x0406 #CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I
0xB3 0x0456 #CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I
0xB4 0x0491 #CYRILLIC SMALL LETTER GHE WITH UPTURN
0xB5 0x00B5 #MICRO SIGN
0xB6 0x00B6 #PILCROW SIGN
0xB7 0x00B7 #MIDDLE DOT
0xB8 0x0451 #CYRILLIC SMALL LETTER IO
0xB9 0x2116 #NUMERO SIGN
0xBA 0x0454 #CYRILLIC SMALL LETTER UKRAINIAN IE
0xBB 0x00BB #RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
0xBC 0x0458 #CYRILLIC SMALL LETTER JE
0xBD 0x0405 #CYRILLIC CAPITAL LETTER DZE
0xBE 0x0455 #CYRILLIC SMALL LETTER DZE
0xBF 0x0457 #CYRILLIC SMALL LETTER YI
0xC0 0x0410 #CYRILLIC CAPITAL LETTER A
0xC1 0x0411 #CYRILLIC CAPITAL LETTER BE
0xC2 0x0412 #CYRILLIC CAPITAL LETTER VE
0xC3 0x0413 #CYRILLIC CAPITAL LETTER GHE
0xC4 0x0414 #CYRILLIC CAPITAL LETTER DE
0xC5 0x0415 #CYRILLIC CAPITAL LETTER IE
0xC6 0x0416 #CYRILLIC CAPITAL LETTER ZHE
0xC7 0x0417 #CYRILLIC CAPITAL LETTER ZE
0xC8 0x0418 #CYRILLIC CAPITAL LETTER I
0xC9 0x0419 #CYRILLIC CAPITAL LETTER SHORT I
0xCA 0x041A #CYRILLIC CAPITAL LETTER KA
0xCB 0x041B #CYRILLIC CAPITAL LETTER EL
0xCC 0x041C #CYRILLIC CAPITAL LETTER EM
0xCD 0x041D #CYRILLIC CAPITAL LETTER EN
0xCE 0x041E #CYRILLIC CAPITAL LETTER O
0xCF 0x041F #CYRILLIC CAPITAL LETTER PE
0xD0 0x0420 #CYRILLIC CAPITAL LETTER ER
0xD1 0x0421 #CYRILLIC CAPITAL LETTER ES
0xD2 0x0422 #CYRILLIC CAPITAL LETTER TE
0xD3 0x0423 #CYRILLIC CAPITAL LETTER U
0xD4 0x0424 #CYRILLIC CAPITAL LETTER EF
0xD5 0x0425 #CYRILLIC CAPITAL LETTER HA
0xD6 0x0426 #CYRILLIC CAPITAL LETTER TSE
0xD7 0x0427 #CYRILLIC CAPITAL LETTER CHE
0xD8 0x0428 #CYRILLIC CAPITAL LETTER SHA
0xD9 0x0429 #CYRILLIC CAPITAL LETTER SHCHA
0xDA 0x042A #CYRILLIC CAPITAL LETTER HARD SIGN
0xDB 0x042B #CYRILLIC CAPITAL LETTER YERU
0xDC 0x042C #CYRILLIC CAPITAL LETTER SOFT SIGN
0xDD 0x042D #CYRILLIC CAPITAL LETTER E
0xDE 0x042E #CYRILLIC CAPITAL LETTER YU
0xDF 0x042F #CYRILLIC CAPITAL LETTER YA
0xE0 0x0430 #CYRILLIC SMALL LETTER A
0xE1 0x0431 #CYRILLIC SMALL LETTER BE
0xE2 0x0432 #CYRILLIC SMALL LETTER VE
0xE3 0x0433 #CYRILLIC SMALL LETTER GHE
0xE4 0x0434 #CYRILLIC SMALL LETTER DE
0xE5 0x0435 #CYRILLIC SMALL LETTER IE
0xE6 0x0436 #CYRILLIC SMALL LETTER ZHE
0xE7 0x0437 #CYRILLIC SMALL LETTER ZE
0xE8 0x0438 #CYRILLIC SMALL LETTER I
0xE9 0x0439 #CYRILLIC SMALL LETTER SHORT I
0xEA 0x043A #CYRILLIC SMALL LETTER KA
0xEB 0x043B #CYRILLIC SMALL LETTER EL
0xEC 0x043C #CYRILLIC SMALL LETTER EM
0xED 0x043D #CYRILLIC SMALL LETTER EN
0xEE 0x043E #CYRILLIC SMALL LETTER O
0xEF 0x043F #CYRILLIC SMALL LETTER PE
0xF0 0x0440 #CYRILLIC SMALL LETTER ER
0xF1 0x0441 #CYRILLIC SMALL LETTER ES
0xF2 0x0442 #CYRILLIC SMALL LETTER TE
0xF3 0x0443 #CYRILLIC SMALL LETTER U
0xF4 0x0444 #CYRILLIC SMALL LETTER EF
0xF5 0x0445 #CYRILLIC SMALL LETTER HA
0xF6 0x0446 #CYRILLIC SMALL LETTER TSE
0xF7 0x0447 #CYRILLIC SMALL LETTER CHE
0xF8 0x0448 #CYRILLIC SMALL LETTER SHA
0xF9 0x0449 #CYRILLIC SMALL LETTER SHCHA
0xFA 0x044A #CYRILLIC SMALL LETTER HARD SIGN
0xFB 0x044B #CYRILLIC SMALL LETTER YERU
0xFC 0x044C #CYRILLIC SMALL LETTER SOFT SIGN
0xFD 0x044D #CYRILLIC SMALL LETTER E
0xFE 0x044E #CYRILLIC SMALL LETTER YU
0xFF 0x044F #CYRILLIC SMALL LETTER YA
]]
-- <------------- end of part to be replaced for other languages ------------->
-- convert from above code page into UTF-8
unicode_table = {}
function OnPluginInstall ()
require "getlines"
for line in getlines (conversion) do
from, to = string.match (line, "^0x(%x+)%s+0x(%x+)")
if from and to then
from = tonumber (from, 16) -- convert from hex to decimal
to = tonumber (to, 16) -- ditto
unicode_table [string.char (from)] = utils.utf8encode (to)
else -- look for an undefined code point
from = string.match (string.lower (line), "^0x(%x+)%s+%#undefined")
if from then
from = tonumber (from, 16) -- convert from hex to decimal
unicode_table [string.char (from)] = "?" -- don't want bad UTF-8
end -- if undefined code
end -- if found
end -- for
ColourNote ("white", "green",
GetPluginInfo (GetPluginID (), 1) .. " plugin installed")
end -- OnPluginInstall
-- replace bytes with high-order bit set with UTF-8 equivalents
function OnPluginSend (sText)
Send ((string.gsub (sText, "[\128-\255]", unicode_table)))
return false
end -- OnPluginSend
]]>
</script>
</muclient>
Neat, thanks! (Though while it's not a complete fix, any sort of progress is progress :D)
I'm not actually Russian. (though I do know a bit of it) For an all-Russian MUD I'd imagine people would at least attempt to change commands to be more native.
On a game that I work on we've been looking at using UTF8 for various stuff and have a working implementation of it MUD-side, but many of us use MUSHclient so it's a bit tricky. :)
Have you taken a look at Uniscribe? http://www.microsoft.com/typography/developers/uniscribe/default.htm
Sadly, I have no GUI experience to speak of, so I'm not really of much help.
I'm not actually Russian. (though I do know a bit of it) For an all-Russian MUD I'd imagine people would at least attempt to change commands to be more native.
On a game that I work on we've been looking at using UTF8 for various stuff and have a working implementation of it MUD-side, but many of us use MUSHclient so it's a bit tricky. :)
Have you taken a look at Uniscribe? http://www.microsoft.com/typography/developers/uniscribe/default.htm
Sadly, I have no GUI experience to speak of, so I'm not really of much help.
I hadn't read it but am doing so now.
Internationalization of MUDs is somewhat of a tricky issue - I had to recompile SMAUG to even test my code, as it assumed that every character over 0x7F should be discarded.
Anyway, my suggested solution has raised its own issues. It seems that once you turn on UTF-8 in the Output display configuration, any text entered into the command window, other than straight ASCII (ie. less than 0x80) then fails alias processing (if you have any aliases at all), with a message about "Error execution regular expression: Bad UTF8".
I hadn't noticed this before, because I normally type English text.
It seems I have to release a new version of MUSHclient that doesn't both attempting to match UTF-8 in the *command* window (that is, aliases), because it won't be UTF-8, it will be text localized to a particular code page.
There is also an issue of copying and pasting from the output window to the command window (eg. copying some text to echo it, or a player's name) - if the text in the output window is UTF-8 then it turns into gibberish in the command window. So, the conversion from code page to UTF-8 has to be reversed when copying and pasting.
Internationalization of MUDs is somewhat of a tricky issue - I had to recompile SMAUG to even test my code, as it assumed that every character over 0x7F should be discarded.
Anyway, my suggested solution has raised its own issues. It seems that once you turn on UTF-8 in the Output display configuration, any text entered into the command window, other than straight ASCII (ie. less than 0x80) then fails alias processing (if you have any aliases at all), with a message about "Error execution regular expression: Bad UTF8".
I hadn't noticed this before, because I normally type English text.
It seems I have to release a new version of MUSHclient that doesn't both attempting to match UTF-8 in the *command* window (that is, aliases), because it won't be UTF-8, it will be text localized to a particular code page.
There is also an issue of copying and pasting from the output window to the command window (eg. copying some text to echo it, or a player's name) - if the text in the output window is UTF-8 then it turns into gibberish in the command window. So, the conversion from code page to UTF-8 has to be reversed when copying and pasting.
The plugin below lets you copy selected text from the output window, if it is in UTF-8 format, and converts it back to the appropriate code-page symbols.
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE muclient>
<!-- Saved on vrijdag, augustus 03, 2007, 2:06 -->
<!-- MuClient version 4.14 -->
<!-- Plugin "CopyScript" generated by Plugin Wizard -->
<!-- Amended slightly by Nick Gammon, from Worstje's version, on 17 Feb 2008 -->
<!-- Also amended on 3rd June 2008 to convert UTF-8 back to non-UTF-8 for use in the command window -->
<muclient>
<plugin
name="Unicode_Copy_Output"
author="Worstje"
id="29ce226131a0af3140c35141"
language="Lua"
purpose="Allows you to use CTRL+C for the output window if 'All typing goes to command window' is turned on."
save_state="n"
date_written="2007-08-03 02:04:12"
requires="4.00"
version="2.0"
>
</plugin>
<aliases>
<alias
match="^Copy_Output:Copy:29ce226131a0af3140c35141$"
enabled="y"
regexp="y"
omit_from_output="y"
sequence="100"
script="CopyScript"
>
</alias>
</aliases>
<!-- Script -->
<script>
<![CDATA[
-- THIS VERSION CONVERTS UTF-8 back to code-page text
-- See: http://www.gammon.com.au/forum/?id=2681&page=4
-- Thank you, Shaun Biggs, for taking your time to write the CopyScript
-- (formerly Copy2) function below. It was slightly altered by me to suit
-- my usage (wordwrapped lines and no \r\n at start of selection).
-- See forum: http://www.gammon.com.au/forum/?id=8052
-- see http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT
-- <------------- replace here for other languages ------------->
conversion = [[
0x80 0x0402 #CYRILLIC CAPITAL LETTER DJE
0x81 0x0403 #CYRILLIC CAPITAL LETTER GJE
0x82 0x201A #SINGLE LOW-9 QUOTATION MARK
0x83 0x0453 #CYRILLIC SMALL LETTER GJE
0x84 0x201E #DOUBLE LOW-9 QUOTATION MARK
0x85 0x2026 #HORIZONTAL ELLIPSIS
0x86 0x2020 #DAGGER
0x87 0x2021 #DOUBLE DAGGER
0x88 0x20AC #EURO SIGN
0x89 0x2030 #PER MILLE SIGN
0x8A 0x0409 #CYRILLIC CAPITAL LETTER LJE
0x8B 0x2039 #SINGLE LEFT-POINTING ANGLE QUOTATION MARK
0x8C 0x040A #CYRILLIC CAPITAL LETTER NJE
0x8D 0x040C #CYRILLIC CAPITAL LETTER KJE
0x8E 0x040B #CYRILLIC CAPITAL LETTER TSHE
0x8F 0x040F #CYRILLIC CAPITAL LETTER DZHE
0x90 0x0452 #CYRILLIC SMALL LETTER DJE
0x91 0x2018 #LEFT SINGLE QUOTATION MARK
0x92 0x2019 #RIGHT SINGLE QUOTATION MARK
0x93 0x201C #LEFT DOUBLE QUOTATION MARK
0x94 0x201D #RIGHT DOUBLE QUOTATION MARK
0x95 0x2022 #BULLET
0x96 0x2013 #EN DASH
0x97 0x2014 #EM DASH
0x98 #UNDEFINED
0x99 0x2122 #TRADE MARK SIGN
0x9A 0x0459 #CYRILLIC SMALL LETTER LJE
0x9B 0x203A #SINGLE RIGHT-POINTING ANGLE QUOTATION MARK
0x9C 0x045A #CYRILLIC SMALL LETTER NJE
0x9D 0x045C #CYRILLIC SMALL LETTER KJE
0x9E 0x045B #CYRILLIC SMALL LETTER TSHE
0x9F 0x045F #CYRILLIC SMALL LETTER DZHE
0xA0 0x00A0 #NO-BREAK SPACE
0xA1 0x040E #CYRILLIC CAPITAL LETTER SHORT U
0xA2 0x045E #CYRILLIC SMALL LETTER SHORT U
0xA3 0x0408 #CYRILLIC CAPITAL LETTER JE
0xA4 0x00A4 #CURRENCY SIGN
0xA5 0x0490 #CYRILLIC CAPITAL LETTER GHE WITH UPTURN
0xA6 0x00A6 #BROKEN BAR
0xA7 0x00A7 #SECTION SIGN
0xA8 0x0401 #CYRILLIC CAPITAL LETTER IO
0xA9 0x00A9 #COPYRIGHT SIGN
0xAA 0x0404 #CYRILLIC CAPITAL LETTER UKRAINIAN IE
0xAB 0x00AB #LEFT-POINTING DOUBLE ANGLE QUOTATION MARK
0xAC 0x00AC #NOT SIGN
0xAD 0x00AD #SOFT HYPHEN
0xAE 0x00AE #REGISTERED SIGN
0xAF 0x0407 #CYRILLIC CAPITAL LETTER YI
0xB0 0x00B0 #DEGREE SIGN
0xB1 0x00B1 #PLUS-MINUS SIGN
0xB2 0x0406 #CYRILLIC CAPITAL LETTER BYELORUSSIAN-UKRAINIAN I
0xB3 0x0456 #CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I
0xB4 0x0491 #CYRILLIC SMALL LETTER GHE WITH UPTURN
0xB5 0x00B5 #MICRO SIGN
0xB6 0x00B6 #PILCROW SIGN
0xB7 0x00B7 #MIDDLE DOT
0xB8 0x0451 #CYRILLIC SMALL LETTER IO
0xB9 0x2116 #NUMERO SIGN
0xBA 0x0454 #CYRILLIC SMALL LETTER UKRAINIAN IE
0xBB 0x00BB #RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
0xBC 0x0458 #CYRILLIC SMALL LETTER JE
0xBD 0x0405 #CYRILLIC CAPITAL LETTER DZE
0xBE 0x0455 #CYRILLIC SMALL LETTER DZE
0xBF 0x0457 #CYRILLIC SMALL LETTER YI
0xC0 0x0410 #CYRILLIC CAPITAL LETTER A
0xC1 0x0411 #CYRILLIC CAPITAL LETTER BE
0xC2 0x0412 #CYRILLIC CAPITAL LETTER VE
0xC3 0x0413 #CYRILLIC CAPITAL LETTER GHE
0xC4 0x0414 #CYRILLIC CAPITAL LETTER DE
0xC5 0x0415 #CYRILLIC CAPITAL LETTER IE
0xC6 0x0416 #CYRILLIC CAPITAL LETTER ZHE
0xC7 0x0417 #CYRILLIC CAPITAL LETTER ZE
0xC8 0x0418 #CYRILLIC CAPITAL LETTER I
0xC9 0x0419 #CYRILLIC CAPITAL LETTER SHORT I
0xCA 0x041A #CYRILLIC CAPITAL LETTER KA
0xCB 0x041B #CYRILLIC CAPITAL LETTER EL
0xCC 0x041C #CYRILLIC CAPITAL LETTER EM
0xCD 0x041D #CYRILLIC CAPITAL LETTER EN
0xCE 0x041E #CYRILLIC CAPITAL LETTER O
0xCF 0x041F #CYRILLIC CAPITAL LETTER PE
0xD0 0x0420 #CYRILLIC CAPITAL LETTER ER
0xD1 0x0421 #CYRILLIC CAPITAL LETTER ES
0xD2 0x0422 #CYRILLIC CAPITAL LETTER TE
0xD3 0x0423 #CYRILLIC CAPITAL LETTER U
0xD4 0x0424 #CYRILLIC CAPITAL LETTER EF
0xD5 0x0425 #CYRILLIC CAPITAL LETTER HA
0xD6 0x0426 #CYRILLIC CAPITAL LETTER TSE
0xD7 0x0427 #CYRILLIC CAPITAL LETTER CHE
0xD8 0x0428 #CYRILLIC CAPITAL LETTER SHA
0xD9 0x0429 #CYRILLIC CAPITAL LETTER SHCHA
0xDA 0x042A #CYRILLIC CAPITAL LETTER HARD SIGN
0xDB 0x042B #CYRILLIC CAPITAL LETTER YERU
0xDC 0x042C #CYRILLIC CAPITAL LETTER SOFT SIGN
0xDD 0x042D #CYRILLIC CAPITAL LETTER E
0xDE 0x042E #CYRILLIC CAPITAL LETTER YU
0xDF 0x042F #CYRILLIC CAPITAL LETTER YA
0xE0 0x0430 #CYRILLIC SMALL LETTER A
0xE1 0x0431 #CYRILLIC SMALL LETTER BE
0xE2 0x0432 #CYRILLIC SMALL LETTER VE
0xE3 0x0433 #CYRILLIC SMALL LETTER GHE
0xE4 0x0434 #CYRILLIC SMALL LETTER DE
0xE5 0x0435 #CYRILLIC SMALL LETTER IE
0xE6 0x0436 #CYRILLIC SMALL LETTER ZHE
0xE7 0x0437 #CYRILLIC SMALL LETTER ZE
0xE8 0x0438 #CYRILLIC SMALL LETTER I
0xE9 0x0439 #CYRILLIC SMALL LETTER SHORT I
0xEA 0x043A #CYRILLIC SMALL LETTER KA
0xEB 0x043B #CYRILLIC SMALL LETTER EL
0xEC 0x043C #CYRILLIC SMALL LETTER EM
0xED 0x043D #CYRILLIC SMALL LETTER EN
0xEE 0x043E #CYRILLIC SMALL LETTER O
0xEF 0x043F #CYRILLIC SMALL LETTER PE
0xF0 0x0440 #CYRILLIC SMALL LETTER ER
0xF1 0x0441 #CYRILLIC SMALL LETTER ES
0xF2 0x0442 #CYRILLIC SMALL LETTER TE
0xF3 0x0443 #CYRILLIC SMALL LETTER U
0xF4 0x0444 #CYRILLIC SMALL LETTER EF
0xF5 0x0445 #CYRILLIC SMALL LETTER HA
0xF6 0x0446 #CYRILLIC SMALL LETTER TSE
0xF7 0x0447 #CYRILLIC SMALL LETTER CHE
0xF8 0x0448 #CYRILLIC SMALL LETTER SHA
0xF9 0x0449 #CYRILLIC SMALL LETTER SHCHA
0xFA 0x044A #CYRILLIC SMALL LETTER HARD SIGN
0xFB 0x044B #CYRILLIC SMALL LETTER YERU
0xFC 0x044C #CYRILLIC SMALL LETTER SOFT SIGN
0xFD 0x044D #CYRILLIC SMALL LETTER E
0xFE 0x044E #CYRILLIC SMALL LETTER YU
0xFF 0x044F #CYRILLIC SMALL LETTER YA
]]
-- <------------- end of part to be replaced for other languages ------------->
-- convert from above code page into UTF-8
unicode_table = {}
function OnPluginInstall ()
require "getlines"
for line in getlines (conversion) do
from, to = string.match (line, "^0x(%x+)%s+0x(%x+)")
if from and to then
from = tonumber (from, 16) -- convert from hex to decimal
to = tonumber (to, 16) -- ditto
unicode_table [utils.utf8encode (to)] = string.char (from)
end -- if found
end -- for
end -- OnPluginInstall
-- some long alias that no-one will ever want to type
Accelerator ("Ctrl+C", "Copy_Output:Copy:29ce226131a0af3140c35141")
function CopyScript(name, line, wildcs)
-- find selection in output window, if any
local first_line, last_line = GetSelectionStartLine(),
math.min (GetSelectionEndLine(), GetLinesInBufferCount ())
local first_column, last_column = GetSelectionStartColumn(), GetSelectionEndColumn()
-- nothing selected, do normal copy
if first_line <= 0 then
DoCommand("copy")
return
end -- if nothing to copy from output window
local copystring = ""
-- iterate to build up copy text
for line = first_line, last_line do
if line < last_line then
copystring = copystring .. GetLineInfo(line).text:sub (first_column) -- copy rest of line
first_column = 1
-- Is this a new line or merely the continuation of a paragraph?
if GetLineInfo (line, 3) then
copystring = copystring .. "\r\n"
end -- new line
else
copystring = copystring .. GetLineInfo(line).text:sub (first_column, last_column - 1)
end -- if
end -- for loop
-- Get rid of a spurious extra new line at the start.
if copystring:sub (1, 2) == "\r\n" then
copystring = copystring:sub (3)
end -- if newline at start
-- correct UTF-8
-- see: http://www.gammon.com.au/forum/?id=2681&page=4
copystring = string.gsub (copystring, "[\192-\247][\128-\191]+", unicode_table)
-- finally can set clipboard contents
SetClipboard(copystring)
end -- function CopyScript
]]>
</script>
</muclient>
Quote:
I'm not actually Russian.
I'm not actually Russian.
Me neither. ;)
If the general idea works, you should be able to publish customized plugins for the languages which are in use on your MUD. I presume there aren't hundreds.
Basically look up the code-page translations on unicode.org, insert them into the plugin at the appropriate place, and save with some sort of suffix (eg. Translate_Unicode.RU.xml).
I should point out that this is, fairly obviously, a hybrid "work around" solution.
The main problem really is that MUSHclient is not Unicode-enabled, and despite my best efforts previously, it was hard to get Unicode out of the command window.
What you will have here is, if you set the code page appropriately (in International settings), one language you can use in the Command window (presumably your native language), however by enabling UTF-8 in the Output window, all languages can be displayed (with a suitable font).
The proposed "copy from output window" plugin would let you copy your *own* language from the output window, for editing and resending. Copying a different code set would result in gibberish still, because the Command window is not Unicode.
The main problem really is that MUSHclient is not Unicode-enabled, and despite my best efforts previously, it was hard to get Unicode out of the command window.
What you will have here is, if you set the code page appropriately (in International settings), one language you can use in the Command window (presumably your native language), however by enabling UTF-8 in the Output window, all languages can be displayed (with a suitable font).
The proposed "copy from output window" plugin would let you copy your *own* language from the output window, for editing and resending. Copying a different code set would result in gibberish still, because the Command window is not Unicode.
Version 4.26 has now been released, which should work properly with copying and pasting from the output window, if you install the above plugin.
Alas, pasting from other programs is still broken, and what's most important, so is typing things from the keyboard.
My knowledge of Windows is sketchy (I'm an Unix guy), but I would tackle the problem the following way:
* no 16-bit internal strings. That's a pain in the rear, and as you said, there's 6k strings you would have to change, not to mention all function calls and what not...
* the input box itself needs to be an Unicode window, though
* when taking data from it, you would call GetWindowTextW() (always!) then WideCharToMultiByte() to CP_UTF8 if mushclient is in UTF-8 mode and to CP_ACP otherwise.
* ... and the same with MultiByteToWideChar() and SetWindowTextW() the other way
My knowledge of Windows is sketchy (I'm an Unix guy), but I would tackle the problem the following way:
* no 16-bit internal strings. That's a pain in the rear, and as you said, there's 6k strings you would have to change, not to mention all function calls and what not...
* the input box itself needs to be an Unicode window, though
* when taking data from it, you would call GetWindowTextW() (always!) then WideCharToMultiByte() to CP_UTF8 if mushclient is in UTF-8 mode and to CP_ACP otherwise.
* ... and the same with MultiByteToWideChar() and SetWindowTextW() the other way
You should at least be able to get typing into the window working, as I am no expert but I managed after trying a couple of things.
First, I added a second language as a keyboard option:
Then, I set my code page to Russian, so that when I used my extra keyboard settings, Russian is what I would see:
Now in the bottom-right corner I see a keyboard code (EN) which shows what I type will appear in English:
I type "say " into the command window (which appears as such), and then hit the keyboard modifier to switch to Russian (Alt+Left_Shift which is the default). Now the keyboard code changes:
Now what I type appears in Russian, which is what I want. However when I hit <Enter> to send it, the plugin switches the code page data into UTF-8, which is what arrives at the MUD.
Applocale is a much simpler way to change the code page, as an aside.
Quote:
the input box itself needs to be an Unicode window, though
the input box itself needs to be an Unicode window, though
Well that is the hard bit. I tried for a few days previously, but I don't think you can make individual windows in a non-Unicode program to be Unicode.
The source code is freely available, if someone can make it work I would be pleased to hear from them.
Quote:
Alas, pasting from other programs is still broken,
Alas, pasting from other programs is still broken,
To make that work, as I presume the code on the clipboard is UTF-8, would be to make a plugin that does what my "copy from the output window" plugin does. You would hit a function key (eg. F8), the plugin kicks in, grabs the clipboard contents, switches it from UTF-8 to the code page, and puts it back. Or that is the theory at least.
I know this isn't perfect, but my initial tests with Russian seemed to show it worked smoothly enough, providing you didn't try to get too fancy and copy and paste from one application to another.
I suspect the copying/pasting thing might be related to clipboard "types" where the UTF-8 data is not necessarily stored in the TEXT data type (but I could be wrong).
As an example of another language, if you wanted Greek encoding, you could go to this page:
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT
Grab the mappings for 0x80 to 0xFF (see below) and paste them into the two plugins instead of the Russian ones:
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT
Grab the mappings for 0x80 to 0xFF (see below) and paste them into the two plugins instead of the Russian ones:
0x80 0x20AC #EURO SIGN
0x81 #UNDEFINED
0x82 0x201A #SINGLE LOW-9 QUOTATION MARK
0x83 0x0192 #LATIN SMALL LETTER F WITH HOOK
0x84 0x201E #DOUBLE LOW-9 QUOTATION MARK
0x85 0x2026 #HORIZONTAL ELLIPSIS
0x86 0x2020 #DAGGER
0x87 0x2021 #DOUBLE DAGGER
0x88 #UNDEFINED
0x89 0x2030 #PER MILLE SIGN
0x8A #UNDEFINED
0x8B 0x2039 #SINGLE LEFT-POINTING ANGLE QUOTATION MARK
0x8C #UNDEFINED
0x8D #UNDEFINED
0x8E #UNDEFINED
0x8F #UNDEFINED
0x90 #UNDEFINED
0x91 0x2018 #LEFT SINGLE QUOTATION MARK
0x92 0x2019 #RIGHT SINGLE QUOTATION MARK
0x93 0x201C #LEFT DOUBLE QUOTATION MARK
0x94 0x201D #RIGHT DOUBLE QUOTATION MARK
0x95 0x2022 #BULLET
0x96 0x2013 #EN DASH
0x97 0x2014 #EM DASH
0x98 #UNDEFINED
0x99 0x2122 #TRADE MARK SIGN
0x9A #UNDEFINED
0x9B 0x203A #SINGLE RIGHT-POINTING ANGLE QUOTATION MARK
0x9C #UNDEFINED
0x9D #UNDEFINED
0x9E #UNDEFINED
0x9F #UNDEFINED
0xA0 0x00A0 #NO-BREAK SPACE
0xA1 0x0385 #GREEK DIALYTIKA TONOS
0xA2 0x0386 #GREEK CAPITAL LETTER ALPHA WITH TONOS
0xA3 0x00A3 #POUND SIGN
0xA4 0x00A4 #CURRENCY SIGN
0xA5 0x00A5 #YEN SIGN
0xA6 0x00A6 #BROKEN BAR
0xA7 0x00A7 #SECTION SIGN
0xA8 0x00A8 #DIAERESIS
0xA9 0x00A9 #COPYRIGHT SIGN
0xAA #UNDEFINED
0xAB 0x00AB #LEFT-POINTING DOUBLE ANGLE QUOTATION MARK
0xAC 0x00AC #NOT SIGN
0xAD 0x00AD #SOFT HYPHEN
0xAE 0x00AE #REGISTERED SIGN
0xAF 0x2015 #HORIZONTAL BAR
0xB0 0x00B0 #DEGREE SIGN
0xB1 0x00B1 #PLUS-MINUS SIGN
0xB2 0x00B2 #SUPERSCRIPT TWO
0xB3 0x00B3 #SUPERSCRIPT THREE
0xB4 0x0384 #GREEK TONOS
0xB5 0x00B5 #MICRO SIGN
0xB6 0x00B6 #PILCROW SIGN
0xB7 0x00B7 #MIDDLE DOT
0xB8 0x0388 #GREEK CAPITAL LETTER EPSILON WITH TONOS
0xB9 0x0389 #GREEK CAPITAL LETTER ETA WITH TONOS
0xBA 0x038A #GREEK CAPITAL LETTER IOTA WITH TONOS
0xBB 0x00BB #RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
0xBC 0x038C #GREEK CAPITAL LETTER OMICRON WITH TONOS
0xBD 0x00BD #VULGAR FRACTION ONE HALF
0xBE 0x038E #GREEK CAPITAL LETTER UPSILON WITH TONOS
0xBF 0x038F #GREEK CAPITAL LETTER OMEGA WITH TONOS
0xC0 0x0390 #GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS
0xC1 0x0391 #GREEK CAPITAL LETTER ALPHA
0xC2 0x0392 #GREEK CAPITAL LETTER BETA
0xC3 0x0393 #GREEK CAPITAL LETTER GAMMA
0xC4 0x0394 #GREEK CAPITAL LETTER DELTA
0xC5 0x0395 #GREEK CAPITAL LETTER EPSILON
0xC6 0x0396 #GREEK CAPITAL LETTER ZETA
0xC7 0x0397 #GREEK CAPITAL LETTER ETA
0xC8 0x0398 #GREEK CAPITAL LETTER THETA
0xC9 0x0399 #GREEK CAPITAL LETTER IOTA
0xCA 0x039A #GREEK CAPITAL LETTER KAPPA
0xCB 0x039B #GREEK CAPITAL LETTER LAMDA
0xCC 0x039C #GREEK CAPITAL LETTER MU
0xCD 0x039D #GREEK CAPITAL LETTER NU
0xCE 0x039E #GREEK CAPITAL LETTER XI
0xCF 0x039F #GREEK CAPITAL LETTER OMICRON
0xD0 0x03A0 #GREEK CAPITAL LETTER PI
0xD1 0x03A1 #GREEK CAPITAL LETTER RHO
0xD2 #UNDEFINED
0xD3 0x03A3 #GREEK CAPITAL LETTER SIGMA
0xD4 0x03A4 #GREEK CAPITAL LETTER TAU
0xD5 0x03A5 #GREEK CAPITAL LETTER UPSILON
0xD6 0x03A6 #GREEK CAPITAL LETTER PHI
0xD7 0x03A7 #GREEK CAPITAL LETTER CHI
0xD8 0x03A8 #GREEK CAPITAL LETTER PSI
0xD9 0x03A9 #GREEK CAPITAL LETTER OMEGA
0xDA 0x03AA #GREEK CAPITAL LETTER IOTA WITH DIALYTIKA
0xDB 0x03AB #GREEK CAPITAL LETTER UPSILON WITH DIALYTIKA
0xDC 0x03AC #GREEK SMALL LETTER ALPHA WITH TONOS
0xDD 0x03AD #GREEK SMALL LETTER EPSILON WITH TONOS
0xDE 0x03AE #GREEK SMALL LETTER ETA WITH TONOS
0xDF 0x03AF #GREEK SMALL LETTER IOTA WITH TONOS
0xE0 0x03B0 #GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS
0xE1 0x03B1 #GREEK SMALL LETTER ALPHA
0xE2 0x03B2 #GREEK SMALL LETTER BETA
0xE3 0x03B3 #GREEK SMALL LETTER GAMMA
0xE4 0x03B4 #GREEK SMALL LETTER DELTA
0xE5 0x03B5 #GREEK SMALL LETTER EPSILON
0xE6 0x03B6 #GREEK SMALL LETTER ZETA
0xE7 0x03B7 #GREEK SMALL LETTER ETA
0xE8 0x03B8 #GREEK SMALL LETTER THETA
0xE9 0x03B9 #GREEK SMALL LETTER IOTA
0xEA 0x03BA #GREEK SMALL LETTER KAPPA
0xEB 0x03BB #GREEK SMALL LETTER LAMDA
0xEC 0x03BC #GREEK SMALL LETTER MU
0xED 0x03BD #GREEK SMALL LETTER NU
0xEE 0x03BE #GREEK SMALL LETTER XI
0xEF 0x03BF #GREEK SMALL LETTER OMICRON
0xF0 0x03C0 #GREEK SMALL LETTER PI
0xF1 0x03C1 #GREEK SMALL LETTER RHO
0xF2 0x03C2 #GREEK SMALL LETTER FINAL SIGMA
0xF3 0x03C3 #GREEK SMALL LETTER SIGMA
0xF4 0x03C4 #GREEK SMALL LETTER TAU
0xF5 0x03C5 #GREEK SMALL LETTER UPSILON
0xF6 0x03C6 #GREEK SMALL LETTER PHI
0xF7 0x03C7 #GREEK SMALL LETTER CHI
0xF8 0x03C8 #GREEK SMALL LETTER PSI
0xF9 0x03C9 #GREEK SMALL LETTER OMEGA
0xFA 0x03CA #GREEK SMALL LETTER IOTA WITH DIALYTIKA
0xFB 0x03CB #GREEK SMALL LETTER UPSILON WITH DIALYTIKA
0xFC 0x03CC #GREEK SMALL LETTER OMICRON WITH TONOS
0xFD 0x03CD #GREEK SMALL LETTER UPSILON WITH TONOS
0xFE 0x03CE #GREEK SMALL LETTER OMEGA WITH TONOS
0xFF #UNDEFINED
An example of the first plugin (on this page) in operation is here:
It looks like defining UNICODE would (technically) do it, but it'll cough up a whole slew of errors (about 5000 or so) when you attempt this.
Yes I tried that, and the problem is not easily solved. For example, some things like the PCRE regexp-matcher don't use Unicode, they use 8-bit strings. It accepts UTF-8, but that means you need to convert back and forwards from wide strings to UTF-8. And then there are the MUDs, most of which send 8-bit text, not UTF-8 nor wide strings.
Isn't Unicode (e: Well, UTF8 that is, causing additional fun/grief because Windows uses UTF16, which is a bit different.) more or less backwards compliant with ASCII characters below 0x7F anyways?
Granted, you would have to go from UTF16 to UTF8 for regexp, true enough.
But, for most MUDs, it shouldn't be a problem if they don't use characters over 0x7F, but if they do (if it's a non-unicode, non-english MUD), you end up with a bit of a problem. Which, I suppose, is one reason why other clients aren't unicode.
Granted, you would have to go from UTF16 to UTF8 for regexp, true enough.
But, for most MUDs, it shouldn't be a problem if they don't use characters over 0x7F, but if they do (if it's a non-unicode, non-english MUD), you end up with a bit of a problem. Which, I suppose, is one reason why other clients aren't unicode.
Unicode isn't one single thing for a start. Just check out www.unicode.org to see what I mean. Basically the idea is to represent various characters (glyphs) in a consistent way by assigning a different number to each one. But how that number is stored can vary somewhat. UTF-8 uses an encoding system that is indeed identical to non-Unicode for characters <= 0x7F, however once you move to higher values you have heaps of options. Do you want 16-bit characters? 32-bits? Which orders are the bytes? Big-endian or little-endian?
Under the Windows compiler, enabling the UNICODE define switches the representation of characters from char (8 bit) to long (16 bit). Straight away this won't work for Unicode characters > 0xFFFF. Also you can't just copy stuff from the MUD (8 bit characters) into the internal spaces (16-bit characters) without using a special call.
It's a can of worms, one I don't propose to open in the near future.
Under the Windows compiler, enabling the UNICODE define switches the representation of characters from char (8 bit) to long (16 bit). Straight away this won't work for Unicode characters > 0xFFFF. Also you can't just copy stuff from the MUD (8 bit characters) into the internal spaces (16-bit characters) without using a special call.
It's a can of worms, one I don't propose to open in the near future.
It's a whole range, yes, and UTF8 is a superset of ASCII, which, as I understand, is one of its biggest advantages. The larger problem, I suppose, is that (as perviously stated) Windows uses UTF16 internally (source: http://msdn.microsoft.com/en-us/library/ms776459(VS.85).aspx), which complicates matters somewhat. (additionally, you get into the endianness issue) With UTF8 you get 'pretty much' any character in regular use. (the entirety of the BMP, past this most fonts don't even have representations anyways, but that's getting wildly off topic. e:Plus, to my understanding, UTF8 supports up to U+10FFFF anyways.)
All in all, I suppose it's a relatively minor issue (since those honestly needing client-side UTF8 support can't be all that numerous) and development time may be better spent elsewhere.
edit: That is to say, endianness doesn't matter in UTF8 as it does in UTF16/32, since UTF8 is byte oriented. One thing to note though, in both UTF8 [i]and[/i] UTF16, is that characters aren't fixed width. (as in size) Therefore, UTF16 can handle codes above U+FFFF (and indeed, so can UTF8)
UTF8 is as widely supported as it is simply because it's (more or less) backwards compatible with ASCII right out of the box, so it can take a standard ASCII string (if the characters are all <=0x7F, that is) and be happy with it. In any case, it's quite an undertaking to convert a program as big as MUSHclient into a 100% UTF8 program.
All in all, I suppose it's a relatively minor issue (since those honestly needing client-side UTF8 support can't be all that numerous) and development time may be better spent elsewhere.
edit: That is to say, endianness doesn't matter in UTF8 as it does in UTF16/32, since UTF8 is byte oriented. One thing to note though, in both UTF8 [i]and[/i] UTF16, is that characters aren't fixed width. (as in size) Therefore, UTF16 can handle codes above U+FFFF (and indeed, so can UTF8)
UTF8 is as widely supported as it is simply because it's (more or less) backwards compatible with ASCII right out of the box, so it can take a standard ASCII string (if the characters are all <=0x7F, that is) and be happy with it. In any case, it's quite an undertaking to convert a program as big as MUSHclient into a 100% UTF8 program.
the first plugin shown on http://www.gammon.com.au/forum/bbshowpost.php?id=2681&page=4 currently has two entries for date_written, which will cause the plugin to fail to load
Changed to date_modified, thanks.