Pickling

Posted by Poromenos on Fri 08 Apr 2005 06:09 PM — 58 posts, 214,866 views.

Greece #0
Any idea why this doesn't work?

from pickle import *
a = {}
world.SetVariable("a", dumps(a))
a = loads(world.GetVariable("a"))
Russia #1
That's because of Unicode. You need to convert the result of world.GetVariable back to ascii before it can be loaded. To do that use:


import pickle, codecs

#get the encoder function, this will return a tuple with the 
#string in first element
enc = codecs.getencoder('utf8')

a= {}
world.SetVariable("a", pickle.dumps(a))

#first encode then loads()
d = pickle.loads(enc(world.GetVariable("a")[0]))

#check what it is
world.Note(repr(d))
Russia #2
If you want to do the pickling/unpickling alot then you can just proxy the loads function:


import pickle, codecs

def loads(uString):
    enc = codecs.getencoder('utf8')
    return pickle.loads(enc(uString)[0])

d = loads(world.GetVariable("a"))
Greece #3
Oh great, thanks! I would have never thought about Unicode, is it MC that converts to unicode, or Python that expects a unicode string? By the way, there is a small error in your first post,
d = pickle.loads(enc(world.GetVariable("a")[0]))
should be
d = pickle.loads(enc(world.GetVariable("a"))[0]).
Thanks again.
Greece #4
Okay, I CANNOT get this to work. Where's a debugger when you need one :/ This is my code:

def OnPluginInstall():
    if world.GetVariable("varAreas"):
        encUnicode = getencoder('utf8')
        dicAreas = pickle.loads(encUnicode(world.GetVariable("varAreas"))[0])

def OnPluginSaveState():
    world.SetVariable("varAreas", pickle.dumps(dicAreas))

def fnShowInfo(strName, strLine, strWildcards):
    for strItem, lstContents in dicAreas.items():
        world.note(strItem)

I populate the dictionary and reload the plugin. It shows me that it saved and loaded the correct pickled string, but when I call fnShowInfo after loading, it displays nothing. Any ideas?
Greece #5
Hmm, somehow I don't think it's the same variable after unpickling... It is dead inside, it is not the same variable I married. I dumps, loads, it appears to be OK when I repr() it, but the enumeration doesn't work. And, if I dumps it again, the second loads returns nothing.
Greece #6
OK, I think I have found the problem. I think it is scoping, it's OK when unpickled, but it somehow doesn't survive past that function. I'm a bit hazy on using globals in python, doesn't "global dicAreas" means that it'll act the same as a global in VBScript, i.e. be the same variable in all scopes underneath it? I should really learn to edit my posts, by the way.
Greece #7
Okay, the folks at #python helped me realise the error of my ways. For future reference, you have to add
global dicAreas
wherever you want an assignment to go to a global variable instead of a local one.
Russia #8
Yes, if you try to assign to a global object from local scope, Python looks in that scope and if it doesn't find an object with that name it just presumes that you want a new local object created and does that. So once you exit the scope the new object is destroyed and the global one had never had any assignments to it. A very common mistake. You can read from a global object without declaring it though.
Russia #9
Oh, and I think that Unicode is the pywin32 thing actually, though I could be wrong but that's what is solely responsible for COM.
Greece #10
Now I get an "insecure string pickle" error when I try to load the dumped dictionary. An entry in the dictionary consists of the following:

dicAreas[strWildcards[1]] = [strWildcards[0], strWildcards[2], strWildcards[3], "", [ "", "" ], ""]

Any ideas?
Greece #11
Ah, got it... The last string should have been unicode, u""...
Greece #12
Okay... Now, whenever I load the pickled string, there is an appended \r in the end of each dictionary name. What's up with these pickling problems? Is it MC's variable handling? When I dump to a file I don't have to decode unicode or anything... This is very strange...
Amended on Thu 14 Apr 2005 09:39 PM by Poromenos
Greece #13
Could it be MC's XML library? Maybe it's storing \r\n but removing only the \n on load? Nick?
Amended on Sat 16 Apr 2005 12:32 PM by Poromenos
Greece #14
Anything at all? :P Someone's paying me to write a script and I'm stuck because of this.
Russia #15
Was away from my computer that has both Mushclient and Python installed for a couple of days, but I'll look at what's causing this now.
Russia #16
Can you post the exact code that's demonstrating that behaviour? Because I've tried dumping/loading some stuff and it works without a problem, so it's probably something very specific.
Greece #17
Yes, it works fine if you don't save the world, but if you save and reload from the xml file, it messes up. Here it is:


import pickle, string, textwrap
from codecs import getencoder

dicAreas = {}

dicAreas[u"Test"] = [u"Test", u"Test", u"Test", u"", [u"Shyte", u"Small"], u"http://www.poromenos.org"]
world.SetVariable("varAreas", pickle.dumps(dicAreas))

- SAVE AND CLOSE WORLD -

//import pickle, string, textwrap
from codecs import getencoder

encUnicode = getencoder('utf8')
dicAreas = pickle.loads(encUnicode(world.GetVariable("varAreas"))[0])
world.note(`dicAreas`)
Russia #18
Ok, I see it. This is likely Mushclient fixing up the newlines so that they fit the Windows way (CRLF). Here's an exerpt from the help page on Base64Decode:

Quote:
Note that due to the way strings are represented internally, it is not possible for the decoded string to contain the NULL character (hex 0x00) and be returned correctly.


The 0x00 is exactly what \r stands for in Python. So the solution looks to be stripping the NULL characters from the string stored in a Mushclient var before loads'ing it. This is fairly simple to do:


dicAreas = pickle.loads(encUnicode(world.GetVariable("varAreas").replace("\r",""))[0])


Or you can move the replace() method to the result of decoding, rather than the initial Unicode string returned by world.GetVariable, though it shouldn't make much difference. And once again, you are better off with using proxy functions unless you want to type in all that crap by hand every time.
Greece #19
Ah, thanks. I don't understand though, why would the variable contain \0s? I thought dumps produced a string, isn't that supposed to not have \0? I'll go with your solution, but what if I want to store newlines? Won't they break when I loads?
And I just use it once in OnPluginInstall, so a proxy function is not really needed.
Russia #20
Dumps doesn't put them in there, Mushclient does. And what it does is actually 0x0d not 0x00, as I originally thought. When Python dumps() the dictionary, it delimits lines with 0x0a (\n). However, either when saving the variable, or loading it from XML, Mushclient seems to convert the line endings to 0x0d0x0a. I think there was some function somewhere in Python's standard lib that converted between the '\r\n' newlines and the '\n' ones, but I can't recall where I saw it.

And if you want to store newlines then you can always just store '\n' and rely on Mushclient to convert them for you. Using the '\n' by itself always worked for me in Mushclient, for example:

world.Tell("I want it to be a\n world.Note")

or:

world.Send("I am too lazy to do\nmany\nworld.Send's\nso\nI\nuse\nalot\n.")

Besides, pickling always stores newlines inside pickled strings by escaping them: '\n' -> '\\n' and then unescapes them when loading. So you can store whatever you want and it should come out as expected, this issue only has to do with the formatting of the pickle string, not what is actually pickled inside it.
Amended on Sun 17 Apr 2005 01:50 PM by Ked
Greece #21
Ah, I see. Well, it's good to know that everything works, but I must have misunderstood something, because my question remains: Shouldn't dumps not include \0s in the strings (for MC to misconvert)? If I understood you correctly, either dumps includes 0x00 in the string, which MC saves as 0x0d, or MC converts \n to 0x0d0x0a (the windows default) and then does not remove the 0x0d. Is that it?
Russia #22
The answer is: MC converts 0x0a to 0x0d0x0a. Sorry for the confusion. I am not sure what happens when you try to store 0x00, but it looks like... nothing happens - you get the 0x00 back when you loads, even after closing and reopenning the world. Same should happen with \r and \n in a pickled string, they are just escaped by the pickler in its picklerish way, and then unescaped when loading to produce the initial string.
Greece #23
Right, right... Well, that's understandable, since it runs on windows... But its inability to remove the \r is a MC bug, isn't it?
Russia #24
Well, I guess you could say that Mushclient doesn't preserve strings the way they were saved with it. But why would it want to battle against the OS its running on? So you could equally say that certain parts of Python's standard library (namely the pickler) don't obey the rules of the OS where they are ran. Then again - if pickler bends the Windows way when running on Windows, then so much for portability. So since pickler insists on being independent and doing things its own way (essentially the Unix way), it's up to its user to watch out for rough edges like newlines of different formats. Thus, I don't see anyone's fault here.
Greece #25
Ah, but the pickler saves extensions as \r\n in windows and \n in linux, doesn't it? Anyway, I'm not saying MC should save them as \n, but it should give me back exactly what I saved... If it adds a \r, it should remove it when it loads.
Netherlands #26
I ran into this problem with MUSH not giving back precisely what it stores too. Nick, is there a chance you could have a look into this and see if it can be fixed up? The use of replace() like Ked suggested seems more of a workaround for a bug in MUSH than a work around for a bug in Python, and I can imagine MUSHclient undoing what it did being quite a bit faster than a Python replace function could for really huge pickles.
USA #27
Umm. Saving it exactly as it was given would be a lot easier than "undoing what it did". Mind you, even the former would require changing the code to **not** use what ever OS dependent function is mangling things, but how would MUSH know that the original format "was" different than what it saved?

Frankly, I have always found the whole line ending stuff BS anyway. I mean, **technically** a CR is, on the old teletypes, supposed to, "return the print position back to the start of the line ***without*** moving to a new line." New line is supposed to, "move the print position to a new line, ***without*** changing the current column position." Ironically, this means that Windows is the one ding it right, not Linux. But, because Linux does it wrong, and that is the widest used standard, it also does something even stupider and treats CR and LF as the "same" thing when found, thus double spacing everything in some applications that are not smart enough to strip out the redundant information. Its all around a complete fracking mess. And the Linux/Unix convention was probably a result of some BS like, "Saving one byte of space per line, so documents don't take up as much when stored.", or something similarly absurd today. That's only a guess though.
USA #28
Quote:
And the Linux/Unix convention was probably a result of some BS like, "Saving one byte of space per line, so documents don't take up as much when stored."
I would say it's because the only reason you would go to a new line without going to the beginning of the line is when you had physical limitations of the machine you were on. I see basically no reason to keep the notion of "new line, same column". I actually think that what Linux does makes lots of sense. I find it somewhat amusing that you say Windows acting like an old-fashioned type-writer is the "right" thing. :-) I also note that you are self-contradictory, in that you think saving one byte per line is absurd today, yet you think emulating ancient typewriters is the way to go.

Quote:
it also does something even stupider and treats CR and LF as the "same" thing when found, thus double spacing everything in some applications that are not smart enough to strip out the redundant information.
This doesn't make any sense. Why would a Windows application choose to add lines for \r characters? It can just always add lines for the \n and be done with it. I'm not sure what redundant information you're referring to.

You run into problems with Mac, which for some reason decided to go with yet another line convention and uses \r for everything.
Australia Forum Administrator #29
Quote:

I ran into this problem with MUSH not giving back precisely what it stores too. Nick, is there a chance you could have a look into this and see if it can be fixed up?


I am a bit lost about what "pickling" is exactly, I thought you did that to vegetables, like onions.

Anyway, I am assuming from the general gist here that you are using some function to dump something into binary format, which is saved in a variable, which is in a plugins save file or a world file, and then that variable either won't load in correctly next time, or has some characters omitted or replaced. Is that it?

Maybe it would help to construct a very small test case so we can agree on the exact problem. For example:


 SetVariable ("test", "test\n")


A quick test seems to show that any combination of \n, \r, and \r\n save correctly - that is, you can examine the saved variable and it is saved as written. However, attempting to save \0 does not work because MUSHclient uses 0x00 as a string terminator in a number of places, hence the warning under functions like the Base 64 decode.

It has been a standard practice when using C libraries for a long time to use 0x00 as a string terminator (rightly or wrongly), and when I started writing MUSHclient I used that convention. Various functions (like strlen) which are used extensively, detect string lengths by scanning a string for that terminator.

Lua uses a different method, which is to store the length seperately, which is why the Lua versions of those functions can handle the 0x00 byte, however only for storage in Lua variables. Once they are placed into MUSHclient variables the same problem is likely to rear its head.

However back to the pickled onions problem, things change when *loading* the saved variables. You can look at the source for the XML parser in MUSHclient (xmlparse.cpp), to see a couple of things it does to the string data whilst loading it:


  // convert tabs to spaces, we don't want tabs in our data
  m_strxmlBuffer.Replace ('\t', ' ');  // line 216


And a bit further on (line 675 onwards):


   // copy if not nested, and not inside an element definition
   //  -- omit carriage returns
   if (iDepth == 0 && 
      !bInside && 
      *pi != '\r')
     {
     // make linefeeds into carriage return/linefeed
     if (*pi == '\n')
       *po++ = '\r';
     *po++ = *pi;    // copy if not inside an element
     }


What I read from this is that you will find:

  • Tabs (0x09) will be converted to spaces
  • Carriage returns (0x0D) will be omitted
  • Line feeds (0x0A) will be loaded as 0x0D 0x0A


I don't remember the reason to convert tabs to spaces, there must have been one. Perhaps it was because people were using external editors to create plugins, and put tabs inside the XML, which were then not recognised later on in the XML parser. As for the carriage-returns/linefeeds it is basically doing what it does to incoming MUD data - try to normalise the various line endings into standard Windows CR/LF form. For example, if you edited a multi-line trigger on a Unix platform, and only had linefeeds between lines, it would not display correctly on Windows.

What I recommend you do is convert "binary" strings yourself before saving them (eg. using Base64Encode) and then converting them back on loading (eg. using Base64Decode). However you will still have problems with a string with 0x00 in it. In that case I assume if Python is indeed creating strings with 0x00 in them, it will have tools to convert them into printable form - like its own version of
Base64Encode / Base64Decode.

To be honest, when I wrote the variables stuff (and the XML loader) I expected people to want to store things like mob names, not pure binary data with imbedded nulls, carriage-returns and linefeeds. If I had expected that, then MUSHclient would have been written from the start with more robust string handling - that is, the ability to store strings with 0x00 in them, which basically means all the standard C string libraries can't be used.
Amended on Mon 30 Apr 2007 09:35 PM by Nick Gammon
USA #30
Quote:
I am a bit lost about what "pickling" is exactly, I thought you did that to vegetables, like onions.
I think it's another word for serialization. After all you pickle food items to conserve them, so I suppose "pickling" data would be a conservation process of some sort.

Personally I just prefer "saving" or "serializing"... :)
USA #31
Quote:
But, because Linux does it wrong, and that is the widest used standard, it also does something even stupider and treats CR and LF as the "same" thing when found, thus double spacing everything in some applications that are not smart enough to strip out the redundant information.

Actually, most programs I've used just have a ^M at the end of the line when they don't strip out the extra line feed. Nearly any editor I use has settings to deal with a file as DOS, Unix, or Mac formats depending on what you want. Saves a lot of time and effort rather than making you convert the ends yourself.

Quote:
I am a bit lost about what "pickling" is exactly, I thought you did that to vegetables, like onions.

Pickle is a fairly standard serialization for data in python. It converts the data into a byte string similar to the serialize function included with MUSHclient. It does not save the code within the function, so you can edit the code and unpickle the data back into an instance of the new object, allowing you to fix logic errors and such without data loss.
Australia Forum Administrator #32
Quote:

But, because Linux does it wrong ...


I don't think you can say it is wrong, if you define end of a line as a linefeed. Then it is correct, by definition, which I suppose is what Unix does. It seems odd to me to have to use 2 characters to indicate end of line, because you have a whole heap of possible problems then.

If cr/lf (0x0D 0x0A) is end of line, then what is:

  • Linefeed on its own?
  • Carriage-return on its own?
  • lf/cr? (ie. 0x0A 0x0D)


These questions go away if you stick to a single byte.

There have been other schemes to indicate the end of a line - one operating system I worked on had a length indicator at the start of each line (that is, a 2-byte field that was the length of the line).

Netherlands #33
It looks like I resurrected an ancient thread with a fresh discussion. Wonderful!

Pickling is, as some have already mentioned, a way of storing data. In this particular instance, I am using it to save a dictionary containing information about a number of players in a dictionary where the values are containing tuples. I got it all working fine using the workarounds mentioned above, that isn't the issue.

However, I am not dealing with pure binary data here. Pickled objects are, in the current implementation, just strings. All characters I have been able to see so far have been ASCII characters (but I don't deal in unicode). My 'beef' is that, irregardless of the platform, there may be a very good reason we want to store just \r or just \n or maybe the precise combination. Pickling, serializing, saving, whatever you wish to call it, is one of those.

When you serialize objects, there can not be any doubt about bytes being what they are. Removing or changing a single byte could possibly ruin the functionality. As such, I can imagine that the Python library has decide to use strings with \n as line-endings, given the fact it is the most practical solution.

The following is a light bit of code I have written to deal with (un)pickling in MUSHclient, although I am quite afraid what 'other' changes could possibly break the pickled string.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#                     PICKLING IN A NUTSHELL                    #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

def GetPickleVariable(MUSHVariable, Default=None):
	# MUSH stores strings at UTF 8 and returns
	# them in that shape, too. On top of that, it
	# mutilates \n into \r\n when storing as XML, but
	# it won't remove the extra \r when loading again.
	# See: http://www.gammon.com.au/forum/bbshowpost.php?id=5410&page=1
	enc = codecs.getencoder('utf8')
	APickle = world.GetVariable(MUSHVariable)
	if (APickle != None):
		return pickle.loads(enc(APickle.replace("\r", ""))[0])
	else:
		return Default

def SetPickleVariable(MUSHVariable, PythonVariable):
	world.SetVariable(MUSHVariable, pickle.dumps(PythonVariable))


I am well aware I need to implement a catch for the exception loads() throws when the incoming data is not proper, but in its essence, this works for as far I've been able to test it. Stuff saved using SetPickleVariable() seems to be loaded fine by GetPickleVariable() in all tests I did so far.

With the risk on going off on a theoretical rant, and the fact that I may be spoiled that Python is usually a very clean language, but isn't the usage of replace() to remove something -hoping- it won't destroy another part of the string rather crude? On top of that.. the help for SetVariable() and GetVariable() never state anything about changing the values to fit. They 'set' or 'retrieve' the 'value of a variable'.

Anyhow, I'm done with my reasoning in circles here. I promise not to write anymore posts after coming back from a night out. :)
Australia Forum Administrator #34
Quote:

On top of that, it
# mutilates \n into \r\n when storing as XML, but
# it won't remove the extra \r when loading again.


This isn't strictly correct. It stores them OK, but changes them when reading back in.

Quote:

... but isn't the usage of replace() to remove something -hoping- it won't destroy another part of the string rather crude?


Yes, a bit crude. :)

Conceivably the save routine could convert \t to 	 - then it would read back in without changing it, and still fixup tabs to spaces where they are outside a quoted value.

Quote:

On top of that.. the help for SetVariable() and GetVariable() never state anything about changing the values to fit.


They don't, apart from the proviso of the 0x00 value, which I didn't expect people to put into strings in the first place.

It is the *loading* that changes the values, the Get/Set work OK, internally.

Quote:

enc = codecs.getencoder('utf8')


I don't see how this will help. UTF-8 only encodes differently bytes >= the value 128, it won't change \r, \n or \t.



Amended on Tue 01 May 2007 04:07 AM by Nick Gammon
Australia Forum Administrator #35
I have amended the help for GetVariable and SetVariable to clarify this situation:


Please note that the following characters will not be handled correctly:

* The byte value hex 00 (otherwise known as 0x00 or null). This is used as a string terminator in MUSHclient, and attempts to imbed 0x00 values into variables will result in the variable being terminated at the 0x00 position.

* The "tab" character hex 09 (otherwise known as 0x09). You can use the tab character inside variables internally, however when variables are loaded from a plugin state file, or a MUSHclient world file, tabs are converted to spaces.

* Carriage-return (hex 0D or 0x0D) and line-feeds (hex 0A or 0x0A). You can use these internally however you like, however when they are read from a plugin state file, or a MUSHclient world file, carriage returns are dropped, and line-feeds are converted to the sequence 0x0D 0x0A (carriage-return followed by linefeed).


If you are planning to use variables to store "binary" data - that is, data that can have all 256 possible character values, you are advised to convert them to base-64 encoded strings (using Base64Encode) when saving them, and conver them back (using Base64Decode) when loading them. Also be aware that only the Lua version of the base 64 encoding (utils.base64encode and utils.base64decode) will correctly handle the 0x00 value.

Netherlands #36
Sorry, I wrote that code+comment before you pasted the relevant code. :)

And I'm not sure why I need the UTF8 conversion (I merely found this topic after my first attempts to unpickle were in vain) so maybe I can remove it. But right now, I'm just glad it works. And thanks for the adjustment of the help files, although personally I'd think think the html-encoding you mentioned would be a cleaner solution.
USA #37
What are CR and LF on there own? I thought I already explained what those did in the original teletype system where they originated. CR ***only*** returned the Carriage, i.e., the print position to the "start of the line". This was used for doing things like bolding, overstriking, underlining, etc, since you couldn't have those extra characters on a print head, you had to produce them on the printer/teletype by telling it, "Return me to the first carriage position, then proceded." LF was **only** a line feed. In other words, when you used that, it didn't reset the carriage to the start of the line, it left it where it was, then continued printing from there. You had the same thing with a type writer. One lever "moved" the page to the next line, i.e. LF. The "Carriage Return" involved physically pushing the carriage back to the original position, otherwise once you moved down a line you where still typing at the last place the carriage was before you moved to the new line.

When computers and automatics where introduced, someone decided to confuse the issue and make the "Enter" or "Return" key perform "both" actions. Apparently, under Unix, they opted to do that for everything, but not by generating both symbols, but by treating LF (move one line down the page) as though it was "both". This of course fouls up anything treating those characters as their original meanings. DOS and even most simpler text boxes for Windows get it wrong too though. They often show the markers for the LF, if the CR is missing, but do not correctly display the data on a new line. In theory, if either one worked right, then this:

1LF2LF3LF4LF5CRLFEnd

Should display as:

1
 2
  3
   4
    5
End


It doesn't, in *either* system. DOS/Windows will often simply ignore the LF if not paired. Unix/Linux will, in its smarter applications, flag the file internally and treat it as DOS instead of Unix (or convert it by throwing out the redundant info), or it will, in the less smart ones, treat them as two sequential commands to do the same thing, double spacing. Both behaviors are technically wrong, if you follow the original intent of the codes. And as I said, the Unix/Linux version was probably done to save storage space back when 100MG HDs where something almost unheard of and everything larger was stored on huge banks of tape reels.
Amended on Tue 01 May 2007 06:55 PM by Shadowfyr
USA #38
Oh, and just to make things even more absurd... The text modes for both OSes *do* follow the correct usages (usually), for part of it, with LF being one of 4 codes that move the cursor one position up, down, right or left, without printing anything, while CR... Still does what electronic typewriters did and perform both actions at the same time (usually). I presume some may treat it as a, "return to start of line.", but I am not sure.
USA #39
I have the impression we just got a lecture of dubious utility on what the difference between CR and LF is but I also have the impression you're not responding to a single of my points. In particular, you repeat your assertion about why you think Unix made the change, without responding to anything I wrote about other reasons they might have chosen.

And a new reason: why is it such a bad thing to have chosen to eliminate useless characters on very storage-limited systems? You speak of it as if it's the greatest of all stupidities.
Australia Forum Administrator #40
I think we are wandering off-topic a bit here. Forget old-fashioned teletype terminals. The issue is how to get pickling to work.

I just want to point out that I think it is reasonable for MUSHclient to change \n on its own to \r\n. One of the reasons for that is that if you try to display stuff to the notepad window (a Windows Edit control), and you don't use \r\n (but just \n) the window looks all screwy.

Now if someone has scripted a trigger or something that displays stuff in the notepad (eg. "send to notepad") and they (for simplicity) just put \n between each line, they will unexpectedly get bad looking output. So, the ImportXML, assumes that line endings need "fixing up". It doesn't assume that someone will want to try to import binary data, and that \r or \n are likely to appear on their own.

Remember, changes like that (and the tab to spaces thing) were put there to make people happy, and generally in response to a complaint of some sort. Sure, it can be removed, but then one person is happy (the pickler) and I get a flood of complaints from other users that the new version has "broken" ImportXML.
USA #41
Quote:
When computers and automatics where introduced, someone decided to confuse the issue and make the "Enter" or "Return" key perform "both" actions.

I honestly hope that you are not suggesting that in order to go to the beginning of a new line one should have to press return and enter (or whatever two keys you want).

Quote:
Apparently, under Unix, they opted to do that for everything, but not by generating both symbols, but by treating LF (move one line down the page) as though it was "both". This of course fouls up anything treating those characters as their original meanings. DOS and even most simpler text boxes for Windows get it wrong too though. They often show the markers for the LF, if the CR is missing, but do not correctly display the data on a new line.

Ok, so *nix systems got it wrong... M$ got it wrong... I'll assume that you think that Mac has it wrong, since they use just a carriage return... Did anyone get it "right?" Keep in mind, that this is all just how various programs interpret the data as a standard.

One more thing about the teletype machines. They used the Baudot code, not ASCII. If you still care to argue about the teletypes, please enjoy yourself. Might also want to address the issue that the delete (127) character originally turned a character into a null (0) code because you could not just remove hole punches. Line feed also just meant that a line's worth of paper should be pushed out of the printer on teletypes, so should that happen when we look at a text file?

All in all, the only valid argument I can see is that since MUSHclient is intended to be run on Windows systems, it should conform to Windows standards. Either that, or have a toggle for people to choose and watch in amusement as various scripts get completely messed up. In C, "\n" is supposed to default to the native newline code, so it should be interpreted as a CR/LF.
USA #42
Just for fun, I saved three files. Both had "blah blah" twice on separate lines. I saved one as Unix type, one as DOS type, and one as Mac type ( LF, CR/LF, CR respectively).

biggs@omni ~ $ cat temp.nix
blah blah
blah blah
biggs@omni ~ $ cat temp.dos
blah blah
blah blah
biggs@omni ~ $ cat temp.mac
biggs@omni ~ $

That's with cat 6.9. Seems to interpret Unix and DOS formats "correctly" while not dealing with Mac's format well.
USA #43
I have never seen a unixy terminal correctly deal with \r characters. Even the *#*!"@%!&@! terminal on OS X gets confused with \r characters, making it a #"@$"%""@^!@$!@"#" pain to do anything on the command line (e.g. grep) with files written in Mac editors.

And then of course you have the editors/programs that only use \r as the line editor -- the Lisp interpreter we use is like that. What that means is that if you give it Unix line endings, as soon as it sees a line-comment (like C's //) it considers the rest of the file to be a comment.....

This is a very major problem to me. (In case you hadn't noticed. :-) )
Australia Forum Administrator #44
Quote:

Seems to interpret Unix and DOS formats "correctly" while not dealing with Mac's format well.


Well it did what it was told to - moved the cursor to the start of the line twice, where the prompt overwrote it.

I think we are confusing two things here - the physical movement of the cursor (typewriter head in the old technology), and the logical end of a line.

In the olden days, the ASCII codes simply told a piece of mechanical equipment what to do:

  • carriage return - return the "carriage" (the thing that carried the print head) to the start of the line
  • line feed - move the paper up one line
  • backspace - move the carriage back one space
  • form feed - feed a new form (new piece of paper)
  • delete - in the case of paper tape, where you had 7 holes punched for the 7 possible bits, "delete" the character by punching all 7 holes (so actually you would backspace over the bad character, and then hit delete to remove it)


With the advent of modern terminals (and PCs and programs that run on them), all of these have morphed into a slightly different meaning:

  • carriage return - move the cursor back to the start of the line - perhaps. In the case of the Mac operating system, the carriage return character became the "logical end of line" marker.
  • line feed - start a new line. In the case of the Unix operating system this also became a "logical end of line marker". That is to say, you expect more from your shell (or MUD client) than it simply moving the cursor to the start of the line. You expect it to interpret the previous line as a "command". Thus, you don't normally backspace over a linefeed.

    In the case of Windows, I'm not sure, where it uses carriage-return/linefeed combinations, whether the carriage-return, or the linefeed, or the combination together, trigger the "logical end of line" processing.
  • backspace - delete the character to the left of the cursor (this is more than simply moving the cursor, you notice).
  • form feed - blank the screen (it can hardly feed the paper on your screen, as it doesn't use paper)
  • delete - delete the character to the right of the cursor. This is different behaviour to the old paper tape punches, where you actually have the "deleted" character left on the tape.


Thus I think you have to be careful if you are mixing paradigms, just because the old teletype terminals moved the carriage around, is no real reason these days to say that a carriage return "must mean" that the "carriage" "returns".

Amended on Wed 02 May 2007 10:39 AM by Nick Gammon
USA #45
Quote:
Well it did what it was told to - moved the cursor to the start of the line twice, where the prompt overwrote it.

I understand, which is why I had correctly in quotes. It's just that for the first two, the text is displayed in the way that the respective OSes would normally display them, whereas the last one does not.

Quote:
In the case of Windows, I'm not sure, where it uses carriage-return/linefeed combinations, whether the carriage-return, or the linefeed, or the combination together, trigger the "logical end of line" processing.

The combination of the two is the "logical end of line" for Windows. Unicode gets around all of this by having U+0085 be "new line".
USA #46
What I don't understand about the Mac decision is that by only using \r, it is unclear what purpose \n is supposed to serve. At least in Unix, \n has a very clear purpose (end of line, start a new one) and \r has a very clear purpose (go to the beginning of the line). Windows does more or less the same thing. But in Mac, they've put \n functionality into \r, and left \n as useless, and as far as I know, there is no easy way to return to the beginning of the line.

In any case the real problem to me with the Mac's decision is the lack of integration; their GUI tools work in \r and the Unix environment under the hood works in \n.
USA #47
Hmm. Just going to throw one more thing in here. Under the old Apple IIs, the codes worked the same as the old machines. Clear screen, which on IBM systems worked by literally "clearing" the data, didn't exist on Apple IIs, so you used the "Form Feed" function. This worked much the same even *on* PCs if you where using ANSI based applications, like Telemate, to access a BBS. Form feed was "clear screen" and the ANSI command to clear the screen simply triggered a form feed, which dumped x lines to the screen, where x was 25 or 50, depending on the video mode. Later implementations broke this, and in the process created major confusion, since in something like a text box, form feed is "variable", so has to instead work as though it was clearing the data. This leads to the obvious confusion, such as a previous discussion here on text positioning functions and how they work, where no one is quite sure how the "clear screen" **should** work or how to handle that in a buffer.

In the case of telemate, it just inserted a form feed. As long as you didn't scroll "past" that "page" the output would just scroll normally. As soon as the form feed appeared in the text, designating a new page, and further scrolling up or down past the marker caused it to clear the output and redraw with the next page. Not knowing how to handle that causes serious issues in figuring out what the heck should be happening.

In any case. This is one of those cases where "technically" we are getting a conflict between what telnet, which was based on teletype systems in the first place, might be expected to do, and what compromises where made for things like storage and proper operation in file systems and document editors. And in this case we are running bang into a conflict because "technically" telnet *should* be handling the files as teletype, while the OS is mangling things into what the OS, file system and text editors expect to be dealing with. The two exist in literally different universes, and so apparently do all three OSes. lol
Amended on Wed 02 May 2007 08:07 PM by Shadowfyr
Australia Forum Administrator #48
Quote:

The combination of the two is the "logical end of line" for Windows.


Well, let's look at the keyboard for a moment. I take the phrase "logical end of line" to indicate "process the line I have typed" (or read from a file).

Now when I sit at a keyboard I don't hit two keys (carriage return, followed by linefeed) to make it process my command.

Interestingly, a small test indicates that when running under Linux (or Cygwin) pressing the Enter key is shown as being \n (linefeed), but under "Windows mode" the Enter key returns a carriage return. For example, in the MUSHclient source, you can see that in the code for the command window (sendvw.cpp) it detects VK_RETURN as the trigger for processing a "command" in the command window.

Then I tried to see what actually happens when you read a file in (eg. a .bat file read by the command processor).

I made 3 files using Ultra Edit in "hex editing" mode:

test1.bat : echo test1 \r\n echo test2 \r\n
test2.bat : echo test1 \n echo test2 \n
test3.bat : echo test1 \r echo test2 \r

Then using a command window I typed test1, test2, and test3 (to execute the bat files) to see which ones would echo the commands.

Both test1.bat and test2.bat worked, indicating that it is the \n on its own that terminates a "logical line" and the \r is really a filler, plus it is used to move the cursor to the start of the line under certain circumstances.

However test3 echoed "test1echo test2", which seemed to show that the command processing was triggered by the end-of-file condition, and indeed it did not even return the cursor to the start of the line.
Australia Forum Administrator #49
Just to make things perfectly clear, I have amended the documentation on this page:

http://www.gammon.com.au/scripts/doc.php?general=data_types


This makes it clear that strings cannot contain imbedded null characters.

This should make the documentation clearer, that in cases where you might be tempted to try to use hex 00 bytes (eg. in SendPkt) you will not be able to.
USA #50
Umm. .bat is handled differently than say looking at the same file in something like DOS edit, and it may differ if ANSI.sys is loaded, which it isn't by default. Like I said, its a mess how this stuff actually gets dealt with on PCs, and it only gets odder when jumping OSes. The "hardware" can never generate anything other than 0x000D for the main key and 0x??0D for the keypad version. Most applications just strip off the first null/control and return the 0x0D. Linux apparently internally converts 0x0D from the hardware into 0x0A (If I remember the right code for LF). The hardware though is "still" producing the carriage return instead.
USA #51
Erm. This is the output from 'xev', a program that prints out every event that goes to it:

KeyPress event, serial 30, synthetic NO, window 0x4400001,
    root 0x1a5, subw 0x0, time 250593019, (-925,536), root:(466,946),
    state 0x10, keycode 36 (keysym 0xff0d, Return), same_screen YES,
"   XLookupString gives 1 bytes: (0d) "
"   XmbLookupString gives 1 bytes: (0d) "
    XFilterEvent returns: False

KeyRelease event, serial 33, synthetic NO, window 0x4400001,
    root 0x1a5, subw 0x0, time 250593107, (-925,536), root:(466,946),
    state 0x10, keycode 36 (keysym 0xff0d, Return), same_screen YES,
"   XLookupString gives 1 bytes: (0d) "
    XFilterEvent returns: False

KeyPress event, serial 33, synthetic NO, window 0x4400001,
    root 0x1a5, subw 0x0, time 250593443, (-925,536), root:(466,946),
    state 0x10, keycode 108 (keysym 0xff8d, KP_Enter), same_screen YES,
"   XLookupString gives 1 bytes: (0d) "
"   XmbLookupString gives 1 bytes: (0d) "
    XFilterEvent returns: False

KeyRelease event, serial 33, synthetic NO, window 0x4400001,
    root 0x1a5, subw 0x0, time 250593539, (-925,536), root:(466,946),
    state 0x10, keycode 108 (keysym 0xff8d, KP_Enter), same_screen YES,
"   XLookupString gives 1 bytes: (0d) "
    XFilterEvent returns: False


As you can see, the first press/release pair is for the normal enter, keysym 0xff0d, and the second pair is for the keypad, keysym 0xff8d.

Not sure where your ideas about what Linux is doing come from. Nor am I sure why you think the hardware is necessarily producing both cr/lf. The hardware is just sending in what key is pressed, not what a typewriter would do...
Australia Forum Administrator #52
Quote:

... but isn't the usage of replace() to remove something -hoping- it won't destroy another part of the string rather crude?


In an attempt to partly redress this problem, version 4.07 will now save tabs as 	 and then convert them when reading back in, thus tabs inside variables (and the "send" fields of triggers / aliases / timers) will now be preserved.
USA #53
Quote:
As you can see, the first press/release pair is for the normal enter, keysym 0xff0d, and the second pair is for the keypad, keysym 0xff8d.

The two keys do show up differently though. One identifies as "Return," and the other as "KP_Enter." Even the two control keys and the two alt keys have different codes. For proof, old versions of firefox didn't like it when I used the right control key to switch tabs. I had to use right control-tab. It's completely up to the software to interpret the hardware as you said. Also, when using the live CD for Gentoo and several other Linux distros, you need to select your keyboard type, which caused no end of frustration for a friend of mine who accidentally picked a German keyboard.
Amended on Fri 04 May 2007 06:10 AM by Shaun Biggs
USA #54
Well, yes, they show up differently, but that's my point: they're different keys and it's just a keycode. Like you and I have said, it's not sending anything more than which key was pressed, the interpretation of which is up to the OS. I'm not sure where the claim that pressing return sends two hardware signals comes from.

That would, incidentally, mean that a Mac keyboard wouldn't work right on the PC, and a PC keyboard wouldn't work on a Mac either, and this is demonstrably false. It's kind of funny, actually, to see the "windows" key act as the "apple" key. :-)
USA #55
I didn't say they produce CR+LF, I said that, according to what Nick was saying, Linux, or the application he was using, was translating it into LF, the same way that Windows apps tend to translate it into CR+LF. My point is that the **hardware** will still generate the "same" code no matter what the software insists it turns into.

And, just to be clear **modern** keyboards between MAC and PC are equivalent. The older model of MACs had an unusual feature. While most software couldn't tell the difference between keypresses from multiple keyboards, you could *chain* them and if your application requested the unique ID as part of the key code, you could have up to like 32 (64?) keyboards, or something, each generating its own data, all identifiable as separate devices. In some ways the shift to PC style hardware undermined some of their own innovations. Not that a lot of people used them that way, but you could, in theory, chain any number of different input devices through the connector, not just keyboards. Sadly, I don't think anyone ever really figured out what the hell to do with that feature. lol
#56
Wikipedia's entry on Newlines (and specifically the section on History and Newlines-in-programming) is helpful.

Pressing the enter key generates a mechanical or electrical signal that is interpreted by the keyboard's controller and translated to PS/2 keycodes (this is important for keys that share part of their mask(?) like on the g15 the right-shift key is ignored if you are already holding the cursor-up and cursor-right keys down) which are then forwarded to the motherboard's BIOS and from there as an interrupt to software... not necessarily the OS.

"modern" Mac and PC keyboards are interoperable because they are both conformant to the PS/2 standard (mostly because of economies of scale). The Windows/Mac Command key being identified as equivalent /could/ be because they use the same keycode, but it is just as likely to be OS X recognizing the windows key in software.

(As a side note I can chain up to 127 G15s together... but unless you have software that can identify which keyboard the event came from they are treated identically by Windows.)

Applications have the option to handle keycodes themselves but most rely on the operating system for the low level stuff (for.ex on Linux, assuming stdin has something to do with a keyboard...), and that's where libc bites you. Files are binary data and newlines have a concrete representation on disk but in memory, provided it acts like a newline your implementation of libc decides how it is actually represented. So fread and fwrite do a conversion if the file was opened in text mode ('r' and 'w') instead of binary mode ('rb' and 'wb'). It's a nuisance. (Amongst other reasons it tends to make ftell useless.)
USA #57
Interesting.. I wasn't aware that you "could" tell keyboards apart under the PS/2 architecture or that they made any that could even be chained for that... Explains I supposed why I could never figure out why the Cue Cat gadget didn't generate keycodes in software without a driver (or modification of the Cue Cat/both??). In theory, if it was simply sending the same keycodes to the hardware, any software *should* have seen them, encoded with their screwy encryption or not.

Which reminds me, I need to figure out how to declaw mine at some point. One of these days I need a better system to catalog my books and DVDs, which scanning the barcode would make a lot easier. lol