MUDs : General : Suggested protocol for server to client "out of band" messages

Gammon Software Solutions forum

See www.mushclient.com/spam for dealing with forum spam. Please read the MUSHclient FAQ!

Entire forum
MUDs
General
Suggested protocol for server to client "out of band" messages

Home \| Users \| Search \| FAQ
Username:	Register forum user name
Password:	Forgotten password?

Suggested protocol for server to client "out of band" messages

Reply to this subject Start a new subject Refresh page

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Posted by

Twisol USA (2,230 posts)

Date

Reply #30 on Wed 03 Feb 2010 10:27 PM (UTC) [ quote ]

Message

My only big, overarching problem here is that we seem to be building a protocol because we want to, instead of because it fills a need that's not already filled. We have plenty of handcrafted protocols out there already for exactly that reason, and I'm not so keen on seeing yet another niche protocol added to the mix.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by

Worstje Netherlands (867 posts)

Date

Reply #31 on Wed 03 Feb 2010 10:27 PM (UTC) [ quote ]

Message

Okay, more replies:

Message length:
The HTTP protocol has the Content-Length as an optional field if I recall. You can't trust on it. Also, in this case.. what if the length is shorter than what you encounter? Do you terminate it? How do you deal with the data that you get as a part of subnegotiation and doesn't belong? Ignore it? How about a longer length than the subnegotiation lasts? Skip the subnegotiation and go on good faith? It is ugly.

The memory issues are simple: don't worry about them as long as you allocate in sane chunks at a time. For one, there is virtual memory, and two, if they really send too much, their players will hate them for having the mud freeze on them. You are worrying about a problem that will only exist when you already have far worse problems to worry about.

IAC telnet issue:
Skip the <IAC> <unknown> in the middle of a subnegotiation. I'd throw an error into a notepad page like you do for some other questionable things concerning MXP entities and such. It is what I do.

My mud server source:
I am going to dig it up, extract the relevant bits and pastebin it for you so you can see my implementation since you are rewriting. I like to think it is very much optimized for what it does: handling telnet negotiations.

Posted by

Worstje Netherlands (867 posts)

Date

Reply #32 on Wed 03 Feb 2010 10:29 PM (UTC) [ quote ]

Message

@Twisol:

I disagree. We are not adding a protocol just for the hell of it. Lasher obviously sees a need for a Lua protocol, or he wouldn't be working with Nick as he is implementing it on Aardwolf. All we are doing right now is throwing in suggestions to not make it a half-assed spec that is doomed to fail like so many have (in my eyes, anyhow).

Posted by

Twisol USA (2,230 posts)

Date

Reply #33 on Wed 03 Feb 2010 10:30 PM (UTC) [ quote ]

Message

I assumed that this was Nick's idea, apologies. However, I still want to know what the other protocols lack, and why it can't be made to work with one of the others.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

Posted by Nick Gammon Australia (18,800 posts) bio Forum Administrator

Date Reply #34 on Wed 03 Feb 2010 10:32 PM (UTC) [ quote ]

Message

David Haley said:

My reading of the spec is that "IAC x" will be ignored unless 'x' is the code for a known command.

So my interpretation of

IAC SB c <some data> IAC x

is that we should keep on processing the subnegotiation, and pretend that we never received the 'IAC x' if 'x' is not a known command.

I'm not sure why 'x' couldn't be any of the several other defined commands, as given at the bottom of http://www.faqs.org/rfcs/rfc854.html. It might be weird, but I guess it's consistent.

Ah, so you are suggesting yet another approach? "pretend that we never received the 'IAC x'" - do you mean keep the negotiation but strip out the 'IAC x', or keep the negotiation and include 'IAC x' in it? (in which case it is like my suggestion of assuming IAC not followed by SE is in fact a single IAC.

I'm not sure about the "if it is a valid command" bit. Let's say we are dealing with some string which may be in German, or UTF-8 or something, where they might validly be "saying" IAC DO 0x46 (which might mean in German "my cat").

Unless you can show in the spec where it says IAC SB c <something> followed by IAC <not IAC> means definitely terminate the sequence and start a new one, with IAC <not IAC> I think this is just guessing.

I've looked at a few pages now, and since they say "including an IAC as part of the string it must be doubled", therefore an IAC not followed by SE is effectively invalid. However my question is, how to recover from that invalidness, short of shutting the client down saying "invalid protocol received, terminating session".

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

David Haley USA (3,881 posts)

bio Moderator

Date

Reply #35 on Wed 03 Feb 2010 10:33 PM (UTC) [ quote ]

Message

Quote:
Awesome, no more limit.

So what if a server starts subnegotiating, and then just sends an endless stream of random bytes? Either the client program will keep on allocating memory until it crashes due to running out of memory (and causing a lot of pain on the computer in the process), or it will have some kind of internal limit anyway in which case the problem hasn't really been solved.

You do have a point, though, that if the MUD server is being malicious, then there are more issues to worry about in the first place.

Quote:
First of all, it keeps mentioning commands, and I don't see anywhere about plain notifications that do not require a response.

Not sure what you're referring to here. Responses aren't required, in general.

Quote:
Last of all, it seems to have packages and parameters, but beyond that it is all quite unstructured.

I think that's kind of the idea. The idea is that you define your packages, commands and parameters on top of the data transport layer.

Quote:
Nothing like arrays/lists/dictionaries unless you go and hack around a bit. In essence, really simple commands can easily be implemented, once you have anything with a bit more structure you'll be hacking away outside of the spec.

Yes, this is an issue with the ZMP spec. It's worth noting that in many cases, lists can be passed as "the x^th argument onwards".

Quote:
And for Python and other languages, writing a simple parser would be quite elementary (syntax is simple, C parser is available freely also, as are other languages I'm quite sure).

This still doesn't really solve the problem of figuring out that it's not a dict but a list, for example.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

Posted by

Nick Gammon Australia (18,800 posts)

bio Forum Administrator

Date

Reply #36 on Wed 03 Feb 2010 10:37 PM (UTC) [ quote ]

Amended on Wed 03 Feb 2010 10:39 PM (UTC) by Nick Gammon

Message

Worstje said:

@Twisol:

I disagree. We are not adding a protocol just for the hell of it. Lasher obviously sees a need for a Lua protocol, or he wouldn't be working with Nick as he is implementing it on Aardwolf.

Actually Lasher wanted to use JSON. I note you said "JSON has flaws".

I am really throwing this out as a personal project since I tend to find that once you start negotiating a design document you spend months doing it, don't reach agreement, and nothing is done.

I am certainly willing to modify MUSHclient to improve telnet negotiation (and indeed, use your suggestion Worstje of handing off *all* subnegotiations to a plugin).

My project on page 1 of this thread, which will probably be improved after listening to everyone's comments, is intended to demonstrate a *working* system at both the client and server end.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

David Haley USA (3,881 posts)

bio Moderator

Date

Reply #37 on Wed 03 Feb 2010 10:37 PM (UTC) [ quote ]

Message

Quote:
Ah, so you are suggesting yet another approach? "pretend that we never received the 'IAC x'" - do you mean keep the negotiation but strip out the 'IAC x', or keep the negotiation and include 'IAC x' in it? (in which case it is like my suggestion of assuming IAC not followed by SE is in fact a single IAC.

My proposal was to keep everything and simply strip out 'IAC x', yes.

Quote:
I'm not sure about the "if it is a valid command" bit. Let's say we are dealing with some string which may be in German, or UTF-8 or something, where they might validly be "saying" IAC DO 0x46 (which might mean in German "my cat").

I'm not sure what you're referring to here to be honest. Are you saying that in UTF-8, you might be sending characters that happen to look like the beginning to a valid IAC+command sequence?

Quote:
I've looked at a few pages now, and since they say "including an IAC as part of the string it must be doubled", therefore an IAC not followed by SE is effectively invalid. However my question is, how to recover from that invalidness, short of shutting the client down saying "invalid protocol received, terminating session".

If push came to shove, I would probably terminate the subnegotiation and pretend that everything after that is normal output, and complain to the user (who would then presumably complain to the MUD author). You might get a lot of gibberish until the telnet subneg stopped, but at least after that you'd be in a sane state again.

If the MUD needs to send the character IAC (as part of UTF-8 or whatever) then they need to follow the normal rules of escaping it.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

Posted by Nick Gammon Australia (18,800 posts) bio Forum Administrator

Date

Reply #38 on Wed 03 Feb 2010 10:48 PM (UTC) [ quote ]

Amended on Wed 03 Feb 2010 10:49 PM (UTC) by Nick Gammon

Message

David Haley said:

I'm not sure what you're referring to here to be honest. Are you saying that in UTF-8, you might be sending characters that happen to look like the beginning to a valid IAC+command sequence?

Well, yes. But it doesn't have to be UTF-8. Someone might be sending a PNG image that is a map. Now given that they *should* convert IAC (0xFF) to IAC IAC the issue shouldn't arise. But what if they don't?

It seems to me, unless you can find a spec that says:


IAC SB c <data> IAC x

Where x it not IAC and not SE, is to be definitely treated as:


IAC SB c <data> IAC SE IAC x

... then we are just in the dark, and trying to work it out based on what x is, is just wishful thinking. If the sequence is undefined, then potentially any treatment of it is wrong.

Following your suggestion of dropping "IAC x" and pressing on, may potentially gives us our 1 Mb of data in the string variable, as the server really meant to terminate the negotiation, and is now sending down a lengthy room description.

However *not* dropping the negotiation might be just as bad, as the player may then see a screenfull of gibberish.

I think I would drop the negotiation, as seeing gibberish and complaining to the MUD admins is probably preferable to having the client "lock up" and having them complain to me. After all, the MUD admins in this case are to blame, not me.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

Worstje Netherlands (867 posts)

Date

Reply #39 on Wed 03 Feb 2010 10:52 PM (UTC) [ quote ]

Message

Malformed IACs in sub-negotiation:
To my knowledge, the telnet protocol works on a byte level. At that point, you have no worries about the used code page at all. As such, any multi-byte character sequences need to be consider binary data like anything else, thus any IACs as part of them need to be doubled. There is no room for confusion there, unless I am misinformed in which case I do appreciate a link setting me straight. :)

I vote for: 'ignore the malformed IAC and the byte following it, and keep processing afterwards'. It seems closest to the spec, which to my knowledge never says malformed IACs can be a reason for bailing on the sub-negotiation. However, I do see the point with people worrying about an invalid state forever clogging the mud. But you already have a World Configuration->Info window which has information about MCCP, MXP and so forth. It should likely always be at the same state of doing nothing on a telnet level. If it were to show anything other than that, you can raise a little bell as to say 'stuff might be wrong here'. Or use another method to inform the user - there are plenty.

Mud server code: http://mushclient.pastebin.com/f469bb629
I think it has all the most important parts in it. If you want the subnegotiation parts as well, let me know.

Posted by

Worstje Netherlands (867 posts)

Date

Reply #40 on Wed 03 Feb 2010 10:56 PM (UTC) [ quote ]

Amended on Wed 03 Feb 2010 10:57 PM (UTC) by Worstje

Message

@ David Haley:

Sorry, missed your raised point about the endless stream of junk until just now. You'd have no other issue than you do right now, I believe. The client wouldn't lock up (you can send a ton of stuff down the pipeline and while MUSHclient might become a bit slower, it wouldn't totally lock up), at most the problem of running out of memory again at which point Windows will complain big time, likely increase the pagefile and throw a popup etc first.

Posted by Nick Gammon Australia (18,800 posts) bio Forum Administrator

Date Reply #41 on Wed 03 Feb 2010 11:00 PM (UTC) [ quote ]

Message

Worstje said:

It seems closest to the spec, which to my knowledge never says malformed IACs can be a reason for bailing on the sub-negotiation.

The spec seems to be silent on that point. Since it says an imbedded IAC *must* be doubled, a single one seems to be outside the spec.

Anyway, what is the justification for throwing away the IAC x?

Say I have:


IAC SB c 1 2 3 IAC 4 5 6 IAC SE

You are saying treat that as "1 2 3 5 6". Why not "1 2 3 4 5 6"? Or "1 2 3 IAC 4 5 6"?

Saying the spec does NOT say to NOT do something is not the same as saying the spec says to DO something. If the spec ignores a point, I could just as easily say the spec "never says" that malformed IACs are a reason to *continue* the subnegotiation.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

Worstje Netherlands (867 posts)

Date

Reply #42 on Wed 03 Feb 2010 11:08 PM (UTC) [ quote ]

Message

Fair point. :) I see the logic in dropping it alltogether out of a client perspective and urging MUD creators to fix their mistakes. In practice, it is likely the best course of action and I won't hold it against you if that is the path taken. It does make me scared about people beginning to use it as a de-facto termination though, and no more usage of the final IAC SE whatsoever. As such, I would recommend a Notepad error, or a script error, or a popup, or SOME kind of error to make it clear to the user that the mud is doing stuff wrong, rather than allow mud authors to live in a grey area of 'hey it works in MUSHclient, there are no errors, so XXX client is at fault'. In my experience, many mud authors are all about adding features, but rarely implement them correctly, and only implement them to the point it works on their own client.

My reason for having the opinion I have, though, is that (again, going on memory) the spec also doesn't mention IAC codes CAN'T happen inside a sub-negotiation sequence. I believe that in practice, most of the useful stuff ends up being done in sub-negotiation, but I think there's a few things that aren't handled in sub-negotiation too, although I could be mistaken.

Are you handling anything other than the IAC WILL/DO/WONT/DONT and IAC SB in the standard state? If you don't, I'll completely agree with you that keeping it is probably useless, since if you don't have such codes supported in MUSHclient after this many years, they probably aren't being used anymore at all.

Posted by Nick Gammon Australia (18,800 posts) bio Forum Administrator

Date Reply #43 on Wed 03 Feb 2010 11:08 PM (UTC) [ quote ]

Message

Worstje, I looked at your code and am gratified to see that you identified the "third" case:


    else    /* received IAC during subnegotiation */
    {
        if (*p == IAC)   /* double IAC == single IAC */
        {
            chain_appendchar(d->buffer_telnet, IAC);
            d->state_telnet = d->state_telnet - STATE_SB_IAC_DELTA;
        }
        else if (*p == SE)
        {
            /* Process subnegotiation */
            telnet_gotsubnegotiation(L, d,
                (d->state_telnet - STATE_SB_IAC_DELTA),
                d->buffer_telnet);
            chain_reset(d->buffer_telnet);
            d->state_telnet = STATE_PLAIN;
        }
        else
        {
            debugf("process_chunk: came across SB+IAC+%d", *p);
            d->state_telnet = d->state_telnet - STATE_SB_IAC_DELTA;
        }
    }

- Nick Gammon

www.gammon.com.au, www.mushclient.com

Posted by

Nick Gammon Australia (18,800 posts)

bio Forum Administrator

Date

Reply #44 on Wed 03 Feb 2010 11:12 PM (UTC) [ quote ]

Message

Worstje said:

Are you handling anything other than the IAC WILL/DO/WONT/DONT and IAC SB in the standard state? If you don't, I'll completely agree with you that keeping it is probably useless, since if you don't have such codes supported in MUSHclient after this many years, they probably aren't being used anymore at all.

No, but the state machine is more complex than that because it also handles colour sequences (eg. ESC something), and MXP sequences (eg. <bold>).

Anyway, things have worked so far, I agree, so it is probably a bit academic. Interesting that there is no really well defined way of handling it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

61,014 views.

This is page 3, subject is 23 pages long: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Reply to this subject Start a new subject Refresh page

Go to topic: Search the forum

Comments to: Gammon Software support
Forum RSS feed ( http://www.gammon.com.au/rss/forum.xml )