Gammon Forum : MUSHclient : Tips and tricks : PPI - A plugin communications script

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to "verify" your details, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.

Entire forum

MUSHclient

Tips and tricks

PPI - A plugin communications script

PPI - A plugin communications script

It is now over 60 days since the last post. This thread is closed. Refresh page

Pages: 1 2 3 4 5 6 7 8 9 10

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #120 on Fri 22 Jan 2010 10:59 PM (UTC)

Amended on Fri 22 Jan 2010 11:05 PM (UTC) by Twisol

Message

Getting into a bad habit of missing peoples' posts... >_<

David Haley said:

Quote:
I go by feel and intuition, with a good helping of imaginative forethought/critical thinking. That probably says more than anything else about our disagreements.

I'm going to assume that you didn't mean I don't use forethought or critical thinking. :-P

*laughs* No, I didn't mean that at all! I meant that I think that the way I think is tempered by different drives than the way you think, in particular feel and intuition. It seems like you prefer to think at a very logical level, fleshing everything out. I could be wrong, but that's the impression I'm getting?

David Haley said:
I don't mean any offense at all here, but I think it's hard to argue for core modules on a particular person's feelings; objective metrics are at least "fair". (And if you don't like the objective metrics I proposed, you are definitely encouraged to post others.) Surely you agree that in order to have any useful discussion, we must be basing our judgments on criteria we share. There's really no point in discussing the CS equivalent of vanilla vs. chocolate.

For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!

My main metric is whether it makes it easier on the user. You can do everything with CallPlugin or BroadcastPlugin, clearly, and indeed that's what PPI does at its core. It allows users to communicate and transport data between plugins.

Some metrics I think I tend to use, although implicitly (usually part of my feel/intuition).
1) Does it make complex operations easier to use?
2) Is it clear? Is it intuitive?
3) Does it open new possibilities? (Subjective, but important IMO)
4) Does the code itself look "good"? (Subjective/circumstantial, but if it doesn't look good then it's probably not clear)

David Haley said:

Quote:
My point is that I am also not so strict that the interface becomes some kind of neutered API. (not to imply yours is, I'm just saying.)

The best APIs are the ones that impose a very clear policy which restricts usage just enough that it's obvious how to do the very common things, and still possible to do rather unusual things. Flexibility always comes at a cost (if anything, it becomes more confusing to do things if you have to figure out which of several options is appropriate).

See, everything goes through exactly the same interface. There's no "multiple ways" with PPI, unless the user him/herself has multiple possible designs. There's always multiple routes to take no matter what you're writing. I think the great thing about my PPI is that I built it to just do the basics that I came up with at the time, and in the end it can do so much more, through exactly the same interface, and everything is still clear and makes sense semantically.

Again, you just have Load, Expose, and a handful of acceptable parameter/return types, and you can do so much right there. The procedure doesn't ever change just to pass/return/call different things.

David Haley said:

Quote:
Yes, I believe I have, and I've given several examples of how other languages (in particular VBScript) can easily implement a PPI interface.

But VBScript has no notion of function values, so all the arguments made about things being "natural" in Lua do not apply at all. The PPI interface would look dramatically different. This is a loss of consistency. It's not just that function calls look different in different languages; it's that the entire approach to a problem changes even though the claim is made that it's all nominally language-neutral.

But VBscript CAN store names and use them quite simply because of its inherent limitations, even though it still has to keep a list/dictionary (which I know VBscript has, I checked) to map between names and the protocol's function identifiers. I believe that you are still assuming that a PPI in any language has to look like a PPI in Lua. This is firmly not true, and I have never assumed that to be the case at any point in PPI's development.

David Haley said:
Mind you, I'm quite sympathetic to the idea that a crappy language like VBScript shouldn't drive down nicer languages. But, we're talking about cross-language development here, so we're a little restricted in how much we can ignore that crappiness.

So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease, and this includes things that Fadedparadox brought up during a discussion I had with him (which was itself a rather interesting take on the whole PPI ordeal). If VBscript can do it, I'm sure 99% of the other MUSH-supported languages can.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

Nick Gammon Australia (23,042 posts)

bio Forum Administrator

Date

Reply #121 on Fri 22 Jan 2010 11:23 PM (UTC)

Message

Twisol said:

For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!

No, I just wrote them and posted them as part of the install files.

I think I got pulled up after the event in the case of my serialization code, where some cases were not handled correctly (funny variables names, from memory).

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #122 on Sat 23 Jan 2010 06:34 AM (UTC)

Message

Twisol said:
I meant that I think that the way I think is tempered by different drives than the way you think, in particular feel and intuition. It seems like you prefer to think at a very logical level, fleshing everything out. I could be wrong, but that's the impression I'm getting?

I guess. I start with intuition, as I think everybody does. I do try to expand intuition such that it passes clear tests; of course, intuition eventually encompasses those tests implicitly. Occasionally something will just "smell funny" and it's only after some reconstruction that I can point to the exact problems with it. If anything, this whole discussion has been an interesting exercise in fleshing out thoughts to proper arguments. :)

When I go into code-review mode, especially for something that is being considered for a standard, it is true that I refuse to accept anything unless it is very clearly motivated. APIs are so easy to screw up and bloat that great care should be taken. There's a very good reason why Lua semantics are so clean compared to, say, PHP; niceties are added with extreme caution. I've said it before, but anything based on feeling or personal preference will fail to sway me; I need very clear examples of why the simpler solutions are inappropriate.

For example, when I rewrote the complicated meta-table requiring dialog code into something far simpler, you said it was no good because it was "limiting". Well, ok. But to justify that statement, you made arguments based on which particular method more appealed to your sense of naturalness. You also said that mine was "harder to use" than yours, although I admit to finding that claim rather dubious. Basically, I think you object to it because it's not as "cool" and "shiny" (not meant to be derogatory). I have no objection to coolness and shininess per se, except when it adds complexity, and surely you agree that your meta-table-requiring approach is necessarily more complex in terms of stuff happening (for instance, it requires meta-tables...). I think you have a tendency to design things to "requirements" that you state as necessary, but haven't often called into question -- or at least justified here -- some of those basic requirements.

A question worth asking yourself re: complexity isn't so much if you understand your own code now, but if you think that coming back to it in 18 months you'll find it just as easy to understand without putting time into re-understanding it.

Twisol said:
For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!

As Nick said, no, but I think they were also rather simpler in scope. I find it quite interesting that even the relatively simple serialization library required revisions down the road.

Twisol said:
My main metric is whether it makes it easier on the user.

This is a reasonable enough metric. But it's important to understand that it's not a one-dimensional question. If you're not careful with endless additions of syntactic sugar etc., you can end up with an implementation that's such a mess that fixing bugs and adding features becomes much harder than it needs to be.

Also, sometimes, having a toolkit with fifty different kinds of hammers for very specific variations of nails can be much harder to (learn to) use than a toolkit with a two or three hammers that cover the vast majority of cases satisfactorily.

Superficial usefulness (as in, usefulness to the user) can be countered by internal complexity; it's also extremely important to make sure that things aren't any more complex than they need to be.

I think that your other metrics are interesting but slightly misleading. An API isn't usually meant to "open possibilities", for example; it's meant to solve a problem, and the possibilities that are opened are those that come about from solving the problem easily. An API (of this sort) usually isn't a big open-ended project that ends up implementing a whole paradigm unto itself.

Twisol said:
See, everything goes through exactly the same interface. There's no "multiple ways" with PPI, unless the user him/herself has multiple possible designs.

I get the impression that you're only thinking about this from the implementor's angle, not the user. You've stated explicitly that you want all these ways of passing functions around. Yes, yes, internally PPI doesn't care if the function has a name or not. But now, users have to decide how to structure their code around all these different "possible designs".

Twisol said:
I think the great thing about my PPI is that I built it to just do the basics that I came up with at the time, and in the end it can do so much more, through exactly the same interface, and everything is still clear and makes sense semantically.

I would have thought that by now it was clear that what might be the most sensible thing semantically to you is not necessarily the most sensible thing semantically to others. Also, two words: "feature creep". Doing "so much more" is not always a good thing.

Twisol said:
I believe that you are still assuming that a PPI in any language has to look like a PPI in Lua. This is firmly not true, and I have never assumed that to be the case at any point in PPI's development.

"Has to"? No, of course not... "Should"? Actually, probably yes. Do not underestimate the value of consistency. I would say that you also should not underestimate simplicity, but I would feel like I'm repeating myself to an unsympathetic ear. ;-)

Twisol said:
So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease

Ah, well, that's kind of the rub right there. The more stuff you add, the more "elbow grease" you'll need for other languages.

If I had to choose a single takeaway for you from this entire discussion, it would be to realize that there isn't always one appropriate way of doing things; that there isn't necessarily a solution that is the best of all worlds; that there are always trade-offs. Adding features is never free.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #123 on Sat 23 Jan 2010 07:53 AM (UTC)

Amended on Sat 23 Jan 2010 08:00 AM (UTC) by Twisol

Message

David Haley said:
I guess. I start with intuition, as I think everybody does. I do try to expand intuition such that it passes clear tests; of course, intuition eventually encompasses those tests implicitly. Occasionally something will just "smell funny" and it's only after some reconstruction that I can point to the exact problems with it. If anything, this whole discussion has been an interesting exercise in fleshing out thoughts to proper arguments. :)

Heh, indeed. *grin*

David Haley said:
When I go into code-review mode, especially for something that is being considered for a standard, it is true that I refuse to accept anything unless it is very clearly motivated. APIs are so easy to screw up and bloat that great care should be taken. There's a very good reason why Lua semantics are so clean compared to, say, PHP; niceties are added with extreme caution. I've said it before, but anything based on feeling or personal preference will fail to sway me; I need very clear examples of why the simpler solutions are inappropriate.

I'm doing things similarly to how Lua's doing them. I'm honestly curious, isn't that good enough reason, considering this is like a language extension unto itself?

David Haley said:
For example, when I rewrote the complicated meta-table requiring dialog code into something far simpler, you said it was no good because it was "limiting". Well, ok. But to justify that statement, you made arguments based on which particular method more appealed to your sense of naturalness. You also said that mine was "harder to use" than yours, although I admit to finding that claim rather dubious. Basically, I think you object to it because it's not as "cool" and "shiny" (not meant to be derogatory). I have no objection to coolness and shininess per se, except when it adds complexity, and surely you agree that your meta-table-requiring approach is necessarily more complex in terms of stuff happening (for instance, it requires meta-tables...). I think you have a tendency to design things to "requirements" that you state as necessary, but haven't often called into question -- or at least justified here -- some of those basic requirements.

Heheh, I didn't say it was no good, I just think it was limiting in the particular case I gave. I also never said anything about implementing metatables, I pretty much dropped that example once you pointed it out.

David Haley said:
A question worth asking yourself re: complexity isn't so much if you understand your own code now, but if you think that coming back to it in 18 months you'll find it just as easy to understand without putting time into re-understanding it.

I haven't so much as peeked at the code since my last posted change; maybe I'll give it a few more days and check back. *grin*

David Haley said:
As Nick said, no, but I think they were also rather simpler in scope. I find it quite interesting that even the relatively simple serialization library required revisions down the road.

Fair enough. I think it would be reasonable to come to a compromise on what we think we need and what we can do without for now, release it, and see what the developers think. We can always fix things down the road, as with serialize.lua. Although it might be worth noting that, whatever the complexities of my version, I think it might be easier to add/remove from without breaking things when all's said and done (although I could be wrong - I've never particularly had to maintain a library before).

Two things I would like to see kept are the protocol, and relatedly my method of passing functions around. You should be able to solve the vast majority of problems that way, with enough wiggle room to not feel closed in by my design choices.

Oh, that reminds me - that's what I meant before, about catering to personal preference. I don't want to actually restrict the user, but instead give them some wiggle room, a comfort space.

Tree/graph serialization of tables could go either way, but I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion. As for whether or not they're useful, I'm using a singly-linked list in my ATCP plugin to maintain the callbacks, since it makes it easy to remove one during iteration if there's an error when it's called (if the source plugin was disabled/removed). A doubly-linked list is only a step away.

Back to the protocol itself, I want to keep it because I think it's very easy for any language to translate/deserialize, since it uses Array*() methods rather than a Lua-specific serialization technique.

David Haley said:

Twisol said:
My main metric is whether it makes it easier on the user.

Well, I personally think that my implementation is fairly open to changes. The protocol can easily be expanded by adding another set of conditionals to the serialization methods, for example. Everything's pretty clearly separated.

David Haley said:
Also, sometimes, having a toolkit with fifty different kinds of hammers for very specific variations of nails can be much harder to (learn to) use than a toolkit with a two or three hammers that cover the vast majority of cases satisfactorily.

I'm not understanding this, though. There's only a few multi-purpose hammers, and you use them just like the other hammers you might be used to. You can access arbitrary values and expose arbitrary values. And clearly, if the value happens to be a function, there's a way to invoke it. That's all I see. What specific hammers do you see, so I can understand your point of view?

David Haley said:
Superficial usefulness (as in, usefulness to the user) can be countered by internal complexity; it's also extremely important to make sure that things aren't any more complex than they need to be.

Well, I've stated several times that I think everything in my implementation is justified (and I've explained my views on why), and furthermore I do actually like the design when it comes right down to it.

David Haley said:
I think that your other metrics are interesting but slightly misleading. An API isn't usually meant to "open possibilities", for example; it's meant to solve a problem, and the possibilities that are opened are those that come about from solving the problem easily. An API (of this sort) usually isn't a big open-ended project that ends up implementing a whole paradigm unto itself.

Sorry, that's basically what I meant. The idea is like, you only have a certain arm length, so you can only reach so far. If you simplify a problem area, lending your own reach to the mix, then that problem becomes simpler, and new ideas might come to light because they're closer within reach.

In the end, techniques such as callback functions, which were previously available but too much work to bother with, are now within easy grasp through the same syntax that you're used to. Hence, new ideas can and will come to light that use callback functions.

David Haley said:
I get the impression that you're only thinking about this from the implementor's angle, not the user. You've stated explicitly that you want all these ways of passing functions around. Yes, yes, internally PPI doesn't care if the function has a name or not. But now, users have to decide how to structure their code around all these different "possible designs".

What I mean is, even in Lua you have a variety of ways to structure your code. PPI is no different, and it adapts readily to whatever techniques are available. PPI doesn't care if the function has a name or not because at the protocol level, it's not even a function, but a message ID akin to those sent by BroadcastPlugin. The Lua PPI simply maps them to proxy functions, which provides a deeper integration into the language's semantics.

David Haley said:

I never intended to do "so much more". It just happened to come out of the design I made. No, seriously - I never made that many big changes. The only game-changing additions I made were (1) allowing functions to be passed and returned (speaking from the Lua terminology; the protocol was simply expanded to include 'f:'), and (2) separating ACCESS and INVOKE into the two messages they are now, which I am very happy I did.

David Haley said:

I went into this believing that the PPI could and should be adapted to the semantics of whatever language it was ported to, to give a closer feel to what users experienced with that language are used to. This is a core design choice that you clearly don't agree with, but I'd prefer to work with something that feels like part of the language than something that feels shoehorned in to fit. I'm not trying to convince you, but this is one of the design decisions that I won't go back on.

I'm not unsympathetic... I understand what you mean, I just disagree very much on this point. I made a choice - one that I think is a good one - and I've stuck with it.

David Haley said:

Twisol said:
So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease

Ah, well, that's kind of the rub right there. The more stuff you add, the more "elbow grease" you'll need for other languages.

I'm not planning on adding anything else, now or ever. This is sufficient for everything I would need, and everything else I've considered would be far more complex than anything else I've added, or would absolutely kill the cross-language goal. Nobody says you can't use CallPlugin yourself, either.

It's not like this was a breeze to write in Lua. I did have to put in some elbow grease of my own here. The point is that PPI would have to be ported for every language that wants to use it, but PPI is the only part that needs to be re-implemented to understand the protocol. All existing PPI-using plugins don't have to change a thing, which is in stark contrast to my original LoadPPI, which used loadstring()() on a service plugin's 'Library' MUSHclient variable, requiring the service author to write the exposed methods for every supported language themselves.

David Haley said:
If I had to choose a single takeaway for you from this entire discussion, it would be to realize that there isn't always one appropriate way of doing things; that there isn't necessarily a solution that is the best of all worlds; that there are always trade-offs. Adding features is never free.

PPI has been an exercise in compromise, though it might not seem that way to anyone else. I did have to weigh everything I added, and consider whether it would have undesirable adverse effects. There are plenty of things I could have added, but did not, because of this metric. I refer you to the post when I said "this must be desirable, that must be desirable, or the alternative must not be desirable" - that's another good metric for me. I understand your points, and believe me, I take them to heart.

EDIT: I tend to speak of the protocol and the implementation in the same breath, without noting which is which. If it's ever unclear what context I'm speaking in, pester me and I'll try to me more precise!

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #124 on Sat 23 Jan 2010 08:02 AM (UTC)

Message

Incidentally, I have to wonder what on earth you would say to me if you reviewed the original LoadPPI code. I shudder to think! *laughs*

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by David Haley USA (3,881 posts) bio

Date Reply #125 on Sat 23 Jan 2010 05:49 PM (UTC)

Message

Quote:
I'm doing things similarly to how Lua's doing them. I'm honestly curious, isn't that good enough reason, considering this is like a language extension unto itself?

I don't really think this should be even considered a "language extension". At its most basic, it's a way of shuttling data between plugins. That is a much simpler mission statement than a full language extension that completely drops all barriers between plugins. I also think that it covers the vast majority of use cases, and certainly all the ones we've seen so far.

Quote:
I just think it was limiting in the particular case I gave.

How? Is it limiting in functionality or not as nifty?

Quote:
I also never said anything about implementing metatables, I pretty much dropped that example once you pointed it out.

Well, ok, but how will you implement something similar? If you dropped metatables, you can't do it your way with things like ok = dialog.getChild("button_ok"). But apparently the alternative I proposed is limiting. What is your ideal solution to this?

Quote:
Although it might be worth noting that, whatever the complexities of my version, I think it might be easier to add/remove from without breaking things when all's said and done (although I could be wrong - I've never particularly had to maintain a library before).

As a general statement, internals are very easy to change except insofar as they affect externals; changing the externals is generally annoying. We wouldn't want to have something released and then say "oh hey, the API is now completely different, yay".

Quote:
I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion.

I also have no doubt that somebody will want to serialize a counter function implemented with upvalues, pass it over to another plugin, and then expect the upvalues to be correctly shared across both plugins. But this isn't going to happen. :-) The point: there's lots of things that won't be possible, but the question to be asked is not "what have we limited", it's "what have we limited that actually matters".

For example: why would somebody want to send a doubly-linked list around? (And why is it important to send it as a doubly-linked list, and not just a sequence of elements?)

You already have local functions with upvalues (something I think is too complex to begin with) -- the counter example above is "only a step away". :-)

Quote:
Back to the protocol itself, I want to keep it because I think it's very easy for any language to translate/deserialize, since it uses Array*() methods rather than a Lua-specific serialization technique.

I don't think your causation is correct here (is it really the "Array" methods that make it so easy, or the text format, or...?) but yes, I agree that the serialization output should be easy to deal with. I am tempted to agree that the Lua-like serialization format might be a little annoying to deal with in VBScript, and having the type encodings makes it easy. Conversely I am tempted to believe that deserializing anything more complex than a simple map from atomic elements to atomic elements will be annoying in VBScript.

Quote:
You can access arbitrary values and expose arbitrary values. And clearly, if the value happens to be a function, there's a way to invoke it. That's all I see. What specific hammers do you see, so I can understand your point of view?

It's the "arbitrary values" part that means that people have to think more about how to interface between plugins. For example, they need to decide if they should be passing functions around as part of a table, as locals, etc. They can call functions in all kinds of ways. Your initial dialog example (the one with metatables) could have just as well implemented my version (whereas my proposal cannot have done the metatable version) -- so that's a good example of having to choose between two ways of designing something.

Quote:
In the end, techniques such as callback functions, which were previously available but too much work to bother with, are now within easy grasp through the same syntax that you're used to. Hence, new ideas can and will come to light that use callback functions.

These can be accomplished without an awful lot of the fanciness you have introduced, as my example showed.

Quote:
PPI doesn't care if the function has a name or not because at the protocol level, it's not even a function, but a message ID akin to those sent by BroadcastPlugin. The Lua PPI simply maps them to proxy functions, which provides a deeper integration into the language's semantics.

I'm not disagreeing that your PPI proposal implements more stuff. I'm just not convinced that stuff is worth the complexity at the moment. It sounds like you're trying to implement full, rich, object-oriented RPC; that seems like a much bigger scope than just shuffling data around.

Quote:
I'm not trying to convince you, but this is one of the design decisions that I won't go back on.

Well, be that as it may, but you sort of have to convince people that your approach is correct (where 'people' is a set containing at least Nick).

Quote:
I'm not planning on adding anything else, now or ever. This is sufficient for everything I would need

Frankly, I don't think it's realistic at all for you to think that you have predicted all future needs. Hence my insistence on designing to known requirements, not hypothetical requirements, because we don't know if those hypothetical requirements will ever become realized, and we certainly don't know if we've covered them all.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #126 on Sun 24 Jan 2010 01:07 AM (UTC)

Message

David Haley said:

Quote:
I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion.

Just checking in briefly, but I wanted to respond to this. What do you mean, "correctly shared across both plugins"? As far as I can understand you, they are shared correctly, because functions are not copied/executed within the client plugins, but the request is forwarded to the source plugin. It's not shared across, because there is no across for the function: it's only ever in exactly one place!

Now, if you mean you do want to copy a function to other plugins, and have it refer to that plugin's own copy of a variable, that's not hard either. You can base64-encode the string.dump() of a function, pass it over PPI as a string, and have the other side base64-decode and loadstring() it. However, that's entirely limited to Lua, and it's something the service designer would have to decide on anyways.

For the record, I'm working on a VBScript version of PPI. So far, the only troubles I'm having are dealing with the language itself. I hope to be able to post a finished version sometime soon.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #127 on Sun 24 Jan 2010 01:19 AM (UTC)

Message

I suppose that "correctly shared" wasn't the best term. "Shared as expected" might be a better term. In some cases you might expect the upvalues to be only in one space, in others you might expect the upvalues to be unique to each scripting space.

Incidentally, this is another inconsistency. Lua semantics dictate that tables are passed by reference, and yet you pass them by value. Lua semantics also dictate that tables are passed by reference, and this time you do pass them by reference. (I.e., you're sending the closure, not a copy of the closure. But you send copies of tables, not the table reference.)

The very idea of serializing string.dump results sounds like insanity to me, by the way. :P

Here's another cross-language problem you will have. When you send a table around, how are you going to know that a given table should be created as a list or a map in the target language? Lua is somewhat unique in its agnosticism between these two types. If I send a list to VBScript (or Python or Ruby or ...) I expect to be able to treat that as a list when I receive it in Lua. You will therefore have to do all kinds of tricky business to automagically guess if a table should be converted to a list or a full map, and any automagic business is dangerous.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #128 on Sun 24 Jan 2010 06:35 AM (UTC)

Amended on Sun 24 Jan 2010 06:38 AM (UTC) by Twisol

Message

David Haley said:
I suppose that "correctly shared" wasn't the best term. "Shared as expected" might be a better term. In some cases you might expect the upvalues to be only in one space, in others you might expect the upvalues to be unique to each scripting space.

Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

David Haley said:
Incidentally, this is another inconsistency. Lua semantics dictate that tables are passed by reference, and yet you pass them by value. Lua semantics also dictate that tables are passed by reference, and this time you do pass them by reference. (I.e., you're sending the closure, not a copy of the closure. But you send copies of tables, not the table reference.)

You're quite right. I'm just treating everything as copied 'values', and using referred-to functions (method IDs at the protocol level) for the communication. I could make it work by reference for tables, if you think I should. I could probably generalize the ACCESS message concept, from just being used by the PPI table, to being used by all passed tables. Hmm... I've been considering doing something similar within the VBScript port anyways. What do you think?

(EDIT: It might be worth noting that such a solution would make our tree/graph argument irrelevent, and also mean that there's only one MUSHclient variable used for any given INVOKE.)

David Haley said:
The very idea of serializing string.dump results sounds like insanity to me, by the way. :P

Absolutely. ;) I was just pointing out that it was possible!

David Haley said:
Here's another cross-language problem you will have. When you send a table around, how are you going to know that a given table should be created as a list or a map in the target language? Lua is somewhat unique in its agnosticism between these two types. If I send a list to VBScript (or Python or Ruby or ...) I expect to be able to treat that as a list when I receive it in Lua. You will therefore have to do all kinds of tricky business to automagically guess if a table should be converted to a list or a full map, and any automagic business is dangerous.

What I'm planning on doing in VBScript is keeping two sets of arrays, one for numeric keys and the other for string keys. I'll also provide a general Access() function that checks the type of its argument and gets from the corresponding array. It's theoretical at this point, but I think it will work.

Incidentally, I've read about a 'Scripting.Dictionary' object you can create in VBscript using CreateObject. Is that built into the engine? I haven't been able to tell for sure. If it is I'll probably use it in my port.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by David Haley USA (3,881 posts) bio

Date Reply #129 on Sun 24 Jan 2010 08:22 AM (UTC)

Message

Quote:
Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

Yes, we get consistency in one way at the cost of expectations not being met elsewhere. And all of this comes around in the first place by claiming that we do interesting, complex things.

Quote:
I could make it work by reference for tables, if you think I should. I could probably generalize the ACCESS message concept, from just being used by the PPI table, to being used by all passed tables. Hmm... I've been considering doing something similar within the VBScript port anyways. What do you think?

Honestly, I think it would be kind of disastrous. :-) If the theme of "simplify" hasn't come across enough from me, then I have been expressing myself very poorly indeed! :P

My point with the comment was simply that "consistency with Lua semantics" has been held up as an important requirement and yet it is already being violated.

Quote:
What I'm planning on doing in VBScript is keeping two sets of arrays, one for numeric keys and the other for string keys. I'll also provide a general Access() function that checks the type of its argument and gets from the corresponding array. It's theoretical at this point, but I think it will work.

I want a list back. I don't want anything else. Unless whatever proposal you have gives me a list, it's not good.

Python is easier to illustrate this with:

widgets = plugin.GetWidgetChildren("top_level_window")
for w in widgets:
  do_something_with_w()

If "widgets" is a map type, this won't do at all what you want. It must come back as something that obeys the list iteration protocol.

(Yes, if you have a map type, then widgets[1] will do what you want whether it's a list or a map. But iteration is another story.)

You could paint a similar picture in VBScript but I don't feel like looking up the syntax etc. If the end result is a Scripting.Dictionary, that's extremely cumbersome for the VBScript programmer if all they wanted was an array. Since many languages do not make any claims about the order of keys of a mapping type, you wouldn't even be able to necessarily iterate over the mapping in the same order that Lua would use.

Basically, the point is that Lua has a somewhat peculiar way of treating arrays and maps as the same thing, whereas other languages really care about the difference. I need to see how you plan on handling this translation between idioms. (My belief: it will be hard for you, and will require a lot of magic behind the scenes. My solution? Go back to that "high level semantics" stuff I was talking about where you pass something explicitly as a sequence, a mapping, or a primitive.)

Quote:
Incidentally, I've read about a 'Scripting.Dictionary' object you can create in VBscript using CreateObject. Is that built into the engine? I haven't been able to tell for sure. If it is I'll probably use it in my port.

Yes, it exists, but no, it's not in the core and you need to include it via the "reference" mechanism. I don't know what MUSHclient does with external library references for VBScript. In Outlook etc., you need to explicitly add the references in the VBScript editor.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #130 on Sun 24 Jan 2010 07:21 PM (UTC)

Amended on Sun 24 Jan 2010 07:48 PM (UTC) by Twisol

Message

David Haley said:

Quote:
Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

Yes, we get consistency in one way at the cost of expectations not being met elsewhere. And all of this comes around in the first place by claiming that we do interesting, complex things.

What expectations? If you want to have a single upvalue per client plugin, use a closure function returned from a factory function. Just like with vanilla Lua, the service designer decides what their API does. The client doesn't have to care in any meaningful way.

David Haley said:

I think it's the only real violation I've seen so far, and I'd like to think I have a decent reason for it: treating the table and its structure simply as data rather than another function-like communication mechanism. (Function-like because the table could return new values depending on when you access it.)

David Haley said:

I want a list back. I don't want anything else. Unless whatever proposal you have gives me a list, it's not good.

Python is easier to illustrate this with:

widgets = plugin.GetWidgetChildren("top_level_window")
for w in widgets:
  do_something_with_w()

If "widgets" is a map type, this won't do at all what you want. It must come back as something that obeys the list iteration protocol.

(Yes, if you have a map type, then widgets[1] will do what you want whether it's a list or a map. But iteration is another story.)

You could paint a similar picture in VBScript but I don't feel like looking up the syntax etc. If the end result is a Scripting.Dictionary, that's extremely cumbersome for the VBScript programmer if all they wanted was an array. Since many languages do not make any claims about the order of keys of a mapping type, you wouldn't even be able to necessarily iterate over the mapping in the same order that Lua would use.

Basically, the point is that Lua has a somewhat peculiar way of treating arrays and maps as the same thing, whereas other languages really care about the difference. I need to see how you plan on handling this translation between idioms. (My belief: it will be hard for you, and will require a lot of magic behind the scenes. My solution? Go back to that "high level semantics" stuff I was talking about where you pass something explicitly as a sequence, a mapping, or a primitive.)

Fair point on the list thing. Another solution I'm thinking of would be to create a key/value class, and use that for the entries. I could still use the same general API, but you could also treat it as a list and iterate over it manually.

In case you think it's strange for an access through the API to give the value, but a manual access over iteration to give the pair, C++ does do this with its std::map type.

(EDIT: I don't know why I thought it could be strange; Lua, C++, and pretty much every language I know of has this behavior on iteration! *scratch*)

David Haley said:

I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by David Haley USA (3,881 posts) bio

Date Reply #131 on Sun 24 Jan 2010 08:37 PM (UTC)

Message

Twisol said:
Another solution I'm thinking of would be to create a key/value class, and use that for the entries. I could still use the same general API, but you could also treat it as a list and iterate over it manually.

How would this work in Python?

Twisol said:
I have a decent reason for it: treating the table and its structure simply as data rather than another function-like communication mechanism.

I dunno, you already have tables with complex self-referential structures. That seems like more than just data to me.

And hey, you know how you were talking about the difference between referencing other tables, and having subtables? This is a great example of a difference:


my_object = { ... }
my_object.configuration = get_config() -- returns a table ref

Let's say that we send this over the wire to another plugin -- heck let's even say that the other plugin is Lua.

The configuration table will be copied.

Now the original plugin changes the global configuration for the "my_object" type.

Now, when the other plugin does stuff with "my_object", it has an outdated copy of the configuration!

(If you want a concrete example, consider things that belong to a widget, and that have a link to the widget they're contained in. You certainly don't want to copy that entire widget table. Not only is it very wasteful, it can also lead to simply incorrect and confusing behavior.)

The difference between owning subtables and referring to external tables is precisely this: when you own the subtables, it is clear that the structure is self-contained. When you refer to external tables, changing that reference is changing the semantics of the structure.
You are correct that from an implementation perspective, it's really the same thing. But from a design perspective, it's entirely different. Sometimes copying tables is fully appropriate. In other cases, it's completely wrong!

The point here: you have created the illusion of passing around stuff with full support for all kinds of things. You make a lot of effort to actually do this. But you invariably fail in some cases unless you do a lot more work in preserving references across boundaries here.

What has happened is that we have preserved (kindasorta) low-level details but completely forgotten what we were trying to do with these structures. It matters immensely that that widget referred to in the content item is a reference, and not a contained, owned subtable.

Funnily enough, a simpler protocol that makes the semantics more explicit, and tries less to preserve full implementation structure, will make this kind of thing easier. It also solves the list problem we are discussing.

Twisol said:
In case you think it's strange for an access through the API to give the value, but a manual access over iteration to give the pair, C++ does do this with its std::map type.

(EDIT: I don't know why I thought it could be strange; Lua, C++, and pretty much every language I know of has this behavior on iteration! *scratch*)

Iterating over a Python dictionary gives you the keys, although you can also iterate over the keys or values or "items" (pairs) explicitly. Iterating over a list should give you the values, not the indices.

In C++, you would most certainly not want to have to treat a list as a map from index to value.

So regarding this list stuff, we have seen that we've run into a problem you hadn't foreseen (despite thinking that you'd hammered this out and covered people's needs). This is how design works: it's extremely difficult to cover all hypothetical cases, and dangerous to think that you have. To address this whole in the complex-version-of-PPI, you will have to do a whole bunch of stuff, adding yet more complexity. And there's no guarantee that this won't happen again, or even that the solution will actually behave correctly. We've just rolled a snowball over a cliff and are watching it grow. By adding features, we have required the addition of even more features to make things "work". :-/

Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by Nick Gammon Australia (23,042 posts) bio Forum Administrator

Date Reply #132 on Thu 28 Jan 2010 11:18 PM (UTC)

Message

Twisol said:

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end

The thing that troubles me about this change is, that the caller and callee now have asymmetrical parameter lists.

As an example, with a simple function that adds two numbers:


function add (a, b)
  return a + b
end -- add

You call that like this:


print (add (2, 3)) --> 5

However by passing the id down as a first parameter the called function's argument list does not match how you call it, so you would have to code it:


function add (id, a, b)
  return a + b
end -- add

Now I think that just looks ugly. Especially when the whole idea, I thought, was to make calling functions from one plugin to another fairly transparent, so it looks like they are really in the same plugin.

And in my "add" example, it doesn't care which module wants the numbers added together.

One way around it is to provide a "caller" local variable, which the module can check if it wants to, eg.

  
  -- Call method, get return values
  local _caller = id
  local returns = {func(unpack(params))}
  
  -- Send returns
  send_params(returns)
end

Now func "sees" a variable _caller, which it can test if it really needs to know who called it, but otherwise it can ignore it, and now the parameter lists are symmetrical.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Nick Gammon Australia (23,042 posts)

bio Forum Administrator

Date

Reply #133 on Thu 28 Jan 2010 11:20 PM (UTC)

Message

David Haley said:

Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.

I don't think I did that, can't explain why it works. :)

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #134 on Thu 28 Jan 2010 11:26 PM (UTC)

Amended on Thu 28 Jan 2010 11:27 PM (UTC) by Twisol

Message

Nick Gammon said:

Twisol said:

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end

The thing that troubles me about this change is, that the caller and callee now have asymmetrical parameter lists.

As an example, with a simple function that adds two numbers:


function add (a, b)
  return a + b
end -- add

You call that like this:


print (add (2, 3)) --> 5

However by passing the id down as a first parameter the called function's argument list does not match how you call it, so you would have to code it:


function add (id, a, b)
  return a + b
end -- add

  
  -- Call method, get return values
  local _caller = id
  local returns = {func(unpack(params))}
  
  -- Send returns
  send_params(returns)
end

Now func "sees" a variable _caller, which it can test if it really needs to know who called it, but otherwise it can ignore it, and now the parameter lists are symmetrical.

You have a point, however I'm not sure I see how func() can access _caller, given that it's a local in the method that called it. What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?

Nick Gammon said:

David Haley said:

Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.

I don't think I did that, can't explain why it works. :)

I compiled the version I'm running myself, so if it doesn't work for anyone else, maybe Visual Studio 2005 did something and I didn't realize it.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

304,233 views.

This is page 9, subject is 10 pages long: 1 2 3 4 5 6 7 8 9 10

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.