[Home] [Downloads] [Search] [Help/forum]

Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to "verify" your details, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.
[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  Tips and tricks
. . -> [Subject]  PPI - A plugin communications script
PPI - A plugin communications script

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Pages: 1  2  3  4  5  6 7  8  9  10  

Posted by Nick Gammon   Australia  (23,043 posts)  [Biography] bio   Forum Administrator
Date Reply #75 on Tue 19 Jan 2010 07:33 PM (UTC)

Amended on Tue 19 Jan 2010 07:34 PM (UTC) by Nick Gammon

Message
Twisol said:

I was just trying to explain my version to Nick as per his questions.


Actually I said:

Nick Gammon said:

A programming exercise is not really done until it is documented, otherwise people don't know what the code does, how to use it, and why they might want to.


I wasn't referring to myself as "people" (although I am a person, I admit). I meant "people" as in "other people", Peter the Programmer or Dorothy the Developer.

I don't think Peter or Dorothy would care exactly how you serialize, which is effectively what you described in the reply. I meant to describe:


  • What the module does in general
  • Why you might use it instead of CallPlugin
  • What features it provides to the plugin-writer
  • How to use those features
  • Give some realistic examples of use


There has been some talk about function callbacks. I don't believe my simpler version supports that, and until I see an example I'm not even certain what you are referring to.

Do you mean plugin A uses PPI to call function f in plugin B, passing it a function g (the function g being implemented in plugin A) which plugin B then "calls back" once or many times? Or does it mean something else?

If I have got that bit right (and in the absence of examples or documentation I'm not sure I have) then what is the lifetime of this callback? In other words, once f is called can g be called after f returns? Or only during the call to f? Does the callback use PPI for the called-back function call? If so does it recurse to do so? What provision is there for detecting loops (eg. f calls g which calls f, infinitely)?

Can a third plugin get involved? Eg. f calls g which calls h in a third plugin?

Twisol said:

I'm trying to explain PPI by answering your and Nick's criticisms and questions ...


I'm not trying to criticize here, just trying to arrive at the "best" end solution. Of course, the best solution depends a bit on defining the problem as David is trying to say, more than once.

Let me give you an analogy, let's say one of us has a sports car and the other a large truck.

Now the sports car:


  • Will go faster
  • Will be more comfortable inside
  • Can only carry small loads


The truck:


  • Will go slowly
  • Will be less comfortable inside
  • Can carry large loads


Now if I say the my sports car is faster than your truck, it is an observation rather than a criticism. And if the objective is to move a ton of bricks from place A to place B then the truck is the correct solution.

However if the objective is to take your girlfriend out to the movies, then the sports car would be the more correct solution.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #76 on Tue 19 Jan 2010 08:55 PM (UTC)

Amended on Tue 19 Jan 2010 09:00 PM (UTC) by Twisol

Message
I'm on my iPhone at the moment, so it's hard to write a decent reply. I'll post again later to make up for it.

However, I think my PPI is more of a Hummer (H2). Powerful, carries big loads, and is comfy inside to boot. Impresses the ladies (the ones who don't go crazy about mpg wastefulness, at least).

Quote:
Do you mean plugin A uses PPI to call function f in plugin B, passing it a function g (the function g being implemented in plugin A) which plugin B then "calls back" once or many times? Or does it mean something else?

If I have got that bit right (and in the absence of examples or documentation I'm not sure I have) then what is the lifetime of this callback? In other words, once f is called can g be called after f returns? Or only during the call to f? Does the callback use PPI for the called-back function call? If so does it recurse to do so? What provision is there for detecting loops (eg. f calls g which calls f, infinitely)?


It sounds like you've got it right. Yes, once a function is passed/received, it can be stored and executed at a later date. That's how my ATCP plugin works: clients register a callback against a specific event, and ATCP calls them when it occurs.

Callback do go through the PPI layer, just like any other REQUEST message. There is no facility to detect infinite loops, nor do I think it is feasible to add.
Quote:
Can a third plugin get involved? Eg. f calls g which calls h in a third plugin?

Theoretically, sure. The proxy method that the receiving side gets is still just a regular function.



David, I will respond when I'm home, but it's easier for the user to pass values because the receiving end can work directly with a function object rather than a name. In order to emulate this with Nick's version, you'd need to Expose your method and pass it's name, and both sides would have to agree that it's a function name and not a regular string. It also means any plugin can suddenly easily access the function you intended for just the one target, not to mention clutters the public PPI space.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #77 on Tue 19 Jan 2010 09:22 PM (UTC)
Message
Quote:
David, I will respond when I'm home, but it's easier for the user to pass values because the receiving end can work directly with a function object rather than a name.

Except that the other end might have no means of even conceiving of "function objects". A name is a cross-language-portable means of identifying a function, a function object is a much more language-specific concept.

Quote:
In order to emulate this with Nick's version, you'd need to Expose your method and pass it's name, and both sides would have to agree that it's a function name and not a regular string.

I don't see why it's a problem to assume that a string used as the function name is referring to a function and not a regular string.

In other words, if, as a user, I say:
"Call function F with parameters X, Y, Z..."
it seems that there is no "agreeing" to be done beyond the obvious that "F" is a function name.

As for callbacks, well, presumably you will do something like:

"Call function 'CoolPlugin.RegisterCallback' with parameter 'MyNiftyCallback'"

at which point there is also no "agreeing" to be done beyond the straightforward convention.

In other words: why do we need the most general possible method of passing around function values, when the use cases (given so far) for passing functions are actually quite simple?

Quote:
It also means any plugin can suddenly easily access the function you intended for just the one target, not to mention clutters the public PPI space.

It's more or less a "shared function" anyhow, so I'm not sure why this is a problem. If it's really a problem, security can be introduced by obscurity and the name can be obfuscated in some funky way.

If you need to give a function to just one plugin and no other, this can also be achieved by adding a parameter specifying the calling plugin, and you can then restrict access to symbols by looking at that parameter. I'm not convinced this is actually useful, but it's still simpler than all kinds of state-tracking.

I'm also not sure if this is a general statement of the general problem, or a statement made w.r.t. your current implementation. Are you saying that this necessarily clutters any possible implementation of cross-plugin communication, or that this is the case of your current implementation?
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #78 on Tue 19 Jan 2010 09:38 PM (UTC)
Message
Not home yet, battery low.


David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive. But I am only ever sending a name, an identifier. Another language need not have the same user interface my Lua PPI does; it need not turn this identifier back into a functional object. Lua PPI does this because it is intuitive and it makes sense.

I would continually like to point to my ATCP plugin, and a client Roomname plugin, both of which have been posted recently in another thread.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #79 on Wed 20 Jan 2010 02:19 AM (UTC)
Message
David Haley said:
Quote:
First and foremost, very, very few languages can use an arbitrary type as a key.

Not really. C++, Java, Perl, Lua, Python, Ruby, to name just a few, all allow tables as keys (with restrictions in some cases).

All I'm trying to say is that it seems somewhat arbitrary to argue for generality in one case and then to get rid of another generality by saying it's specific.

I mean, if we're worried about serializing complex tables with cyclical structures and all that, and especially if we're worrying about maintaining Lua semantics, then in fact it seems very reasonable to want to keep Lua semantics in this case. Otherwise, the API is behaving inconsistently: it's keeping the semantics here, but dropping them there -- and silently dropping them, too!

C++ and Java are not MUSHclient scripting languages. Lua is the one we're already discussing. That said, I didn't know that Perl, Python and Ruby allowed table keys. However, I still stand by my reasoning that there is no valid use case for tables as keys. I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.


David Haley said:
Quote:
I would honestly like to ask you what your own solution would be.

My honest response is that I don't really know because I do not feel that there has been an adequate presentation of what users of cross-plugin communication actually need. For example, I don't know if a low-level communication layer is what we want, or if we want a higher-level semantics transport layer. My first and strongest gut reaction is to avoid complexity unless it is clear that it is unavoidable. My second gut reaction is to separate the implementation from the concept.

We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

David Haley said:
There's been a lot of talk about implementation so far, but relatively little talk about the problems we're trying to solve. Examples:
1. Why do we want to send complex tables?
2. Why do we want to send function values (remember, from the user's perspective), and not function names?
3. What are we trying to send in the first place?
4. Why do we need plugins to talk to each other?

1. I assume you mean 'complex' in a more scientific meaning. I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted? Besides, I needed some form of tracking to make sure every table got its own variable. I gave my reasons for this separation-by-table previously.

2. It sounds like this question contradicts itself. We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface. Pass a function, get a function, it all feels like Lua. That's the point. A non-Lua PPI can wrap around the serialization protocol however it wants, and provide its own user interface. It's separation of concerns: the protocol is as language agnostic as I personally can understand it, and the implementation is as intuitive as possible, which means close coupling with the language.

3. I can't answer that properly without an actual use scenario. At a fundamental level, we're just sending lists of data. In the case of ATCP, the client registers intent with the service, and the service pushes data back to the client later. I'm sorry if I misunderstand the question. I just want PPI to be a general-use transportation library to allow scripts to interact more fluidly than raw CallPlugin.

4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force. He's using CallPlugin with all sorts of junk right now, which is driving me crazy. It's ugly and IMO hard to maintain (having seen the code). PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

David Haley said:
For technical questions, "because we can" is not an acceptable answer unless it adds no complexity to the end result.

I tried my best to avoid 'because we can' in my above answers. Thank you for the questions. :)

David Haley said:
I find it remarkably difficult, if not basically impossible, to design a system when you don't know exactly what you're designing to. What are the user's requirements? These are different from an implementation's requirements, which should be driven by the user's requirements in the first place.

I'd like to think my answer to question 4 helped clear this up. If not, please say so!



Nick, next reply will be to you.



Also, Avatar is amazing. Go see it.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #80 on Wed 20 Jan 2010 02:39 AM (UTC)

Amended on Wed 20 Jan 2010 02:41 AM (UTC) by Twisol

Message
Nick Gammon said:
Twisol said:

I'm trying to explain PPI by answering your and Nick's criticisms and questions ...

I'm not trying to criticize here, just trying to arrive at the "best" end solution. Of course, the best solution depends a bit on defining the problem as David is trying to say, more than once.

No, no, I understand. It's all constructive. Do excuse me if I sound frustrated sometimes... I am, but I don't mind. ;)

Nick Gammon said:
*What the module does in general
*Why you might use it instead of CallPlugin
*What features it provides to the plugin-writer
*How to use those features
*Give some realistic examples of use


PPI is a module intended to simplify design and implementation of complex systems. On the surface, it facilitates clear syntax and semantics native to Lua (the port we're discussing), allowing you to 'expose' arbitrary values to other plugins. A 'service' Expose()s these values, while a 'client' simply Load()s a PPI interface table and accesses these values as though they were actually in the table. Supported data types are: string, number, boolean, table, function, and nil. All data retrieved is copied, not referenced; modifications to the data will not be reflected at its source.

Function values may be executed just like any other function. Any of the aforementioned values can be passed, and any number of parameters may be sent. Again, all data passed (and returned) are copies of the original.

As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins. Where CallPlugin() only accepts one (string) parameter, to a given global method, PPI accepts any number of parameters, to a method that need not be global, only exposed (whether that be by Expose(), or by returning one from a PPI-invoked method). It also supports directly accessing values, rather than calling a method to retrieve it, which feels more native - the difference between CallPlugin("fe19016faddd8daa3f862627", "GetVersion", "") and foobar.version.

To expose a value, use PPI.Expose("identifier", value). The identifier will be used by other plugins to access the value.

To access a value, a client must first use foobar = PPI.Load("plugin's ID"), then access the returned table using foobar.identifier.

PPI is largely intended for use in complex multi-plugin systems, ranging from protected resources (ATCP, ZMP, or the prompt) to modular combat systems, and anywhere in-between. It also is very easy to understand as it uses native Lua syntax and semantics.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #81 on Wed 20 Jan 2010 04:05 AM (UTC)

Amended on Wed 20 Jan 2010 04:08 AM (UTC) by David Haley

Message
Quote:
David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive.

Again, I never said you were actually sending function objects. It doesn't matter what you're sending under the hood as long as the interface is that you pass a function object. For all intents and purposes, as far as the user is concerned, you are sending function objects -- and your API must deal with receiving function objects as parameters to be sent around.

Quote:
Lua is the one we're already discussing.

No it's not -- it's the language you happen to have implemented this in so far. But a constantly recurring them is language independence, and the ability to reimplement in any scripting language. Therefore Lua's particularities aren't really relevant.

Quote:
However, I still stand by my reasoning that there is no valid use case for tables as keys.

You've basically just asserted this, but when other people have stated that they don't really see valid use cases for other things, you've basically just asserted that they're wrong. This makes it hard to make forward progress. Clearly if people aren't convinced, then you need more to be more convincing or reevaluate your position. I'm not really sure how else to say it. (And this isn't meant to mean offense!)

Quote:
I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.

I don't see the point either, it's added complexity for extraordinarily infrequent utility. The point was only that there seems to have been a somewhat arbitrary line drawn when it comes to supporting this or that generality.

Quote:
We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

Complicated doesn't mean hard to understand (even though you have said yourself that some things are confusing, like that double-storage thing Nick discussed). It means that stuff is happening that doesn't necessarily need to happen.

And we've already seen that we're actually not all that language-agnostic anyhow.

Quote:
I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted?

I will not say that one would never, ever want to do this. But then again, I could say that about a very great deal of things: including tables as keys. My argument is not that you would never want to send complex tables; my argument is that you would want to so rarely that it does not justify increasing the general complexity to deal with this.

Quote:
We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface.

We're talking about a language-agnostic protocol, in which one plugin can call some arbitrary code in another plugin. The convention that all functions must be passed by their actual name is simple to understand and the most likely to be as portable as possible. Furthermore, if I'm calling out to another plugin, I don't call FunctionWithRandomIdentifierXYZ, I call a specific name. Passing callbacks (or any other function) by name (again, not random identifier) is consistent with calling functions by name.

Now, as a convenience, you could accept functions by value and use debug.getinfo to introspect a useful name, and complain if one cannot be found.

But this is a good example of the distinction to be found between the core of a design and conveniences, or syntactic sugar if you will, built on top of it.
This convenience requires an extremely simple type check followed by a debug.getinfo call, and entirely obviates all the complexity of generating and storing random function identifiers.

Quote:
I can't answer that properly without an actual use scenario.

This isn't meant to be sarcastic or facetious, but how can we possibly design a reasonable, practical API without having actual use scenarios? It's very hard to design things in a void, from theory alone.

It's possible that you're trying to be too general here, which is I think the point that Nick was making. Complexity is warranted if it's addressing real problems. Complexity is not warranted if it's solving problems simply because they can be solved (even if those problems are relatively easy to solve, because it's extraordinarily rare that you can make one change without impacting the rest of the system).

It's a funny fact of CS in general that often, solving more can actually provide less value. There's a very fine line between solving the right amount, and solving too much. After years of doing this, I still find this line hard to navigate. One thing I do know is that as people accumulate years, they tend to get more and more wary of complexity unless it's necessary.

Quote:
4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

These are generally good examples (although the prompt one sounds like you're doing all the stuff in one plugin, so I'm not sure where the cross-plugin communication comes in). To be clear, I also completely believe that being able to communicate between plugins is a Very Useful Thing. The point of the exercise was to discover the essence of what cross-plugin communication is really about. The ATCP example, for instance, only needs to shuffle around very simple data. Similarly for the prompt plugin. So far, we haven't really justified anything more complex than numbers, strings, and maybe basic key:value maps from numbers/strings to numbers/strings. We have perhaps also justified simple callbacks, so that the centralized handler of a shared resource can notify its (potentially multiple) listeners that new data is ready.

Quote:
Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force.

Why does stuff need to be separated into several plugins, as opposed to one plugin with several modules? If the other plugins cannot exist without the core, and the core is useless without the others, it's hard to justify the existence of several plugins. (The existence of several modules is however easily justifiable.)

Quote:
PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

It could also be that his code is just poorly designed, and could be refactored without bringing this considerable tool to bear.

We spoke earlier about encouraging behavior that we might not like. This could be an example. By allowing plugins to send completely arbitrary data, we make it easy (or at least easier) to break stuff into separate plugins when modules might be more appropriate. If the cross-plugin transport layer has clearly defined goals (of which arbitrary communication is not one) people need to think more. In other words, a clear and concise design can propagate upwards to some extent. It's not a good idea, IMO, to force a more-complex API simply because some users are lazy and don't feel like refactoring stuff.

You might argue that there are cases where it reallyreally makes sense to split this stuff into separate plugins and not modules, and where you reallyreally need all these advanced features (along with advanced implementation) -- this is what I'd like to see.


Perhaps some (non-exhaustive) specific questions:
- Why decide that some generality is desired, like complex tables, but other isn't, like more types as table keys? (Why not bools? What about (light)userdata? Where does it stop?)
- Why do we need to support arbitrary function objects at the implementation (not interface) level?
- Are we truly language-agnostic, or have we discovered that maybe we like some things about Lua in particular? If we want to be language-agnostic, how much Lua-specific stuff have we added? How much complexity has been added to other languages' implementations to deal with this stuff? What about things that other languages might like to do, such that new support must be added to Lua? (If your expertise is in Lua, and not Perl/Ruby/Python, how do you really know that you are language-agnostic? How do we know that, with this large system, we won't be adding even more maintenance work as we start expanding to other languages?)

(That second point speaks to the problem of interface vs. implementation: if your interface wants to be convenient that's fine, but that shouldn't necessarily affect the general implementation of the core protocol.)

Regarding the last point, this statement greatly worries me:
Quote:
As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins.

This sounds like it will make implementing PPI semantics in other languages very difficult if Lua's semantics don't map well to those other languages. A simple example is that you have no guarantee whatsoever that other languages can even conceive of passing around function values to be called. The PPI API and its very capabilities would then be remarkably different if you are working in Lua vs. VBScript, for instance.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #82 on Wed 20 Jan 2010 06:21 AM (UTC)

Amended on Wed 20 Jan 2010 06:44 AM (UTC) by Twisol

Message
David Haley said:
Quote:
David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive.

Again, I never said you were actually sending function objects. It doesn't matter what you're sending under the hood as long as the interface is that you pass a function object. For all intents and purposes, as far as the user is concerned, you are sending function objects -- and your API must deal with receiving function objects as parameters to be sent around.

The API does handle this, it just stores them in the same PPI table as any other value. The only special treatment they get is in relation to serialization, where they get their own identifier and are added to the table of callable methods.

David Haley said:
Quote:
Lua is the one we're already discussing.

No it's not -- it's the language you happen to have implemented this in so far. But a constantly recurring them is language independence, and the ability to reimplement in any scripting language. Therefore Lua's particularities aren't really relevant.

PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation. The protocol is the backbone of PPI, no matter what language you're working in. The implementation strives to expose the protocol data in as native a way as possible. It doesn't matter how it does it, it's specific to whatever language we're dealing with.

David Haley said:
Quote:
However, I still stand by my reasoning that there is no valid use case for tables as keys.

You've basically just asserted this, but when other people have stated that they don't really see valid use cases for other things, you've basically just asserted that they're wrong. This makes it hard to make forward progress. Clearly if people aren't convinced, then you need more to be more convincing or reevaluate your position. I'm not really sure how else to say it. (And this isn't meant to mean offense!)

I've tried to give use cases as best as I could. When it comes to serializing 'graphs', I'm not really understanding the opposition. It adds no complexity on its own, because most bits it would need have other reasons for being implemented. It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

David Haley said:
Quote:
I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.

I don't see the point either, it's added complexity for extraordinarily infrequent utility. The point was only that there seems to have been a somewhat arbitrary line drawn when it comes to supporting this or that generality.

The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

David Haley said:
Quote:
We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

Complicated doesn't mean hard to understand (even though you have said yourself that some things are confusing, like that double-storage thing Nick discussed). It means that stuff is happening that doesn't necessarily need to happen.

And we've already seen that we're actually not all that language-agnostic anyhow.

The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

I'm not sure what stuff is happening that doesn't really need to happen, though. Like I've said, everything is based around a core system of three messages, serialization, and saving exposed values so clients can reach them. Function paramters/returns use the same system as PPI-exposed functions, even.

David Haley said:
Quote:
I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted?

I will not say that one would never, ever want to do this. But then again, I could say that about a very great deal of things: including tables as keys. My argument is not that you would never want to send complex tables; my argument is that you would want to so rarely that it does not justify increasing the general complexity to deal with this.

It doesn't increase the complexity, though. I tried to write the serialization routines as simply as possible. Everything in them has its place. I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

David Haley said:
Quote:
We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface.

We're talking about a language-agnostic protocol, in which one plugin can call some arbitrary code in another plugin. The convention that all functions must be passed by their actual name is simple to understand and the most likely to be as portable as possible. Furthermore, if I'm calling out to another plugin, I don't call FunctionWithRandomIdentifierXYZ, I call a specific name. Passing callbacks (or any other function) by name (again, not random identifier) is consistent with calling functions by name.

You can only ever get at a function through the identifier passed from the ACCESS message (i.e. tbl.someMethod). If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice. The protocol's not supposed to be readable, it's supposed to be easy to parse and deserialize. Given an identifier - any identifier, be it name or number - you simply store the identifier somewhere and return a proxy of some kind. (This is assuming Lua-style semantics; a PPI in VBscript would undoubtedly have an entirely different user interface) The proxy later takes that same identifier and makes a REQUEST with it. That's all there is to it.

ACCESS and REQUEST (the latter of which should really be called INVOKE) have direct analogues in the __index and __call metamethods. One gets a value from a table, and the other executes a function. __index does not return a name, nor does it execute the function. It just returns something you can use to invoke that function, whether now or later. It's the same idea.

David Haley said:
Now, as a convenience, you could accept functions by value and use debug.getinfo to introspect a useful name, and complain if one cannot be found.

But this is a good example of the distinction to be found between the core of a design and conveniences, or syntactic sugar if you will, built on top of it.
This convenience requires an extremely simple type check followed by a debug.getinfo call, and entirely obviates all the complexity of generating and storing random function identifiers.

Why? This is needless complexity on its own; I'm doing pretty much exactly the same thing as __index and __call (see above).

David Haley said:
Quote:
I can't answer that properly without an actual use scenario.

This isn't meant to be sarcastic or facetious, but how can we possibly design a reasonable, practical API without having actual use scenarios? It's very hard to design things in a void, from theory alone.

No, heh, I meant that your question is hard to answer without the context of a specific scenario. It's your question that's in the void.

David Haley said:
It's possible that you're trying to be too general here, which is I think the point that Nick was making. Complexity is warranted if it's addressing real problems. Complexity is not warranted if it's solving problems simply because they can be solved (even if those problems are relatively easy to solve, because it's extraordinarily rare that you can make one change without impacting the rest of the system).

It's a funny fact of CS in general that often, solving more can actually provide less value. There's a very fine line between solving the right amount, and solving too much. After years of doing this, I still find this line hard to navigate. One thing I do know is that as people accumulate years, they tend to get more and more wary of complexity unless it's necessary.

See my previous answers. Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now.

I'd like to mention, by the way, that normally if someone says something is wrong with my code or design, I'm very reasonable about it. More often than not, I make lots of changes based on their feedback. I'm certainly not as defensive as I appear here! The problem, as I feel it, is that you're not understanding my reasoning in PPI (and I've put a lot more thought and reasoning into it than many of my previous projects, for what that's worth). I think you may have a point with the graphs, but I've also given my reasoning on why I think it's best that way. It's just frustrating because everything makes so much sense to me, and when I read your and Nick's posts, I (rhetorically) wonder if you've (a general 'you') read over the code, tried what I've explained, or seen my working examples (ATCP and roomname). I understand that Nick has, and that you've looked over PPI at least, but again... we're back to not understanding the logic in the code. (That's probably partially my fault.)

... Eh, no offense meant! I just really want to make sure you understand my point of view. Not trying to start a flame war or anything. =/

(EDIT: What I'm trying to say, I think, is: please don't think I'm throwing a tantrum or something because I'm not getting my way. That's absolutely not it. :( )

David Haley said:
Quote:
4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

These are generally good examples (although the prompt one sounds like you're doing all the stuff in one plugin, so I'm not sure where the cross-plugin communication comes in). To be clear, I also completely believe that being able to communicate between plugins is a Very Useful Thing. The point of the exercise was to discover the essence of what cross-plugin communication is really about. The ATCP example, for instance, only needs to shuffle around very simple data. Similarly for the prompt plugin. So far, we haven't really justified anything more complex than numbers, strings, and maybe basic key:value maps from numbers/strings to numbers/strings. We have perhaps also justified simple callbacks, so that the centralized handler of a shared resource can notify its (potentially multiple) listeners that new data is ready.

In relation to the prompt one, my concept is more about allowing other plugins to contribute bits of data to the prompt plugin, and allowing the user to pick and choose what data to display. Something along those lines, at least - it's one of those ideas that I never really got a chance to work with.

David Haley said:
Quote:
Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force.

Why does stuff need to be separated into several plugins, as opposed to one plugin with several modules? If the other plugins cannot exist without the core, and the core is useless without the others, it's hard to justify the existence of several plugins. (The existence of several modules is however easily justifiable.)

It's easier to unload or mix/match plugins than it is modules, IMO. The core, by the way, is by no means useless without the extra plugins. That's sort of the point. You have outer modules which represent functionality specific to responding to a certain class's afflictions/abilities. You disable/enable them depending on what classes you're fighting (yes, multiple, especially in raid situations), which can be important for anti-illusion mechanisms (illusions allow you to room-emote text, which can mess up your system if they illusion various attack/affliction messages).

It also helps if you want to support addons, i.e. plugins that can access and interact with the core. The popular Vadi System (a proxy system rather than a client-specific one) allows you to create separate DLLs and load them as addons.

David Haley said:
Quote:
PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

It could also be that his code is just poorly designed, and could be refactored without bringing this considerable tool to bear.

We spoke earlier about encouraging behavior that we might not like. This could be an example. By allowing plugins to send completely arbitrary data, we make it easy (or at least easier) to break stuff into separate plugins when modules might be more appropriate. If the cross-plugin transport layer has clearly defined goals (of which arbitrary communication is not one) people need to think more. In other words, a clear and concise design can propagate upwards to some extent. It's not a good idea, IMO, to force a more-complex API simply because some users are lazy and don't feel like refactoring stuff.

Quite possible, even perhaps probable. But see above for why I don't think it's a bad design.

David Haley said:
Perhaps some (non-exhaustive) specific questions:
- Why decide that some generality is desired, like complex tables, but other isn't, like more types as table keys? (Why not bools? What about (light)userdata? Where does it stop?)

Well, I could add bools. The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to. Booleans don't suffer this issue.

David Haley said:
- Why do we need to support arbitrary function objects at the implementation (not interface) level?

I'm not sure what you're saying. I always saw PPI as having two levels: protocol and implementation. The implementation is necessarily locked to a specific language. Other languages could take a completely different approach to implementing a PPI interface in order to give that particular language a more intuitive feel. As I have it, the implementation is tightly coupled with the interface for that very reason.

The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

David Haley said:
- Are we truly language-agnostic, or have we discovered that maybe we like some things about Lua in particular? If we want to be language-agnostic, how much Lua-specific stuff have we added? How much complexity has been added to other languages' implementations to deal with this stuff? What about things that other languages might like to do, such that new support must be added to Lua? (If your expertise is in Lua, and not Perl/Ruby/Python, how do you really know that you are language-agnostic? How do we know that, with this large system, we won't be adding even more maintenance work as we start expanding to other languages?)

Again, the implementation itself is necessarily locked to a language. The only part that you absolutely have to have is a compatible set of serialization routines, which act as the decoupling filter between implementation and protocol.

David Haley said:
(That second point speaks to the problem of interface vs. implementation: if your interface wants to be convenient that's fine, but that shouldn't necessarily affect the general implementation of the core protocol.)

See above. (I'm only being brief because I don't want you to think I'm ignoring parts of your post by not including them.)

David Haley said:
Regarding the last point, this statement greatly worries me:
Quote:
As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins.

This sounds like it will make implementing PPI semantics in other languages very difficult if Lua's semantics don't map well to those other languages. A simple example is that you have no guarantee whatsoever that other languages can even conceive of passing around function values to be called. The PPI API and its very capabilities would then be remarkably different if you are working in Lua vs. VBScript, for instance.

Wrong, as I see it. The serialization routines themselves can be written in pretty much any supported MUSHclient language, IIRC. The actual representation of the protocol data will necessarily be different depending on the language in use, and as such, the implementation/interface would necessarily be different. The API might be remarkably different given a different language, and likely the implementation too (in a language such as VBScript, for example), but the only important part is that they can understand the protocol.

I would write a Ruby port if I understood Ruby well enough, but I haven't used Ruby enough and probably would do a shoddy job of it.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Reply #83 on Wed 20 Jan 2010 08:03 AM (UTC)
Message
Good lord, you all are spammy. :)

Since I really am not willing to study the plugin to the amount of depth it is being discussed in, I will bring up another matter instead:

All pre-supplied plugins tend to be useless for experienced plugin writers and new users alike.

Why? Everything is in the same directory, there is no real glossary what each plugin does other than opening them one by one, and at least half of them are highly specific tasks showing how to do one thing that most people aren't interested in, or do not know they are interested in.

The few plugins that might be useful, unmodified and all, are lost in the trees that make up the forest. Mostly because while they are awesome, they lack examples and documentation to point the users in the right direction.

For examples, I do NOT see the practical use of your PPI script at all. Does it solve a problem? Yes. It is even a problem I already solved well enough for my own purposes in the past, which is a far far simpler method than yours and admittedly far less functional.

In a nutshell, what does it do? It sets up a more advanced interface for cross-plugin communication, expanding CallPlugin to the point where you can pass more complicated types on both sides.

I ask you: when is it useful? The way I see it, it has far more limitations than you would like to think. Most of the people I know that dabble with MUSH scripting have trouble understanding how to adjust a trigger in a plugin, nevermind comprehend inter-plugin communication or why they would need your script more-so than plain CallPlugin (if they can even find the latter).

I will make a bold statement and say that this script should not be included in MUSHclient. The only people that will use it are people that need it for a particular plugin that does make use of it, in which case it will always be distributed along because you have a newer version or a specific need for an older version, etc.

Everyone else will either roll their own without even considering to look for something to abstract things away, or they will consciously come looking for it on the forums. (The latter due to the high noise:useful ratio in the supplied examples with MUSHclient.)

So please, maintain a post on the forums, maybe have it stickied or something along those lines so the people who script plugins can find it, but don't include it in MUSHclient.
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #84 on Wed 20 Jan 2010 01:50 PM (UTC)
Message
Quote:
PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation.

Then you need to clearly define the difference between these and lay out exactly what the protocol is, so that we're all speaking the same language. You keep saying that things are language-agnostic and yet all these Lua-specific points keep creeping in. It makes it very hard to understand where the language-agnostic protocol is. How exactly is a language-independent function callback to be passed around?

Quote:
It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

It only adds to the user's burden if they actually care about the feature. You've made a very strong assumption that these complex tables are utterly necessary to plugin development. Can you give an example of real-world data that requires complex tables?

Quote:
The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

What is the criterion for "wanting" something?

Quote:
The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

Nothing, really, is confusing once you understand it. So this isn't really a justification for complexity.

Quote:
I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

How can removing code and features increase complexity?

Quote:
If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice.

It would only adds complexity because you're trying to shoehorn it into an already complex solution; my suggestion was that the simpler solution should be the solution.

By the way, the point here is not in any fashion to make things "look nice". That is irrelevant.

Quote:
ACCESS and REQUEST (the latter of which should really be called INVOKE)

Then call it invoke; this is still the design phase of the protocol, and we already have irreparable backwards-compatibility brokenness to work around? ;-)

Quote:
Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now. <etc.>

I don't disbelieve that you have thought this through. But I do think that you have (with the proof-in-the-pudding, as it were) not motivated your decisions clearly enough.

By the way, you mentioned looking through the code. The issue here isn't only complexity of code, it's complexity of the idea or interface.

Quote:
It's easier to unload or mix/match plugins than it is modules, IMO.

I don't know anything about this particular plugin nor frankly do I really care to delve into it. My point was just that this might be using plugins in a way they might not have originally been intended to be used. Or maybe it's a fine usage. Even so, assuming this is perfectly fine, you need to motivate the requirement for complex communication in this instance, as more than just simple data types.

Quote:
The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to.

This is a Lua-specific restriction. Python dictionaries-as-keys don't suffer from this problem. :-)
So why does Lua drive the language-agnostic protocol but not other languages?

Quote:
The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

This is true, except that the extra complexity of generating random function identifiers is only useful to languages that have functions as anything more than names in the global namespace. And now there is inconsistency, because Lua plugins will send out identifiers as random stuff, whereas plugins in, say, VBScript will send out identifiers as actual names.

Quote:
(I'm only being brief because I don't want you to think I'm ignoring parts of your post by not including them.)

That's fine; it's easier to just skip something to reduce clutter. If I think you ignored a point and didn't respond to it at all, I'll bring it up again. :P
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #85 on Wed 20 Jan 2010 04:34 PM (UTC)
Message
This isn't a plugin, Worstje, it's a library. I tend to completely agree with you when it comes to the pre-supplied plugins, to be honest. But libraries tend to be different. Everyone knows about 'tprint', for example. The 'serialize' module is also probably one of the more used. And 'addxml' definitely makes it a lot easier to dynamically add triggers, aliases, etc. All of them are documented and mentioned on these forums, too. When it comes to libraries - especially the most useful ones, which I would like to think PPI is - it's easier to include them so people don't have to make an extra download, whether you're a developer or a user.

Plugin developers tend to have to cater to those who "have trouble understanding how to adjust a trigger in a plugin" anyways, and I speak from experience! But one really, really good thing about PPI is that it allows, no, encourages cooperation and collaboration between plugin authors. For example, my ATCP plugin. It now uses PPI to make it much easier to communicate with the plugin, allowing the ATCP plugin to push data to only those who want that specific kind of data. It also exposes a Send method which you can use to send an ATCP message to the server. Do other developers need to know how ATCP works? No. But the simplicity of my Roomname plugin is testament to the ease of use PPI supports, without needing to know exactly how PPI or ATCP works.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #86 on Wed 20 Jan 2010 04:57 PM (UTC)
Message
David Haley said:

Quote:
PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation.

Then you need to clearly define the difference between these and lay out exactly what the protocol is, so that we're all speaking the same language. You keep saying that things are language-agnostic and yet all these Lua-specific points keep creeping in. It makes it very hard to understand where the language-agnostic protocol is. How exactly is a language-independent function callback to be passed around?


f:1. This is a serialized function, if you want to call it that. 'f' just denotes that this is a function-type parameter. '1' is the PPI identifier of the function. Any language can save that identifier somewhere, wrap a shim around it, whatever, and can send that identifier back at a later date with a REQUEST (aka INVOKE) message to execute the callback by that name.

For clarity: the protocol is the format of the serialized strings passed between plugins. The implementation is anything that occurs after deserialization or before serialization. The implementation itself is irrelevant to the protocol in this design.

David Haley said:
Quote:
It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

It only adds to the user's burden if they actually care about the feature. You've made a very strong assumption that these complex tables are utterly necessary to plugin development. Can you give an example of real-world data that requires complex tables?

Mmm... I don't think they're utterly necessary. I think the point I'm trying to make now is that it's more work and burden to limit this. As for a real-world example, I can't really say I have any. I know there are situations when you want it for ease of use, but I haven't done anything quite so complex in Lua yet. I'm just very worried/irked by having a subtable (and all its own subtables) suddenly duplicated. That would be a real surprise to the user, and a subtle one too.

David Haley said:
Quote:
The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

What is the criterion for "wanting" something?

If it makes sense, if it's clearly useful/desirable, and especially if the alternative fails one of the first previous tests. Table keys arguable fails both tests, IMO, though I can only say it doesn't make sense in the context of PPI for sure (I've explained previously why). Table graphs makes sense, although it might not be clearly useful, but the alternative makes absolutely no sense to me and is absolutely not useful or desirable.

David Haley said:
Quote:
The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

Nothing, really, is confusing once you understand it. So this isn't really a justification for complexity.

Implementation detail, not core design issue. I can change this anytime.

David Haley said:
Quote:
I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

How can removing code and features increase complexity?

It adds to the code because I need the cache for other reasons, which I have explained.

David Haley said:
Quote:
If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice.

It would only adds complexity because you're trying to shoehorn it into an already complex solution; my suggestion was that the simpler solution should be the solution.

By the way, the point here is not in any fashion to make things "look nice". That is irrelevant.

The simpler solution is more complex precisely because you're trying to shoehorn things into ideals. I need to re-read what your simpler solution entailed, but from my PoV it's more work than necessary.

David Haley said:
Quote:
ACCESS and REQUEST (the latter of which should really be called INVOKE)

Then call it invoke; this is still the design phase of the protocol, and we already have irreparable backwards-compatibility brokenness to work around? ;-)

Heh, okay. ;)

David Haley said:
Quote:
Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now. <etc.>

I don't disbelieve that you have thought this through. But I do think that you have (with the proof-in-the-pudding, as it were) not motivated your decisions clearly enough.

By the way, you mentioned looking through the code. The issue here isn't only complexity of code, it's complexity of the idea or interface.

The interface itself is as un-complex as I could make it. Load, Expose, and treat functions as normal functions. The code is also not that complex; the 'server' stuff is at the bottom (where _G is involved), the 'client' stuff is in the middle-ish, and the serialization routines are at the top.

David Haley said:
Quote:
It's easier to unload or mix/match plugins than it is modules, IMO.

I don't know anything about this particular plugin nor frankly do I really care to delve into it. My point was just that this might be using plugins in a way they might not have originally been intended to be used. Or maybe it's a fine usage. Even so, assuming this is perfectly fine, you need to motivate the requirement for complex communication in this instance, as more than just simple data types.

True, this is likely going to be largely just simple data. There is no denying the usability of functions for callbacks, though. And this is only one use case.

David Haley said:
Quote:
The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to.

This is a Lua-specific restriction. Python dictionaries-as-keys don't suffer from this problem. :-)
So why does Lua drive the language-agnostic protocol but not other languages?

Fair point. Because it's not standard or guaranteed that this feature be supported in any given language, or implemented the exact same way. It would be confusing.

David Haley said:
Quote:
The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

This is true, except that the extra complexity of generating random function identifiers is only useful to languages that have functions as anything more than names in the global namespace. And now there is inconsistency, because Lua plugins will send out identifiers as random stuff, whereas plugins in, say, VBScript will send out identifiers as actual names.

It's not random, though. You just maintain an integer and increment it every time you want a new identifier. Even VBScript is capable of this. I said it was possible that identifiers could be strings, but I never said we should mix and match these formats.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #87 on Wed 20 Jan 2010 06:13 PM (UTC)
Message
Quote:
f:1. This is a serialized function, if you want to call it that. 'f' just denotes that this is a function-type parameter. '1' is the PPI identifier of the function. Any language can save that identifier somewhere, wrap a shim around it, whatever, and can send that identifier back at a later date with a REQUEST (aka INVOKE) message to execute the callback by that name.

For clarity: the protocol is the format of the serialized strings passed between plugins. The implementation is anything that occurs after deserialization or before serialization. The implementation itself is irrelevant to the protocol in this design.

I guess that I'm not convinced that the implementation here is so irrelevant. If there is any notion of a "PPI Identifier" for a function, then the protocol must specify what such an identifier is and what one is expected to do with it.

You can completely decouple the "PPI Identifier" from the language's identifier, but now you have created the necessity for a translation layer in between the two. So yes, your implementation makes sense in the context of requiring the translation layer. My argument is that I don't really see the benefits of this in the first place; it's yet another layer meaning yet another thing to understand, maintain, potentially bug-fix over time, etc. I see no tangible advantages to this other than being able to send unnamed functions over the API. (If security is really an issue, the plugin can restrict access to certain symbols by having a list of symbols that should not be resolved over the PPI link.)

Quote:
Mmm... I don't think they're utterly necessary. I think the point I'm trying to make now is that it's more work and burden to limit this.

I don't really understand why it's more work to limit this. It's acceptable for the API to say that behavior is undefined in the case of cyclical tables.

Quote:
As for a real-world example, I can't really say I have any. I know there are situations when you want it for ease of use, but I haven't done anything quite so complex in Lua yet.

I must be missing something here :P You say that you don't have an example but you know of situations?
If I may say, the last bit is particularly telling: you haven't done something so complex in Lua yet, but you're already baking in potential for doing something even though you don't really know what you'll use it for yet?

I don't think implementation needs to be frozen for a protocol: it's perfectly fine to allow it to grow later on if complexity emerges that must be dealt with. But I don't think that the first version should try to be a kitchen sink: it should address the problems we know we have now.

Quote:
I'm just very worried/irked by having a subtable (and all its own subtables) suddenly duplicated. That would be a real surprise to the user, and a subtle one too.

That's assuming you don't say up-front in the API documentation that only simple tables are supported, not cyclical or repetitive tables.

Furthermore, do you have an example where:
(1) you would send a table with repeated subtables,
and furthermore,
(2) the identity relationship actually matters between these subtables?

I.e., why does this duplication really matter? (It might be sending more data than it technically has to, but so what?)

Quote:
If it makes sense, if it's clearly useful/desirable, and especially if the alternative fails one of the first previous tests. Table keys arguable fails both tests, IMO, though I can only say it doesn't make sense in the context of PPI for sure (I've explained previously why).

It seems perfectly reasonable to me, if we're already discussing complex structures like cyclical tables, to want to send a mapping from coordinate to value. The coordinate would be a <x,y> pair, i.e. a table. In fact, such mappings are very easy in Python and probably Ruby, where tables-as-key-lookups are based on logical equality, not pointer equality.

Quote:
Table graphs makes sense, although it might not be clearly useful, but the alternative makes absolutely no sense to me and is absolutely not useful or desirable.

I'm not sure what alternative you're speaking of. The alternative I propose is to just not support it.

Quote:
Implementation detail, not core design issue. I can change this anytime.

When you initially discussed this piece of code, you said something about it creating PPI private space and enforcing namespaces or something along those lines. It sounds like it actually had semantic value, not just a random implementation choice.

Quote:
I need to re-read what your simpler solution entailed, but from my PoV it's more work than necessary.

This is kind of an odd statement to me, because AFAICT it's less work in the absolute and therefore could not possibly be more work than necessary when compared to a more complex solution. My proposal is that the protocol defines that functions be based around by their actual name; that's just a string and nothing needs to be done other than accept the function name from the user. Even the "fancy" solution of using debug.getinfo is less work than creating and maintaining a mapping of function value to generated function identifier.

Quote:
There is no denying the usability of functions for callbacks, though.

Indeed, but nobody has denied that. :-) What has been discussed is the usefulness of supporting arbitrary function values, which is quite different.

Quote:
Fair point. Because it's not standard or guaranteed that this feature be supported in any given language, or implemented the exact same way. It would be confusing.

Hmmmmmmmm......... :-)

Quote:
It's not random, though. You just maintain an integer and increment it every time you want a new identifier.

"Arbitrary" would have been a better word than "random", unless you consider your RNG to be just adding one. The point is that the identifier is indeed arbitrary and unrelated to the function itself. The same function value could get different identifiers depending on the order in which functions are sent around.

Quote:
Even VBScript is capable of this.

VBScript can't create a map from function value to identifier, though. In fact, VBScript doesn't even have function values.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Nick Gammon   Australia  (23,043 posts)  [Biography] bio   Forum Administrator
Date Reply #88 on Wed 20 Jan 2010 08:03 PM (UTC)
Message
Worstje said:

All pre-supplied plugins tend to be useless for experienced plugin writers and new users alike.


I agree with your Worstje, probably some should be moved into an "examples" subdirectory, leaving the useful ones behind, with some better documentation.

It is interesting to look at the history of this, plugins were introduced in version 3.32 in June 2002, and Lua itself only introduced in version 3.52 in November 2004.

Thus it isn't particulary surprising that the original CallPlugin interface was not particularly friendly to Lua, as the design criteria at the time was to make a way of communicating between various plugins that was language-neutral.



At the risk of moving away from the topic a bit, I wonder whether what is being discussed here is really the main problem facing plugin developers (although quite possibly some sort of PPI could be part of the solution).

Having done some development for Aardwolf plugins I faced some interesting issues, which I will summarise and simplify here.

Say that there are two plugin developers (two different people) who want to make plugins:


  • Aard_stats.xml - shows a health bar, including what room you are in

  • Aard_minimap.xml - shows a map


Now, both plugins need to know what room you are in, and so Lasher helpfully provides this in the form of a tag, like this:


{roomname}The Aylorian Temple of Ivar


Now, normal players don't want to see this extra tag (by "normal" I mean ones not using either plugin) so there is another command added:


tag roomname on


The player (or the script) sends this tag to get the extra {roomname} line sent.

Now, both plugin writers need to have the tag turned on, so they make sure that their plugin turns it on. This isn't as easy as it looks, because you can get scenarios like this if you try to do it the moment the plugin is installed:


Welcome to Aardwolf MUD!
What be thy name, adventurer? tag roomname on
There is no player of the name "tag roomname on".


Assuming you work around all that, both plugins turn the roomname tag on (which is OK) and both omit the line "{roomname}The Aylorian Temple of Ivar" from output.

Now the fun starts. The player decides to disable the Aard_minimap.xml plugin because the map is getting in the way. The plugin realises it is being disabled and helpfully turns off the roomname tag, so it doesn't annoy the player:


tag roomname off


Uh oh! The other plugin still needs the tag, and now the Aard_stats.xml plugin stops working properly because it no longer gets the roomname tag.

One method I used to work around this was to make a third plugin (a "helper" plugin). This third plugin doesn't display any information (and thus the player is not particularly motivated to disable it). All it does is grab the {roomname} line, and make it available to other plugins. However this raises further issues:


  • How to send the data to other plugins … I used BroadcastPlugin because then I didn't need to know or care if any other plugins were installed, or wanted to know the room name. They were going to get it anyway! However because each BroadcastPlugin call identifies which plugin it came from, other plugins can either not even process the message (by not having an OnPluginBroadcast function), or if they do handle the message, check if it was from a plugin they recognised.

    This was reasonably simple and effective, and indeed the message being broadcast could be more complex than the pure inbuilt API provides for, by doing something like is done in the PPI modules, and serialising a more complex message (like, tables) into a string.

  • The system breaks if the helper plugin is not installed, or is disabled. I tried to work around this by having the dependant plugins check that the helper plugin was installed. It would be nice to automate this somehow.

  • If the helper plugin is installed, and then subsequently removed, the system may silently fail. The OnPluginListChanged callback, suggested by Twisol, may help detect this (although I don't think it detects plugins merely being disabled).

  • If all the "useful" plugins (that display stuff) are removed, and the helper plugin remains, then for efficiency purposes the helper plugin should also be disabled, to stop getting the MUD to keep asking for a roomname, which no-ones cares about. However there is no particularly easy way of handling that at present.


I can see that some form of PPI could be useful here (although I don't know that the more complex version addresses any issues the simpler one couldn't handle). For example, plugins could "register" with a helper plugin that they want something (like the room name), and when the room name arrives, all registered plugins could get it. Then when they are disabled or removed they could unregister themselves, and if no plugins were registered as wanting the room name, the helper plugin could stop requesting it.

Having said all that, the simple approach of using BroadcastPlugin is probably the easiest, and may only be slightly inefficient. Certainly some plugins may get a message they don't need, but at present Twisol's PPI script does multiple CallPlugin calls anyway, to implement a single transfer of information (the call, getting returned values, and cleaning up).

What I am presenting here is a real-life scenario showing why you may want multiple plugins, and why they might want to communicate with each other. Now the question is, to solve a real problem, as opposed to a hypothetical one (with duplicated tables etc.) would either PPI module help? Also is extra work needed (maybe adding dependency links to plugins)?
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #89 on Thu 21 Jan 2010 08:29 PM (UTC)
Message
Excellent example, Nick.

Nick Gammon said:
One method I used to work around this was to make a third plugin (a "helper" plugin). This third plugin doesn't display any information (and thus the player is not particularly motivated to disable it). All it does is grab the {roomname} line, and make it available to other plugins. However this raises further issues:

This is the approach my ATCP plugin takes as well. *nod*


Nick Gammon said:
*How to send the data to other plugins … I used BroadcastPlugin because then I didn't need to know or care if any other plugins were installed, or wanted to know the room name. They were going to get it anyway! However because each BroadcastPlugin call identifies which plugin it came from, other plugins can either not even process the message (by not having an OnPluginBroadcast function), or if they do handle the message, check if it was from a plugin they recognised.

This was reasonably simple and effective, and indeed the message being broadcast could be more complex than the pure inbuilt API provides for, by doing something like is done in the PPI modules, and serialising a more complex message (like, tables) into a string.

BroadcastPlugin does get annoying when you have multiple parameters to send, and I never liked having to check the sender's ID beforehand. It seems clearer to me if you can autometically direct these messages to specific method, too. PPI (both of ours') helps with all of these things, though since mine lets you pass functions as well, it's much easier to register asynchronous callbacks a la OnPluginBroadcast.

In simple cases like what you describe, it's less about the parameter types and more about the number of parameters for ease of use.

Nick Gammon said:
*The system breaks if the helper plugin is not installed, or is disabled. I tried to work around this by having the dependant plugins check that the helper plugin was installed. It would be nice to automate this somehow.

*nod* PPI (both of ours', I think) helps with this through the return values of PPI.Load, and OnPluginListChanged. Mine in particular (I can't remember if yours added this) stores a plugin nonce to prevent needless reloading of the PPI. If the PPI can't be loaded, you can check this by its return value and act accordingly (whether that be an error or a Note() or whatever).

Nick Gammon said:
*If the helper plugin is installed, and then subsequently removed, the system may silently fail. The OnPluginListChanged callback, suggested by Twisol, may help detect this (although I don't think it detects plugins merely being disabled).

I checked, and it does detect plugins being disabled, but not enabled. Rather odd.

Nick Gammon said:
*If all the "useful" plugins (that display stuff) are removed, and the helper plugin remains, then for efficiency purposes the helper plugin should also be disabled, to stop getting the MUD to keep asking for a roomname, which no-ones cares about. However there is no particularly easy way of handling that at present.

It depends on the helper plugin (which I prefer to call a utility plugin or resource plugin). In the case of ATCP, there's no way to disable messages after you're connected, so it just keeps stripping them out and checking to see if anyone registered for them again.

In any case, the plugin should not be disabled, or else it can't easily re-enable itself. Some part of its functionality should be disabled instead, to allow it to continue operating and re-enable things as circumstances dictate.

...Heh, I wrote this before I read your next paragraph, which pretty much said the same thing. Moving on...

Nick Gammon said:
Having said all that, the simple approach of using BroadcastPlugin is probably the easiest, and may only be slightly inefficient. Certainly some plugins may get a message they don't need, but at present Twisol's PPI script does multiple CallPlugin calls anyway, to implement a single transfer of information (the call, getting returned values, and cleaning up).

I agree, it is probably simplest, but it's also rather rigid. It's like... in C++ it's like having a lot of if statements, and having to recompile the source if you add a new object/flag, versus having a more fluid system where the object/flag is registered elsewhere, and the handler class doesn't need an if-else ladder to deal with newcomers.

I really butchered the comparison, but I hope you see what I mean. And my PPI is really not that slow anyways.

Nick Gammon said:
What I am presenting here is a real-life scenario showing why you may want multiple plugins, and why they might want to communicate with each other. Now the question is, to solve a real problem, as opposed to a hypothetical one (with duplicated tables etc.) would either PPI module help? Also is extra work needed (maybe adding dependency links to plugins)?


Both would help, but mine provides asynchronous callback functionality a la OnPluginBroadcast. The table thing is not a core issue, of course, but I'm pushing it so much because the alternative simply boggles me (duplicating tables), and could also cause infinite recursion on accident.

The separation of ACCESS and INVOKE (myppi.somevalue and somevalue()) allows attributes to be accessed more simply. It's mostly syntactic sugar, because you could emulate it almost precisely by exposing a function that returns the value, rather than the value itself, but personally I find it much clearer. I'd be willing to remove this if you want me to, but there wouldn't be much gain: I would still have to query the service at some point in order to get the requested method's PPI identifier, which is what ACCESS handles currently. And I need to manage PPI identifiers for exposed methods so that you can pass and return functions, which is key to the asynchronous callbacks technique.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

304,244 views.

This is page 6, subject is 10 pages long:  [Previous page]  1  2  3  4  5  6 7  8  9  10  [Next page]

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Go to topic:           Search the forum

[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

[Home]

Written by Nick Gammon - 5K   profile for Nick Gammon on Stack Exchange, a network of free, community-driven Q&A sites   Marriage equality

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( https://gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Hosted at HostDash]