[Home] [Downloads] [Search] [Help/forum]

Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to "verify" your details, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.
[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  Tips and tricks
. . -> [Subject]  PPI - A plugin communications script
PPI - A plugin communications script

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Pages: 1  2  3  4 5  6  7  8  9  10  

Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #45 on Tue 12 Jan 2010 05:54 AM (UTC)
Message
The self-referential serialization came for free by adding the serialization cache to properly cope with tables that appear multiple times. See:

t2 = {1, 2, 3}
t1 = {t2, t2}


Granted, this is extremely contrived, but your serialization routine (at least save_simple) treats both t2 appearances as separate tables, resulting in this:

{
  [1] = {
    [1] = 1,
    [2] = 2,
    [3] = 3,
    },
  [2] = {
    [1] = 1,
    [2] = 2,
    [3] = 3,
    },
  }


That's emphatically something I want to avoid. Like I said, self-referential came for free.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #46 on Tue 12 Jan 2010 05:56 AM (UTC)

Amended on Tue 12 Jan 2010 06:00 AM (UTC) by Twisol

Message
To clarify, I've been arguing against this in particular:

Quote:
Referring to other tables: 

t1 = { 1, 2, 3, 4, 5 }
t2 = { 6, 7, t1 }


In that example, t2 refers to another table (t1). Thus t2 can't be simply serialized because you also need to serialize t1.

[...]

However tables within tables are OK:

t1 = { 
  wolf = { hp = 42, mana = 0 },
  naga = { hp = 100, mana = 10 }
  }



EDIT: And just a couple posts ago, you said:
Nick Gammon said:
[...]or refer to other tables outside the main table, are unlikely to be required in any sort of reasonable API that plugins that might expose.


This is what I'm trying to drive at: there's no difference between a table 'referring' to a separate table, and a table 'containing' a separate table.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #47 on Tue 12 Jan 2010 03:49 PM (UTC)
Message
Nick Gammon said:
You may be solving a problem that doesn't exist, and indeed, that we should not encourage to come into existence.

This is an interesting point and one that should be given consideration.

The goal in these endeavors is not always to produce something that always behaves fully theoretically correctly, but that provides a good working environment for solving problems. This business of handling self-referencing may or may not be "too much", but in general I think it's worth thinking that designing an API/protocol involves making policy decisions, and not just solving technical problems. Solving some technical problem can end up being unhelpful in the end if it encourages (or even allows) confusing behavior that is not helpful to the overall policy.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #48 on Tue 12 Jan 2010 06:19 PM (UTC)

Amended on Tue 12 Jan 2010 06:22 PM (UTC) by Twisol

Message
My goal is/was to create an inter-plugin communications system that behaves as closely as possible to the real thing. With that in mind, I wanted to make sure that table references were not duplicated but preserved, so the end serialization would not be 'lossy', so to speak. Serializing self-referential tables, or even just tables that somehow happen to have themselves as a descendent somewhere along the line, came for free due to the implementation of the cache.

I also try very hard not to make assumptions about what the end user will do with it, but stick to my goal with the software instead. That is, if the end-user has some reason to want to pass a self-referential table, go for it. Theoretically, you might have a plugin that handles on-close serialization for other plugins, so it shouldn't care what the format of the tables are. Its job is just to do something with whatever it was sent. I don't think that's confusing.

There are two things I did decide not to implement: metatables and writing to the PPI. It's rather complex and, yes, not suitable for a protocol that's only really meant to be communicating data back and forth. (By the by, passing methods across is sort of important, as it allows both plugins to communicate in more interesting ways depending on the API of the plugins involved.)


EDIT: Just to make sure you understand, that on-close serialization plugin is probably not a very good idea, mainly because it would have to re-serialize the arguments it's passed. I just brought it up as a theoretical example of a plugin that doesn't require any specific data format.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #49 on Tue 12 Jan 2010 06:33 PM (UTC)
Message
Twisol said:
I also try very hard not to make assumptions about what the end user will do with it, but stick to my goal with the software instead.

As a general comment, the best APIs are the ones that do in fact make (extremely judicious) assumptions about what the user will be doing.

I haven't really thought this very particular issue through, but complexity for the sake of technical correctness is not necessarily a desirable outcome. It very well might be a good thing to allow self-referential tables, but then again maybe those tables are not the kind of thing that would/should be communicated, and allowing the communication would allow things to be passed around that probably shouldn't (e.g., _G).

In this case, it's not really that passing self-referential tables makes the library hard to use, it's that it allows behavior that one might not want to allow from a policy perspective. This is a very different question from the technical question of how one does it. Perhaps the distinction is subtle.

I don't mean, and I don't think Nick meant, this as a comment on this particular library or even this very particular point, but rather as a comment on general API design borne from experience in seeing too many "do it all" libraries. "Complexity" here does not mean "difficulty of understanding", but rather extra functionality that may be nice but not directly relevant to the problem being solved. Perfection being found in the end of removals, and all that.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #50 on Tue 12 Jan 2010 06:47 PM (UTC)

Amended on Tue 12 Jan 2010 06:54 PM (UTC) by Twisol

Message
David Haley said:
As a general comment, the best APIs are the ones that do in fact make (extremely judicious) assumptions about what the user will be doing.

Let's say that my assumption is that they want to transport tables that contain data, which may include other tables. A doubly-linked list sounds like something I'd want to allow, for example.

David Haley said:
[...] would allow things to be passed around that probably shouldn't (e.g., _G).

All I can say: blatant user error. There are plenty of things you shouldn't do with libraries, yet you can. Passing _G would be one of those very stupid things. Using (.*) in PCRE is very slow when you have to backtrack a lot. Using #include to include a .c file is generally not a good idea, unless it happens to deal with C++ templates (in which case it's one of the only easy ways to separate the declaration and definition into two files, IIRC). You -can- do these things, but in general you absolutely should not. Yet there are some cases where you need to. (There's never any excuse for passing _G, but my point is that it's almost impossible to stop it)


EDIT: To be pedantic, impossible to stop it without, say, making it hard/impossible to pass linked lists. Someone could carefully use setfenv() on the PPI to give it a fake _G, and then pass the real one, so comparisons would be futile.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #51 on Tue 12 Jan 2010 06:53 PM (UTC)
Message
Quote:
A doubly-linked list sounds like something I'd want to allow, for example.

Really? Shouldn't you be passing the data, not the implementation? In some languages, a straightforward list would be a vastly superior solution to a contraption from another language implementing a doubly-linked list. Even in Lua, a doubly-linked list is not very natural to work with; you would do so using all kinds of metatables. As a Lua programmer, getting a doubly-linked list instead of a plain old array would be rather annoying unless there's an awfully good reason for the list to be doubly-linked.

This is actually a pretty good example of something you might want to make as policy: data should be passed by its semantics, not by any implementation details.



That said, circular references are something that do come up. It's unclear to me if you actually want to communicate these circular references, as again they are usually implementation details. But, simple examples are tree or other container structures where children refer to their parents.

This is still of course communicating implementation rather than data. A preferable way of communicating a tree across language boundaries, IMO, would be using an API for constructing trees and passing it the parameters. This lets every language choose the most appropriate data structure for trees. (Perhaps the data structure used in one language is absolutely lousy in another compared to a built-in.)
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #52 on Tue 12 Jan 2010 07:12 PM (UTC)
Message
For posterity, a cleaned-up portion of a conversation David and I had on the IMC network:

MudBytes IMC said:
[ichat] DavidHaley@MW: Perhaps the problem is that the "context" of PPI isn't necessarily clear

[ichat] Twisol@Talon: Improving CallPlugin.

[ichat] DavidHaley@MW: OK. But does that mean transporting low-level data structures, or high-level semantics?

[ichat] Twisol@Talon: PPI is supposed to implement as sensible as possible of an interface for the user in whatever language it's ported to.

[ichat] Twisol@Talon: I don't really get the difference.

[ichat] DavidHaley@MW: Do you know Java?

[ichat] Twisol@Talon: Decently well.

[ichat] DavidHaley@MW: OK. So it's like the difference of communicating a List (high-level) vs. an ArrayList or LinkedList (implementation details)

[ichat] Twisol@Talon: I actually had the thought that PPI was as similar a protocol to Java's RNI than I could ever hope to write.

[ichat] Twisol@Talon: The problem is that the data needs to be sent across somehow, yes?

[ichat] DavidHaley@MW: of course

[ichat] Twisol@Talon: you can't think of it in terms of classes/interfaces. *shrug*

[ichat] DavidHaley@MW: Why not?

[ichat] Twisol@Talon: the data is data, and treatign it as a generic List requires morphing the dataset somehow to match your idea of a generic list

[ichat] DavidHaley@MW: I don't mean a literal class in the scripting language, by the way, but a concept that the protocol knows how to deal with.

[ichat] Twisol@Talon: Again, write your own protocol over PPI.

[ichat] Twisol@Talon: Hmm, lightbulb.

[ichat] Twisol@Talon: PPI is the lower-level transport protocol.

[ichat] DavidHaley@MW: Well, that's a good way of defining it

[ichat] Twisol@Talon: An idea that comes to mind is something akin to TCP/IP or UDP/IP. Build a protocol called WTF over PPI, and it would be WTF/PPI.

[ichat] DavidHaley@MW: That mission statement wasn't really clear earlier, IMO

[ichat] Twisol@Talon: No, you're right.

[ichat] Twisol@Talon: I knew what I wanted, I just didn't say it right.

[ichat] DavidHaley@MW: welcome to the world of why API design is so hard :D

[ichat] Twisol@Talon: whatever language the stuff reaches, it can build its own representation of it based on the format received

[ichat] Twisol@Talon: *nod*

[ichat] DavidHaley@MW: yes, it seems that way (again I'd have to poke at it a lot more to make a more definitive statement, but at first glance it seems pretty reasonable)
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Nick Gammon   Australia  (23,043 posts)  [Biography] bio   Forum Administrator
Date Reply #53 on Tue 12 Jan 2010 07:31 PM (UTC)
Message
I'm still not sure what problem the more complex PPI module solves, that needs to be solved, other than as an academic exercise. You should take a look at the Lua forums one day, there are many posts there about adding obscure features to Lua to solve obscure problems, many of which the authors are resisting.

Just as an example, maybe a metatable to detect data changing in a table, rather than a missing index or adding a new index.

However, every enhancement comes at a cost, and in the case of this extra metatable entry, which you could probably argue would be useful, would add an overhead to *every* Lua program, including ones which don't require the feature.

Twisol, you argued in an earlier page in this thread that you wanted a more language-neutral solution (than the Lua serialized table) for passing parameters, but later on you talk about "not serializing metatables". Well of course, metatables are specific to Lua, so even if you attempted to do so, you would then have a Lua-specific solution again.

Even tables, self-referential or not, may not exist in every language. From memory, VBscript actually only has arrays, not tables indexed by arbitrary keys.

Thus a truly language-neutral API would keep to very simple arguments (eg. numbers, strings, booleans) and possibly simple single-level tables indexed by a numeric index.

Even with my method of passing parameters, if you had to set them up in a different language (like, Python) it wouldn't be hard to fudge up something that looked exactly right to match the serialize.save_simple output. OK, we need a number, a string, and a boolean. So it would be:


{
  [1] = 42,
  [2] = "nick",
  [3] = true,
  }


It would be easy for any language to produce something like that (basically you have the argument number in brackets, an "=" sign, and the data after it).

The type of data is implied. If it is quoted, it is a string. If it is a number, it is numeric. The words "true", "false", and "nil" have specific meanings. And if the value starts with a "{" then you have a sub-table which is easy to set up as well.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #54 on Tue 12 Jan 2010 08:01 PM (UTC)

Amended on Tue 12 Jan 2010 08:06 PM (UTC) by Twisol

Message
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

b) I kept in mind the (lack of) functionality of other languages in designing the serialized data protocol. Keys can only be numbers and strings, values can be numbers, strings, booleans, methods, nil, or other tables/lists. The ports of PPI have a few choices when it comes to dealing with data it can't represent exactly how it came in: in the case of keys, defaulting to string should be acceptable. Or it could implement its own custom data model if possible, like David suggested: an API for explicitly adding items to be sent.

c) Methods are simply passed around by ID, so I believe any decent scripting language could write a shim like my new_thunk that eval()s and returns a thunk.

d) My serialization isn't actually that complex, either. Every MUSH variable contains a single 'table' or 'list' of entries. Your example table above would be:

n:1|n:42|n:2|s:nick|n:3|b:1


This is generated using ArrayCreate(), ArraySet(), and ArrayExport(), so the string manipulation isn't much of an issue either.

In the simple case, my serialization supports everything yours does. In the complex case, it supports a bit more, meaning languages that support those features can take advantage of them. In the simple case, every language could theoretically utilize the PPI, using z:~ to serialize values it can't serialize properly, and using nil/null/whatever to deserialize values it doesn't understand. More complex operations could necessitate the use of a more powerful language, but I don't see anything wrong with that. After all, if a language inherently doesn't allow you do to something you want to do...

e) You mentioned metatables and how I didn't choose to support them. The MUSHclient-supported languages that could support any form of metatables are very few AFAIK, and it would be hardly worth it at all when a proxy table could be used instead.


As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #55 on Tue 12 Jan 2010 08:12 PM (UTC)

Amended on Tue 12 Jan 2010 08:14 PM (UTC) by David Haley

Message
Twisol said:
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

I might have misunderstood, but isn't it being proposed as the standardized MUSHclient solution for communication between plugins?

It might be nice to frame the problem as precisely as possible. Having a method of communicating low-level information is useful. But it's also useful to have a method of communicating high-level information. The latter is (probably) a more common problem for scripting authors; I would posit that it's relatively rare to care about the data structures themselves.

This comment might be irrelevant depending on the precise problem being solved and how it will fit in to MUSHclient.

Twisol said:
As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.

It's not really so simple. The proxy table approach presents many complications of its own. (For starters, the public interface is no longer a table with keys and values, but an empty table with a metatable. Serializing such a thing is more difficult.) There are good reasons why one might want a data-change event, just as there are good reasons why one might not want it.
EDIT: for example, a very real problem with proxy tables is determining the set of keys that a proxy table has values for. This is very, very annoying to work around. IIRC, the Lua authors are adding a __keys metamethod to Lua 5.2 for this very reason.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #56 on Tue 12 Jan 2010 08:18 PM (UTC)
Message
David Haley said:

Twisol said:
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

I might have misunderstood, but isn't it being proposed as the standardized MUSHclient solution for communication between plugins?

I meant optional as in, if a plugin doesn't want to communicate with others, it doesn't have to include the library.

David Haley said:

It might be nice to frame the problem as precisely as possible. Having a method of communicating low-level information is useful. But it's also useful to have a method of communicating high-level information. The latter is (probably) a more common problem for scripting authors; I would posit that it's relatively rare to care about the data structures themselves.

This comment might be irrelevant depending on the precise problem being solved and how it will fit in to MUSHclient.

PPI is just a transport library. How the data is interpreted will be defined by the service exposing the methods. If a plugin wants to communicate with the service, it will need to follow the 'contract', so to speak, of the service's interface.

David Haley said:

Twisol said:
As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.

It's not really so simple. The proxy table approach presents many complications of its own. (For starters, the public interface is no longer a table with keys and values, but an empty table with a metatable. Serializing such a thing is more difficult.) There are good reasons why one might want a data-change event, just as there are good reasons why one might not want it.
EDIT: for example, a very real problem with proxy tables is determining the set of keys that a proxy table has values for. This is very, very annoying to work around. IIRC, the Lua authors are adding a __keys metamethod to Lua 5.2 for this very reason.

I'll give you that one. __keys does sound like it would solve those problems quite nicely.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #57 on Tue 12 Jan 2010 08:23 PM (UTC)
Message
At any rate, if everyone really believes PPI needs to be trimmed down, despite all of my reservations, I can do it. What in particular do you think should be changed, and what benefits will it bring to not include it?
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #58 on Tue 12 Jan 2010 08:35 PM (UTC)
Message
It's not that it needs to be trimmed down per se. It's that we need to (a) very clearly define the problem it's solving, (b) identify the problems that people need solved in practice, and (c) figure out how we can best map those two to each other.

The solution adopted in the end of the day (to the problem that we will eventually identify) might very well end up being something else that uses PPI, it's hard for me to tell at this early stage.

This isn't meant to sound patronizing in the slightest -- I hope it doesn't :( -- but one thing I would tell my students is to step away from implementation, and think about the higher-level problem being solved -- as it relates to their practical problem. Obviously we have a problem of getting data between plugins. But that problem statement is very general! It can mean anything from throwing around numbers and strings, to arbitrarily complex data structures, and maybe even a full RPC mechanism, heck why not with objects and classes too. What are the requirements being addressed? If we have callbacks, to what extent does one need to invoke them? Etc.
In other words: what are the use cases being targeted? (This is a different question from: What can we do with this?)
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Twisol   USA  (2,257 posts)  [Biography] bio
Date Reply #59 on Tue 12 Jan 2010 08:47 PM (UTC)

Amended on Tue 12 Jan 2010 08:53 PM (UTC) by Twisol

Message
My original goal was to replace LoadPPI with something decent, safe, and useful. The original use-case was calling to a service plugin (such as ATCP) to tell it something or have it do something. Even with LoadPPI though, the service plugin had to write CallPlugin() calls and come up with some way to pass the parameters onwards. PPI brings that into itself, freeing the service from having to deal with those low-level details.

Asynchronous callbacks, which you mentioned, are something I also want to use in my ATCP plugin. You simply register for a certain message, and when it arrives, it calls directly to your callback. Callbacks are just an extension of the general PPI idea, except instead of calling a method pulled from the loaded PPI interface, you're calling a separate thunk. It's the same internal workings, though.

An alternative is to scan all plugins for an arbitrary OnPluginAtcp function and call that directly, but that would require the client to also implement a service for the purposes of using OnPluginAtcp.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

304,239 views.

This is page 4, subject is 10 pages long:  [Previous page]  1  2  3  4 5  6  7  8  9  10  [Next page]

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Go to topic:           Search the forum

[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

[Home]

Written by Nick Gammon - 5K   profile for Nick Gammon on Stack Exchange, a network of free, community-driven Q&A sites   Marriage equality

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( https://gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Hosted at HostDash]