Plugin dependencies

Is it possible to ensure that one plugin is loaded before another? I have an ATCP plugin that begins ATCP negotiation with the server upon connecting, and I have other plugins that might require a certain ATCP module. However, I need to notify the ATCP plugin of these modules -before- connecting, otherwise I can't enable them at all.

I considered having the dependent plugins notify the ATCP plugin during the plugin installation, but there's no way I know of to be sure that the ATCP plugin has loaded by the time the dependent plugin's OnPluginInstall function runs.

This wouldn't really be a problem, I believe, if the plugins were all loaded before any OnPluginInstall routines were run. In that case, the functions, variables, et cetera would have already been loaded into memory, even if the on-install handler hadn't been run yet. However, I did a quick test by adding Note()s into the plugin scripts, one in the global space and one in the install handler. I ctrl-clicked the two plugins in the plugin list, and ReInstalled them. When one plugin was loaded, its on-install handler was called, and only then did it move to the next plugin.

So, I guess my request is: how can I guarentee that a plugin has been loaded before another? I don't really want to deal with LoadPlugin(), because it's somewhat vague and requires a filename (which can vary in this case, because the plugins could be anywhere and it would be a pain for people who use the plugin to fix).

This is the key phrase. You don't care about the loading order, you just need to get things right when you connect.

I would let things load in any old order, but when the ATCP plugin detects a connect (ie. the OnPluginConnect function) it could do a BroadcastPlugin call which asks each installed plugin (they would all be installed by now) to let it know if it needs to be told something.

Each plugin could then do CallPlugin to tell the ATCP plugin that it needs to be notified.

Oh! That's a seriously good idea, thanks!

Coming back to this... same plugin, but a different scenario this time. I have some utility functions defined in another file, which are supposed to make it easier to work with ATCP, but I really want to have just one file with everything in it. So far I've tried a couple things, but nothing really works 100%. For example, if I put the code in a string, SetVariable it once the ATCP plugin is loaded, and GetPluginVariable and loadstring()() it in the others if they want to use them, it works... but only if the ATCP plugin was loaded before any of the others.

I can't rely on sending the string over a broadcast, either, because there are times when you might want to use the functions during installation. Having two separate files to do the job can be clunky, because the plugin creators using ATCP will have to supply the utility file with their plugin to be sure they have it (or direct them to where elsewhere to get it, which means the user has to go to two places). It's just a lot of hassle.

Would it be possibly to add a new XML tag <library> (or something) for plugins, where the script inside would be importable into another plugin? There's already an <include> tag that imports script from non-plugin files (see constants.lua) - and it also seems to have a 'plugin' attribute that loads the file in a separate code space altogether - so it would make sense if a 'library' attribute was added, which would load only the <library> script?

(A.D.D. note: this setup reminds me of objects, with private members (<script>) and public members (<library>)... well, the public members don't share the same script space, so it doesn't quite work, but it was nice while it lasted.)

I had some similar things I wanted to do with Aardwolf, but I just made that a separate file that you "required" in Lua. Is there any big problem in that? I don't fully understand the problem here.

For one reason or another, the lua include path is borked depending on whether it's used during MUSHclient initialization or normal execution (1). It has a workaround, but it requires around five lines of code for something that should be intuitive.

Another reason against putting them in different files is that the utilties should always be present when the main plugin is: they logically go together. If the utilities are missing, plugins that rely on them will break, and plugins that (for some unknown reason) go directly through CallPlugin (therefore also lacking the much more intuitive interface of a native Lua function that packs the parameters for you) will function fine, seemingly for no reason. Merging the files would solve this issue, and it makes sense: it's like having functionality in the main script section, and public utilities that adjust or modulate that functionality are right alongside, inseparable without effort.

Just one extra attribute and one new tag would make these kinds of things easier. I'm all for external .lua files in other cases, such as when those files aren't actually meant to be used elsewhere - a good example is the large ACS2 plugin for Achaea - but in cases like this, I think it makes sense.


(1) http://www.gammon.com.au/forum/bbshowpost.php?id=6316

I had a thought about this. Actually two, but I think my second one is probably better... lol When creating a plugin, via the menus, how about a "dependencies" option? Same, of course, in the plugin XML. In the load loop, when trying to load in plugins, as the XML is read you would get something like:

1. Parse Fred.lua. Dependencies 'ATCP.lua'.
2. Stop loading Fred.lua, mark as "load after ATCP.lua".
3. Load plugin ATCP.lua.
4. Load other plugins.
5. Check marked list and load Fred.lua.

If, in step 5, your list of "required" plugins that got loaded doesn't include ATCP.lua, you generate an error:

"One or more dependencies for Fred.lua is missing 'ATCP.lua'."

You could have nested dependencies, of course, but that can be handled by going through the lists, as part of the process, to see if your dependency was delayed by another dependency. Of course, you also need to check for dumb things, like someone setting A.lua dependent on B.lua, but B.lua as dependent on A.lua, but that is easy enough to check for, since you will have, in the delay list, something like "A.lua":needs"B.lua", and if you then load B.lua, it asks for A.lua, and well... not hard to work out you have a problem there. lol

Such a change wouldn't effect existing plugins, nor would it change how existing dependencies are handled. It *would* make later designs less... complicated, and give you a more automated way of saying, "Mushclient can't find something you need for this.", and apposed to, "Some plugin expected X to be loaded, but it has no idea why it isn't."

I sent Nick a patch to the MUSHclient source, actually, that addresses these plugin dependency issues. It just ensures that all plugins are parsed, THEN all plugins have their OnPluginInstall handler executed. Currently each plugin is parsed and OnPluginInstall'd at the same time, but with the patch, a plugin can just check IsPluginLoaded() from the OnPluginInstall handler.

It hasn't been approved yet (to my knowledge), but the solution is there. :)

Hmm. Come to think of it, might need something like a dependency tree. Some bozo is bound to try something like A needs B, needs C, needs A, and make the simpler system fail. lol So, you sort it out like a tree:



And then load starting at the deepest, and working out. This means making a "brief" parse, just to load dependencies and build the tree, but all "existing" plugins would land in the wilma level, with no dependencies, and just get loaded in, without regard for tree traversal (well, other than all being on the "first layer"). When adding a new dependency you just move what you need to, to the correct point, and if you can't place it, such as you can't put A->A, to fix A->B->C->A, you throw one error. And, if you can't "find" to do the initial parse, the plugin requested, you throw a different one (and, since you know you are "in" a nested level, you can even react to a failed load with an appropriate error(s), for the other dependent plugins.

Yeah. The only advantage mine has is that such dependencies get handled by the loader, where as just "making sure they all load first", means you still have to have the plugins themselves negotiate with each other over if they *are* loaded or not. Its a small difference, in principle, but coding your own handlers and errors, and such is "slightly" more complicated, and, as I said, its not *quite* so clear what actually went wrong, if you have several dependency layers. Well, not without a lot of complicated checks. But.. Which makes more sense:

B failed because C didn't load, so I am not loading A either.

or

A seemed to work, because it found B, but B generated an error about C being missing.

But, imagine that if you had 4-5 dependencies in that mess, and how you would either a) make sense of what happened, or b) get the plugins to "tell you" what happened in a sensible way.

But, yeah, your way works too. lol I kind of forgot you had added such a patch. The issue came up some place else, and I remembered there was a post about it some place, but not where, or what was done about it.

Well this isn't a plugin, it is a module. But I take your general point.

Twisol's patch is probably simpler because it lets you check dependencies after all the plugins are loaded. However I don't think it addresses the issue of actually loading them automatically, which your suggestion does.

Probably the sensible thing is to combine both suggestions. First you have a method of notifying what else you require, eg. other plugins or other modules like ATCP.lua.

Then the client can go on and load any other required plugins, adding more to the list of required plugins as it goes. At this stage it doesn't execute any code, it just parses the XML.

A failure could propagate back up the tree. For example:

A requires B. B requires C.

C isn't available. Thus B is marked as "can't load". Then since A requires B, A is also marked as "can't load".

I'm not sure exactly how you would recursively handle that. For example, if you checked A first, and B was there you might "pass" A without realizing that B was going to subsequently fail, if C isn't there.

You might start with the ones that have no dependencies (ie. C in this case) but what if someone does:

A requires B. B requires A.

Now we have no plugins that have no dependencies.

Anyway, assuming all this can be solved, then once every required plugin is loaded, then you can start doing Twisol's idea and *now* start doing the OnPluginInstall scripts.

Actually ATCP is a plugin, at least the two versions I've seen (one by Zero and the other adapted from his by me).

Theoretically it would be possible to use both my patch and a built-in dependency checker. One could be used for hard dependencies (which the plugin won't work without), and the other for soft dependencies (extra features and inter-compatability, for example).

Ah, OK, in that case it is ATCP.xml not ATCP.lua.

Well, as an example, when you step through the list you might have something like:



Traversal would work like this:

1. Look at A. A requires B, so is B in the list?
3. Look at B. B requires C, so is C in the list?
5. Look at C. C requires D, so is D in the list?
6. D is in the list, so C is OK, remove D from all parts of the tree.
7. Go back to C, which has no more dependencies, so remove it from the tree.
8. Go back to B, which has no dependencies (removed when checking C and D), so remove it from the tree.
9. Go back to A, which now also has no extant dependencies. Remove A from the tree.
10. No nodes are left, so everything succeeded.

Obviously, and step in the traversal where you go and look at the list of loaded plugins and "oops!" a dependency is missing, is going to fail every one of them that requires it. In this example, there are a lot of nested dependencies, but even if it was A>B>C>D>E, then it doesn't matter if you start with A and hop through to E, then back track, or you start with D. Load E, fixes D, which removes "both" from the chain under A, leaving you with A>B>C. Then you just finish off the next item in the list of plugins, and their dependencies. The more you check to see if they line up, the fewer other chains need to be checked.

Now, you are right, there "is" a problem with looped dependencies. And, I am not sure how you do that.. Best I could say is that you maybe mark the dependency some way, in both. If you had:

A requires B and D.
B requires A and C.

Then you can't know if B is valid for A, until D and C are checked. So, you check C, its OK, which marks B as "codependent", so once D is checked, A becomes valid, and so does B, and both can be removed.. But, that is a simple case.. there has got to be some way to handle that, but I am a bit stumped. It would be easier to deny such interdependence or deny "load all, then sort it out". lol

This is where a merger of our two ideas becomes genuinely useful. All plugins should be loaded before the dependencies are checked, so all you need to do when checking the dependency tree is check if that plugin is loaded. In fact, the dependency tree can just be a list of plugins, each with a single list of dependencies. You don't have to check the dependencies of dependencies, because either they'll be later in the list or they won't have been loaded anyways.

Something like that, anyways.

Hmm. Yeah. I might be over thinking it.

I personally don't think we need much more than the dependency load order fix. Anything more can be implemented in script, ideally as a standard package distributed with MUSHclient.