New feature - array handling for scripting

I would like to introduce a new feature that I have been working on for MUSHclient, in time for comment before it becomes official. :)

There have been some recent posts here about maintaining "lists of things", with the added problem of adding or deleting from the list.

You can do that to a certain extent in VBscript with arrays (dim blah (10)), and also by using Split and Join, however that gets fiddly if you want to just add something to the middle, or remove something from the middle.

I thought it would be nice to leverage off the container management functions provided in STL (Standard Template Library) and offer some similar things for MUSHclient scripts.

This is based around what I will call an "array", with the new script functions being Arrayxxxxxx where "xxxxxx" is some array-based function.

Arrays have the following properties:

• You can have any number of arrays
• They are stored in memory only, one set of arrays per world, and another set per plugin
• By using ArrayImport and ArrayExport you can easily copy arrays into a single string, for loading/saving into variables (eg. MUSHclient variables)
• Each array has a unique name (eg. spells, friends, foes)
• Each array consists of any number of key/value pairs
• You can randomly insert into, or delete from an array
• You can list all available arrays (names of arrays)
• You can list all the keys in a particular array
• There is no restriction on array names, or key names, except that they cannot be empty
• Array names, key names, and values have leading/trailing spaces stripped before they are stored
• The array names, and key names, are automatically sorted into alphabetic order.

I'll work through an example here to show the idea. The examples show the script commands in VBscript, followed by the output from the output window in italics.

First, we'll create an array called "spells" and put some values (for mana) into it ...



Now let's focus on a particular spell, "dispel magic". We'll see if we know about that one ...



And what value is returned for it? ...



Let's export it into a comma-delimited list ...



And put the same thing into a variable for saving with the world file ...



What spells do we know (names only)? ...



Let's delete one (farsight) ...



How many spells do we now know? ...

You can even have "arrays in arrays" by doing something like the following. In this case I want to demonstrate how you might store multiple things for a particular spell, and then look one of them up.
First, create a couple of arrays, one for all spells, and one for the current one ...



We'll set up "onespell" as a separate array (mana, wearoff message, hitvict message) by using those as keys for this array ...



Now we'll export those values into the spells array, choosing a delimiter that won't be likely to occur in SMAUG messages ...



Now we'll clear that spell and start another one ...



Now also export that one into the spells array, using a different key ...



We can check everything looks OK by exporting the whole spells array - we must choose a different delimiter, so that we don't get confused about the data per spell, and the delimiter betweeen spells ...


Note ArrayExport ("spells", "=")

dragonskin=hitvict~Your flesh changes to emulate the scaly skin of a dragon.~mana~45~wearoff~Your flesh sheds its draconian aspects.=ethereal funnel=hitvict~An aura surrounds you, channeling violent energies in your direction!~mana~75~wearoff~The ethereal funnel about you ceases to exist.


I put the major keys in bold to distinguish them from the minor keys (the major keys are the spell names, the minor keys are the attributes per spell).

We can use the new "debug arrays" debug facility to check what we have done ...


world.debug "arrays"

onespell
hitvict = An aura surrounds you, channeling violent energies in your direction!
mana = 75
wearoff = The ethereal funnel about you ceases to exist.
spells
dragonskin = hitvict~Your flesh changes to emulate the scaly skin of a dragon.~mana~45~wearoff~Your flesh sheds its draconian aspects.
ethereal funnel = hitvict~An aura surrounds you, channeling violent energies in your direction!~mana~75~wearoff~The ethereal funnel about you ceases to exist.
2 arrays.



Finally let's get some data back. We'll import the data for "ethereal funnel" into "onespell" ...



Now let's look up the hitvict message ...


Note "hitvict message for ethereal funnel is: " & _
ArrayGet ("onespell", "hitvict")

hitvict message for ethereal funnel is: An aura surrounds you, channeling violent energies in your direction!


So, my questions for anyone who has got this far into the thread are:

1. Do you think it is a good idea to insist that array names, and key names, are not empty? The intention was to save the confusion of having "un-named" items, however maybe that is unnecessarily restrictive.
   
2. Do you think it is a good idea to trim leading/trailing spaces from array names, and key names? The intention was to save confusion over things like an array called "spells" and a (very similar looking one) called "spells " (with the trailing space).
   
3. Do you think it is a good idea to trim leading/trailing spaces from key values (ie. the data stored in the array). The intention again is to save confusion, but it stops you storing things where you might actually want the leading/trailing spaces.
   

The online documentation for the new functions has now been updated, and can be viewed at Arrays Documentation.

Umm.. Split and Join are not really major issues. The only *real* problem with arrays as they exist are glitched in some language implimentations and stupid things that they left out.

Example:

*Things I would like to see arrays do*>
split - That can split to a dynamic array.
join - Most have this.
sort - Optional.
Add - Inserting an item, either at position X or just adding to a sorted list.
Delete - Same as Add but in reverse.

*VBScript*>
split - creates a static array. This is practically useless for some things. This is the one that annoys me a lot and makes coding a version of the last three below more complicated.
join - works fine.
sort - not available.
add - not available.
delete - not available.
Pop - (Theoretically) simple to emulate with Redim.
Push - Also (theoretically) simple to do Redim.

I say theoretically because the effects of redim seemed to have been a bit unpredictable for some people.

*JavaScript*>
Concat - Joins two arrays of the same type.
Pop - Treats array as a stack and returns then deletes the last item.
Push - Treats array as a stack and pushes a new value into the last place.
Reverse - Reverses the current order of the array.
Shift - Like Pop, but first item.
Slice - Returns a section of an array in a new array.
Sort - Sorts the contents.
Splice - Cuts out a section of the array, but can insert a replacement.
toLocalString - Converts date info to a string.
toString - Basically 'join'.
Unshift - Like Push, but inserts at the start of the array.
Length - Number of elements.

*PerlScript*>
Sort - Same as JScript.
Reverse - Same as JScript.
Pop - Same as JScript.
Push - Same as JScript.
Hashes - Key indexed array type. Like a mini database.
Count - number of elements.
Keys - Returns a new array containing keys from a Hash.
Values - Returns a new array containing values from a Hash.
Shift - Same as JScript.
Unshift - Same as JScript.
Delete - Remove something by key.
Exists - True is the key for this item exists.

*Python*>
This is a bit obscure, but since it is based off of containers or something of the sort, then it must support Delitem, Len, Iter, GetItem, SetItem. There is also an array.py module that probably impliments the rest of the stuff.

So, your idea duplicates what already exists in Perl's hash tables and Python's lists, including the use of keys almost exactly. However it can't create simple non-keyed arrays from what I can see, which is imho not useful. It also duplicates most stuff already in JScript, but it doesn't actually go far enough to replace most of them. It would really only be useful for us VBScript people, because while the language is imho probably the easiest to learn, the array implimentation it uses completely sucks.

That said:

1. Good idea for a hash table, bad idea for an array, which shouldn't have keys anyway.

2. Yes. If the user wants to intentionally duplucate a name they can do "Somename1" or even "Somename_".

3. No. The value may have been intentionally added with those spaces, data should come out the same as it goes in.

Whether it is really useful or an improvement over anything other that what VBScript does is more of an issue. Especially since your method to use more than one dimension is probably slower than using the script languages version. A simple library of functions that give VB the same Add, Delete, Push, Pop, Shift, Unshift, etc. features and you could simply 'include' within a plugin would make a lot more sense than duplicating something that already exists in every other script language.

Shadowfyr, congratulations! That post brought your number of posts on this forum up to the 1,000 mark. :)

Looks more like a watered down database than an array but the idea works :) The export functions actually almost identically mimic database exports too now that I've taken a 2nd look at em.

To answer the questions you posed however:
Yes, yes and yes.

Granted, I'm looking at it as a coder (and dba) at this point so I consider all of those features a good thing in designing anything that even loosely mimics a database structure and could very easily be extended to become a database structure.

Having an associative array seems perfect. To solve Shadowfyr's concern about arrays not having keys, there should be a mechanism to "fake" keys, e.g. just use the "next available index" as a key, where next available index is size (or size+1, depending on whether your indices start from 0 or from 1.)

So, it'll be a associate array, but instead of mapping string key to value, it maps "integer index" key to value.

Great stuff, Nick. :)


(Edit:)
Oh, and to answer your questions, yes, yes, yes - or, for the third option, could this be made a flag that is on by default? If the user wants to *not* trim spaces - which should be a fairly rare thing - they can just turn the flag off.
E.g.


I suppose you could have other array options (maybe for the questions 1-2 you asked), but can't think of any more off the top of my head in my tired state.

NONONO please don't mess with the data... What if you want to store formatted output? You can't delete leading or trailing spaces, however I agree with the other two proposals. I think that the programmer is responsible enough to see if he has spaces in his data, so I think it's an unnecessary restriction even in the names, but it will cause problems if you "trim" the data...

An array would not be the best way to capture formatted output in the first place. Capture it to a text file using a fixed width font and read the file if you need the info ingame. Or better yet, setup either a spreadsheet or full database depending on how much formatted data needs to be captured and stored.

It was just an example, what i'm trying to say is that you don't need to trim variables since the programmer should know what he is doing.

OK, thanks everyone. Starting from the top, and after a good night's sleep, I can think a bit more clearly ...



I think this is the nub of it. Let's assume if you are writing a plugin that you want it to be useable on most people's PCs, that means basically VBscript or Jscript, because the other languages are an optional download.

I'm not sure whether Jscript supports keyed arrays, VB certainly doesn't (seem to).

I am using the word "array" loosely here. It is really a variant of a "map" in STL terminology, however I thought that using the word "map" would be confusing in a MUD client, as people associate maps with automappers and such.

I got the idea from PHP (which this web site's dynamic content is written in), where you can have arrays with alpha indexes, like this:

person ['nick'] = 42;

or

person [2] = 55;

As Ksilyan says, it is really an associative array.

This is much more powerful than simple VB arrays, which are really "vectors" in the STL sense.

However based on the comments "it can't create simple non-keyed arrays from what I can see" I will make a change I was thinking of anyway, and that is to modify the sorting/matching behaviour.

What I have in mind is this:

• Entries are keyed by an alpha key
• The key can be a number (eg. "100")
• If the key is a simple number, with or without sign, they will be sorted into numeric sequence

What this means is that you can easily treat the arrays as simple numeric arrays, without worrying about whether a sequence like 1, 11, 2 sorts as 1, 2, 11, or 1, 11, 2.

Straight "alpha" keys *would* sort as 1, 11, 2 as "11" is lower in the sort sequence than "2". This would make numeric keys fiddly to use, as you would have to put in leading zeroes.

That way, if you want a simple sequence of things, you just supply numeric keys (effectively creating a sparse vector), like this:



By "sparse" I mean you can index in with numbers, but not every position will necessarily be taken.

By choosing suitable key numbers you could implement a stack or a queue as well.



I don't think speed is the major issue. I am thinking of things like an alias that might get information about a spell. Doing a keyed lookup is probably faster anyway than a linear search through variables or a language-dependent array anyway.

Having said that, the keyed lookup *would* be faster if you had a lot of them, because you could find (say) a spell out of a thousand by a simple lookup, whereas if you maintain a list, you may have to search through 999 of them to find it.



Yes I think you are right. If you want to get rid of the spaces you always can, but you can't put them back if they are gone.

I think I will also change it so it doesn't matter if the delimiter is in the data, I can see that as trouble waiting to happen.

I think something like the way mySQL exports its database as text files...

Say the delimiter is a comma, and you have data "fish,chips", it could be exported as:

food,fish\,chips

Then, of course, the \ symbol has to be exported as \\.

VB DOES support keyed arrays (well, at least i think so). They're called dictionaries, I haven't messed around with them, but look up the Dictionary object and you'll see what it's about.

Ah, OK - it is an ActiveX object.

Still, my proposed array handling is specifically designed for importing/exporting to standard variables, so that it can be readily used in plugins.

True, I like it, (although i don't completely understand it:P)

Hmm. Yeah, discounting the activex dictionary object, I guess for the default script types it is useful.

I can see how with some tricks you can create a FIFO, reverse stack or stack, but I am slightly unclear about two factors.

1. If you delete all elements in the array, does it destory the array or set an Empty property of some sort?

I suspect ArraySize returns 0 for an empty which is OK, but see the following...

2. Are the arrays accessable by normal numeric indexes? I.e.:

ArraySet "MyArray", "1", "Fred"

retrieves with:

ArrayGet ("MyArray", "1")
ArrayGet ("MyArray", 1)

Or is the second case disallowed. Even nastier, if the array index is an integer, then it will sort by that and the first index, if 0 or 1 may not even exist as a key. For a FIFO type setup you need to be able to find the *first* element, regardless of what its key is. In other cases, like stacks you can use the ArraySize to find the last element and push the next highest number as the key or retrieve and delete the current one. However, there may still be cases where exporting the array to a string is not the best way to find the first item in the list. Some way to index into the array without the key is needed and in fact is available for Perl and other languages that use such indexed arrays.

If you delete all elements (ArrayClear) it merely removes the items, thus giving you an empty array whose size is zero.

To delete the array itself you would use ArrayDelete.

Yes, you can use the numbers without quoting them (at least, in VBscript) because of the type casting provided by the script engine. eg.



This example uses the new techniques recently applied of sorting into numerical order if possible, and as you can see when the array is listed the keys are sorted correctly. Also, I could set and access using pure numeric keys.

As for the first and last item - I hadn't really intended this to be used as a queue or stack, but - what the heck - why not?

I have added two new functions ArrayGetFirstKey and ArrayGetLastKey - these will return the *key* of the first and last elements respectively, regardless of what it might be.

In conjunction with numeric keys, you can now implement a vector (simple list), stack or queue, by simply fiddling with the numbers at either end.

For example, to add to a stack:



Similarly you get add to a queue by getting the first key and subtracting 1.

By the same general method you can retrieve the first and last element (first get its key, then use that to get its value), and then if required delete that element.

To index into the array without knowing the keys is what the function ArrayListKeys is for. That returns a variant array (VB array) with each key as an element. You can then use "for each" in VBscript to iterate through the entire array.

However if you are using numeric keys, and not leaving gaps, simply finding the first and last key would be sufficient to know what range to retrieve the elements from.

OK, thanks to everyone who responded. I have now amended the behaviour a bit. In addition to the new routines ArrayGetFirstKey and ArrayGetLastKey, the general behaviour has been changed as follows:

• There are now *no* checks or changes made to array names, key names or key data. In other words, an array or key can be an empty string, or contain leading and/or trailing spaces. The idea here is that arrays are for use by scripters. If you want to remove spaces, do it yourself.
  
• The import/export functions now "escape" out the delimiter itself (if present) so that you can use the delimiter inside the data. I think this is better than having a script that works 99% of the time, but fails one day because someone happens to put a comma into a spell name or something. The escaping is done by putting a backslash in front of it. Because of this backslashes themselves are now escaped as a double-backslash.
  

One useful side effect of "unnamed" arrays could be that you would use an array with no name as a work array, rather than having to invent a name for it.

The escaping of exported data means you can now nest exported arrays within each other, without having to go mad trying to choose delimiters that won't clash with each other.

You are right in saying that JScript does not have associative arrays. We have a custom object for them where I work and we use it constantly. Good addition imho, Nick.

Some comments:

* I don't think you should allow empty keys.
* You should guarantee that keys are unique, so it can be used to uniquify a list of things. (do this in perl a lot)
* You should provide a way to get the list of values.

> * I don't think you should allow empty keys.
If all of them are blank, then does it act list a normal array? If you use two seperate ArraySet commands with non-unique keys, does this produce two entries or replace the first one with the second? This is even more confusing if you get the key array so you can use 'For Each' to step through the contents. Since you cannot retrieve an array of values, you will end up *retrieving* the first value over and over, since all of them are blank, I would assume....? I agree, this is a bad idea.

> * You should guarantee that keys are unique, so it can be used to uniquify a list of things. (do this in perl a lot)

Same issue as question two above. I agree that as long as you depend on ArrayExport or the keys to get the data from the array, allowing non-unique keys is a bad idea. But even if you could get an array of values, if you intentionally want to replace the value of a unique key, how do you do that?? Delete it first, then ArraySet it again I guess, but then how the heck do you replace the value of any single non-unique key? If you try to delete the key, you delete everything or only the first key found with that value, neither one of which is useful.

> * You should provide a way to get the list of values.
Definitely, this would make some things quite a bit easier.

My own suggestion.. Either:

1) a flag you can set on the array that determines its behavior.

2) or a different command, so you can create 'more or less' keyless arrays.

Flag version -

ArrayType "MyArray", "Indexed"
ArraySet "MyArray","Fred","123" 'Works.
ArraySet "MyArray",,"123" 'Generates an error.
ArraySet "MyArray",5,"345" 'Replaces element 5 with "345" or adds element 5 if it doesn't yet exist.

ArrayType "MyArray", "AutoIndex"
ArraySet "MyArray","Fred","123" 'Generates an error.
ArraySet "MyArray",,"123" 'Adds the next highest integer key.
ArraySet "MyArray",5,"345" 'Replaces element 5 with "345", only if element 5 exists. or produces an error if not.

If you used a different command:

ArrayAutoSet "MyArray", "345"

However this would cause confusion if the user mixed them up.

Both options would solve the issue of blank keys, since the key will always be unique and can never be blank. The array would simply act as though it was a normal array with a normal index.

I think the first version with the flag is more usable. Only issue with this is whether to generate an error, or make ArraySet into a function, so it will return True for success and False when it fails. An error is more visible, but a function is slightly more friendly, especially since you probably can't use 'On Error' to trap Mushclient generated errors.

You haven't commented now on whether the keys should be trimmed of leading/trailing spaces.

I know I put in the restriction about keys not being empty, and then took it out, but what is the great harm? A value can be empty, eg.



Now you wouldn't want a restriction that you can't have empty values (because a 'message' or something might be empty). So what is the problem with a blank key? eg.



This is being done in a script, if you don't want to use un-named keys (or un-named arrays), don't.

As for the spaces issue, I thought I may as well remove the space trimming, after all, again you can do it if you want to, eg. now these are all different keys:



Again, this is done in a program (script), if you add an extra space when writing keys, it is no real difference to mistyping the key altogether.



I may not have clarified that. Keys are indeed unique, so if you use any key (including the "empty" key) it will always only refer to a single entry in the array.

The ArraySet function replaces an existing value if necessary, however it returns a different return code if you are keen to know whether that actually happened.



You can get an array of the key names, just "for each" through that to extract the values.

However to keep my script-writers happy, I have added a new function ArrayGetValues that simply returns an array of all the values, in case you want them and don't particularly care what the keys are.




I don't really see why you would use a keyed array system and then want to make keyless arrays, but in case you do, my earlier suggestion of numeric keys would work. Either keep track of the highest key and just keep adding 1 to it, or use ArrayGetLastKey to find the value of the last key, and add 1 to that, as in my earlier example.

Ok. That clears things up. So an empty key would be one and only 'one' value in that array, not several. It seemed a bit odd, since I assumed that may mean that keys where not always unique, which is why I suggested an alternate way to generate an array that was indexed, but would appear from the users point of view to not have any. Since the blank key is as unique as any others, this becomes irrelevant.

See Plugin to gag players using new array script routines for a real-life example of using the new routines. The adding/deleting of gagged players is somewhat simplified by being able to key into the array.