mushclient + python + multithreading = boom

Posted by Shwick on Thu 27 May 2010 01:21 AM — 30 posts, 108,797 views.

Shwick Thu 27 May 2010 01:21 AM #0

I need to put some random delays in between my commands.

DoAfter isn't cutting it; if I have consecutive triggers with commands to send, DoAfter can mix up the commands.

After reading faq #25 and seeing lua coroutines talked about, I tried threads in python with timer.sleep().

Every time world.Note() is called within the thread, mushclient crashes, even if I use semaphores to limit it to one thread of execution.

Is there a way to code python threads within Mushclient?

test class


import threading

#thread class
class ThreadTest(threading.Thread):        
    
    def __init__(self):
        threading.Thread.__init__(self)
        
    def run(self):
        self.sendHi()

    def sendHi(self):
        world.Note('hi')

#alias
def testIt(thename, theoutput, wildcards):
    world.Note("testIt called")
    t = ThreadTest()
    try:
        t.start()
    except:
        world.Note("Unexpected error:", sys.exc_info()[0])
        raise

result


testIt called
hi

then crash

Twisol USA Thu 27 May 2010 01:29 AM #1

You could try keeping a queue of commands to send, and a timer that sends the next command when it fires. To add randomness, have the timer change its own delay every time it's fired.

EDIT: I have no idea on the threading thing. If it's any consolation, you'd probably have a nondeterministic order of commands in Lua, too. It's a timer thing, not a language thing.

EDIT 2: Lua coroutines aren't really "threads", they're more like fibers. Only one ever executes at a time, and they cooperate by explicitly passing control. Normal threads (which I assume Python uses) are preemptive rather than cooperative, which leads to the usual mess.

Nick Gammon Australia Forum Administrator Thu 27 May 2010 02:37 AM #2

I am no Python expert, but all I can suggest is make a queue of some sort, and use DoAfterSpecial to call a routine that pulls out the top item from the queue, then reschedules for the next one and so on. At least they will be done in order then.

Shwick Thu 27 May 2010 02:53 AM #3

omg it works thank you so much Twisol, perfect solution

PHEW

thought i would have to switch to lua for a second there

god forbid

also thanks Nick for continuous posts helping me get through this quickly

Twisol USA Thu 27 May 2010 02:55 AM #4

Shwick said:

omg it works thank you so much Twisol, perfect solution

PHEW

thought i would have to switch to lua for a second there

god forbid

also thanks Nick for continuous posts helping me get through this quickly

*rofl* No problem!

CJI Poland Tue 13 Jul 2010 12:19 PM #5

Hi!

First thing first - sorry for my bad english, please try to understand me :)

Now to the point - Python threads are not real threads - there is this thing called GIL (Global Interpreter Lock) so at any given time only one thread is running, even on multicore or multiprocessor machine. Switching between 'threads' is done by interpreter every now and then - it's done behind the scenes, and it results in python interpreter running always in one OS level thread, which is good for embedding (I think).

Now, the odd thing is that no amount of Locks, Queues, Semaphores, whatevers - enables one to use Python threads in MUSHClient. MC either blows (program has stopped working) or other threads (besides main) don't run at all.

Consider this code:


import subprocess, time
from threading import Lock, Thread, enumerate as tenum

global_lock = Lock()

def now():
    with global_lock:
        return time.time()

def note( *args ):
    with global_lock:
        s = "".join( [ str( x ) for x in args ] )
        world.Note( s )

def t():
    for i in range( 50 ):
        note( "It's now ", now(), " since epoch" )
        time.sleep( 2 )

note( tenum(), " ", now() )

t = Thread( target = t )
t.start()

note( tenum(), " ", now() )

It really SHOULD work! But it does not. MC stops responding immediately after opening the world. After trying many various approaches, I had to give up - there's something wrong with hardcode or possibly python com extension. Of course multithreading isn't a 'must have' for me, there are plenty of techniques (including coroutines, implemented in python via generators) or subprocesses (which work), but still - this is a bug somewhere. Sorry, Nick, I had to tell you about it ;)

Last thing - there is one thing I can think of that requires some kind of concurency (it's much harder without it) - and I mean doing some very heavy computations in background. Well, I have written quick test:


import subprocess, time

def note( *args ):
    s = "".join( [ str( x ) for x in args ] )
    world.Note( s )

p = subprocess.Popen( [ "ping", "-t", "wp.pl" ], stdout = subprocess.PIPE )
count = 0

def timer( *args ):
    global count
    count += 1
    if count >= 70:
        p.kill()
        return
    note( p.stdout.read( 10 ) )

Which works perfectly (with timer invoking timer routine), producing expected output:


Badanie 
wp.pl [212
.77.100.10
1] z 32 ba
jtami dany
ch:
Odpow
ied« z 212
.77.100.10
1: bajt&#728;w=
32 czas=34
ms TTL=248

and MC is still responding :)

I'm currently writing python framework for MC scripting, so expect me to post here once in a while.

One additional question - is python interpreter in MC ever restarted? I hate it when I have to write something this ugly:


    import sys, os

    SCRIPTS_PATH = "C:\Users\cji\Desktop\MUSHclient\python"
    sys.path.insert( 0, SCRIPTS_PATH )

    import world_wrap; reload( world_wrap )
    world_wrap.init( world )

    import trigreg
    import printer
    import constants

    # I mean THIS ugly:
    def reload_modules():
        for name, m in globals().iteritems():
            try:
                if name == "world_wrap":
                    continue
                __import__( name )
                printer.printf( "reloading %s\n" % name )
                reload( m )
            except:
                pass
    reload_modules()

just because Ctrl+Alt+R seems to reload script file without restarting interpreter.

Nick Gammon Australia Forum Administrator Tue 13 Jul 2010 08:50 PM #6

Quote:

for i in range( 50 ):
note( "It's now ", now(), " since epoch" )
time.sleep( 2 )

Assuming this does what it looks like, I wouldn't do it. In any language, doing a sleep basically just hangs the interpreter, and in your case it looks like you will do it for 100 seconds (50 * 2).

I don't want to push Lua when you obviously want to use Python, but the only real way of doing similar code, that I know of, is to use Lua coroutines. These are a form of co-operative multi-tasking, where coroutines yield execution at places of your choice.

As for the bug, if someone can point it out I'll be happy to fix it, but the fact is that all of the WSH (Windows Script Host) scripting is done exactly the same way. It is the same code, and the only difference is the GUID of the script engine which is selected. There is no real possibility for a "Python bug", only a bug that affects all script engines. However scripting has been active for quite a while now (like, years) and no-one else complains.

Lua is implemented differently, because for one thing, there is (or was) no WSH interface for Lua. So I wrote "glue" routines that connect the Lua calls to the underlying functions that the other script engines use.

Some WSH implementors have difficulty getting it right, for example a while back if you made a compile error in PHP, the entire client simply exited. This only happened in PHP, and I assume that the script writers thought they could display an error message and "exit program" the same way they normally do. However that is a bit unfriendly to the host program.

CJI said:

One additional question - is python interpreter in MC ever restarted? I hate it when I have to write something this ugly ...

Reloading the script file should (does) recreate the script engine. The old engine is "deleted" (in the C++ sense) and a new one started from scratch. Then the script file is reloaded into the new engine, effectively setting all variables and settings back to the default state.

However, if the Python DLL has some "memory" itself, then possibly it will not behave the same as the first time. Again, all script languages other than Lua are done the same way.

A Google for "Python threading WSH" seems to indicate that other people are having problems with Python, threading, and the WSH implementation.

CJI Poland Wed 14 Jul 2010 09:10 AM #7

Nick Gammon said:

Quote:

for i in range( 50 ):
note( "It's now ", now(), " since epoch" )
time.sleep( 2 )

Assuming this does what it looks like, I wouldn't do it.

It works fine in Python. time.sleep() does 'hang' - but only current thread, not the whole interpreter. Try it on Python interactive prompt or - alternatively - just believe me ;)

Nick Gammon said:
is to use Lua coroutines.

Argh! To ... with Lua! (Meaning - I know basics of this language, but I just don't want to use it if I don't have to.) Coroutines are very easy to implement in Python, there are even few external libraries ready to use, but if you want lightweight coroutines you can just use Python native generators. In Lua docs there is comparision of coroutines with generators - but it's very outdated and I'm sure that nowadays there is no single thing you can achieve with coroutines that you can not with generators.

But that's not the point - various techniques, like coroutines or deferreds in Twisted are nice, of course, but are NOT threads. You can not use ANY blocking operation in coroutines, while you can do this in separate thread. Let's say I want to read some data from open socket - socket.recv( 1024 ) will block, if there is no data to read, stopping all scripting. Of course, you can use non-blocking socket and check if there is some data, reading it only when necessary - but using separate thread is simpler.

Nick Gammon said:

There is no real possibility for a "Python bug", only a bug that affects all script engines. However scripting has been active for quite a while now (like, years) and no-one else complains.

It's very easy to explain - only Ruby offers threads similar to Python's and my guess is that they don't work either, but scripting with Ruby is even less popular (I think) than with Python. I'm not sure about PERL (googled - it has threading, done with starting new interpreter for each thread. If I was a betting person I'd say it doesn't work either), but JS and friends are (I think?) single threaded by nature. Quick googling:

Quote:
PHP has no built in support for threading.

Quote:
There's no true threading in JavaScript.

and so on.

Nick Gammon said:

However, if the Python DLL has some "memory" itself,

It probably has - import statements are cached and this cache seems to be cleared only when MUSHClient is restarted.

Quote:

Again, all script languages other than Lua are done the same way.

Well, I can live with that, no big deal. With my reloading routine and occasional restarts while developing it's just a bit uncomfortable. The 'world' object is accessible only in one file, but hey - I wrote simple wrapper module, which - if initialised - is importable everywhere and is friendlier than original, printing meaning of return code if not equal to 0 automatically.
[I can't resist:


class Wrapper( object ):
    def __init__( self, w ):
        self.world = w

    def __getattr__( self, name ):
        if not hasattr( self.world, name ):
            raise Exception( "No such method or attribute!" )

        attr = getattr( self.world, name )
        if not callable( attr ):
            return attr

        func = attr

        def wrap( *args, **kwargs ):
            ret = func( *args, **kwargs )
            if ret is None:
                ret = 0

            try:
                msg = smart_ret[ int( ret ) ]
            except:
                return ret

            if ret != 0:
                self.world.note( name )
                self.world.note( msg )

            return msg
        return wrap

com = None

def init( world ):
    global com
    com = Wrapper( world )

beautiful, isn't it? ]

Working around weird paths which made imports impossible was easy, like with mod_wsgi and apache, so I did it very quickly. "script" attribute of triggers, that needs to be in main script file and has to be directly callable (object.method as a trigger won't work)? No problem, I wrote simple dispatcher and TriggerRegistry class which maps triggers to any callable. world.Note accepting only strings? It's even easier, just wrap it with some printer object. And so on...

Now I have seven script files, still long way to go - and for few weeks I haven't play my MUD. But hey, programming is better than playing, right? ;)

Also, I see you're no python fan and I don't want to convince you to become one right now. I just want to tell you that - while for ordinary users it's just fine - scripting support is quite restrictive for programmers (which hearing words "global variable" scream in horror, for example), and I guess any Ruby or even Perl developer scripting for MC would agree with me. I understand that it's WSH fault (I have to read more about it, I know almost nothing now), but I know for sure, that there are better ways for embedding scripting languages.

[Wow, I missed one thing - MUSHClient source is now open! Sorry, last time I checked (few years ago...) it wasn't.]

Twisol USA Wed 14 Jul 2010 09:44 AM #8

CJI said:
I just want to tell you that - while for ordinary users it's just fine - scripting support is quite restrictive for programmers (which hearing words "global variable" scream in horror, for example), and I guess any Ruby or even Perl developer scripting for MC would agree with me. I understand that it's WSH fault (I have to read more about it, I know almost nothing now), but I know for sure, that there are better ways for embedding scripting languages.

It's almost definitely the fault of WSH. Lua is embedded directly, without using WSH, and is much much easier to use. (Yes, I've tried others with MC, specifically Ruby.)

As a side-note, I can't begin to tell you how hard I'm facepalming right now. XD Do you know how many people protested at the very idea of limiting MUSHclient scripting to one embedded language? Managing multiple scripting interpreters without WSH is almost certainly no easy task. The best design I personally have come up with - just as a mental exercise, nothing concrete - would require someone to create a new DLL with the scripting language embedded within, including glue routines that access a common API within MUSHclient. And do you know, that's exactly how WSH works?

So really, we can use either WSH or just one language. (Correct me if I'm wrong, Nick.) And the "one language" would almost certainly be Lua. Lua is extremely suitable for embedding within an application, with the major benefit that it's extremely easy to sandbox. (God, try doing THAT in Ruby. I love the language to death, but it is just not made for sandboxing.)

On a less... passionate note (sorry!), I'm not really a language fanboy. I treat my languages as tools, and I try to use the best tool for the job whenever possible. I don't know anything about you, so don't take this personally... it's late, and things might not be coming out right. (Wouldn't be the first time.) But it seems to me like, when you only have a hammer, everything starts to look like a nail. In one sense, WSH could actually be a detriment.

Man... I need to get some sleep. Again, I'm sorry if anything came out wrong, trust me when I say I'm not trying to start a fight. I just saw what you said above (that I quoted), and stuff just started pouring out. XD

CJI Poland Wed 14 Jul 2010 10:36 AM #9

Twisol said:

The best design I personally have come up with - just as a mental exercise, nothing concrete - would require someone to create a new DLL with the scripting language embedded within, including glue routines that access a common API within MUSHclient.

Yes, I realised that much - and I was suspecting that WSH does something like that, too. I just thought that "creating a new DLL with the scripting language embedded within" is quite an easy task - I have some experience in directly embedding Python and it was simple and fun for the most part.

But, as I said, I have no knowledge of WSH at all and I don't want to criticize MUSHClient design. I see that using WSH is the most comfortable option which demands the least work and I agree that it is enough for almost every scripting task. This threading thing is just one issue that seems to be impossible - and it's certainly not enough to justify major rewrite of scripting in MUSHClient. I was just curious - why it works that way and if there is something that could be done to correct it without big changes, that's all.

Twisol said:

God, try doing THAT in Ruby. I love the language to death, but it is just not made for sandboxing.

I had to think for a while. At first I was like "why is sandboxing even necessary?" because I never ran any script not written by me. And then I realised that it wouldn't be nice to run some downloaded plugin and have all my files deleted, for example.

Twisol said:

On a less... passionate note (sorry!), I'm not really a language fanboy. I treat my languages as tools, and I try to use the best tool for the job whenever possible. I don't know anything about you, so don't take this personally... it's late, and things might not be coming out right. (Wouldn't be the first time.) But it seems to me like, when you only have a hammer, everything starts to look like a nail. In one sense, WSH could actually be a detriment.

I had to look up 'detriment' in dictionary, and that's only thing that irritated me (joking) in this piece of text ;)

It's certainly true that there are different tools for different problems, I accept that. That said - I use Python mainly to write standalone apps or with web development frameworks (Django, Plone) and I'm just not used to working under restrictions found in MUSHClient.

They are workable around, of course, and I'm doing it right now. But I'm a little disturbed now - I mean your suggestion that 'everything starts to look like a nail'. I'm not offended or anything, but I started to wonder whether my goal is the right thing to do in scripting environment of MUSHClient. I'll write separate post (and topic) explaining what I want to do and I hope that you'll comment there, ok?

Twisol said:

Man... I need to get some sleep. Again, I'm sorry if anything came out wrong, trust me when I say I'm not trying to start a fight. I just saw what you said above (that I quoted), and stuff just started pouring out. XD

No problem, thanks for replying ang get some sleep :) And it's actually noon for me ;)

Nick Gammon Australia Forum Administrator Wed 14 Jul 2010 10:52 AM #10

CJI said:

But that's not the point - various techniques, like coroutines or deferreds in Twisted are nice, of course, but are NOT threads. You can not use ANY blocking operation in coroutines, ...

Hmmmm. Before I answer in more detail (it is getting late here) I just want to say that a while back I thought threads were some sort of magical solution to all sorts of problems (eg. blocking operations). I gradually became more disillusioned with them, partly because of issues with threads needing to have thread-safe locks before accessing variables which might be changed by different threads. More recently I found this:

http://www.sqlite.org/faq.html#q6

Quote:

(6) Is SQLite threadsafe?

Threads are evil. Avoid them.

That link leads here:

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

And to quote from that document:

Quote:

They make programs absurdly nondeterministic, and rely on programming style to constrain that nondeterminism to achieve deterministic aims.

So the gist of my response is to query whether, in saying MUSHclient and Python and threads don't work together that well, is that is this really a huge problem in MUSHclient, or perhaps ... perhaps ... you should revisit using threads? :-)

CJI Poland Wed 14 Jul 2010 11:21 AM #11

Quote:

(6) Is SQLite threadsafe?
Threads are evil. Avoid them.

:D Good point.

Now, where have I seen thread used... Where could it be... Oh, yes, I remember:


// ------------------- script file change monitoring thread -------------------------

void CMUSHclientDoc::ThreadFunc(LPVOID pParam)
{
// ...

// Create script source file monitoring thread
//
void CMUSHclientDoc::CreateMonitoringThread()
{
  KillThread (m_pThread, m_eventScriptFileChanged);

	CThreadData*	pData = new CThreadData;
	pData->m_strFilename = new char [m_strScriptFilename.GetLength () + 1];
  strcpy (pData->m_strFilename, m_strScriptFilename);
	pData->m_hWnd = Frame.GetSafeHwnd ();
	pData->m_hEvent = m_eventScriptFileChanged;
  pData->m_pDoc = (DWORD) this;
	m_eventScriptFileChanged.ResetEvent();

	m_pThread = (HANDLE) _beginthread (ThreadFunc, 0, pData);
  SetThreadPriority (m_pThread, THREAD_PRIORITY_IDLE);

	// Thread will delete data object
}

:P

It's not that I have to use threads at all costs. They just have their place in programming since long ago, that's all.

And to justify myself - I wanted to do in Python same thing you did with CreateMonitoringThread - check whether source code of some module (other than main script file, which above routine takes care of) has changed and automatically restart script engine then. I would do it with timer from the start, but I found this nice (threaded) routine: http://code.djangoproject.com/browser/django/trunk/django/utils/autoreload.py and tried to use it. That was bad idea... Now I use it with timer and it works, so it's not an issue.

Twisol USA Wed 14 Jul 2010 05:13 PM #12

Refreshed and awake(-ish) now!

CJI said:
Yes, I realised that much - and I was suspecting that WSH does something like that, too. I just thought that "creating a new DLL with the scripting language embedded within" is quite an easy task - I have some experience in directly embedding Python and it was simple and fun for the most part.

But, as I said, I have no knowledge of WSH at all and I don't want to criticize MUSHClient design. I see that using WSH is the most comfortable option which demands the least work and I agree that it is enough for almost every scripting task. This threading thing is just one issue that seems to be impossible - and it's certainly not enough to justify major rewrite of scripting in MUSHClient. I was just curious - why it works that way and if there is something that could be done to correct it without big changes, that's all.

*nod* I understand. Yes, generally embedding a language is fairly easy, I've done it myself with Lua. The problem is that someone needs to maintain these new DLLs, and provide the glue routines that translate the generic MUSH calls into language-specific operations, and vice versa. And you rely solely on that person to ensure that your scripting experience isn't lamer than WSH. And of course, someone has to design the generic API that each DLL would use. Everything just adds up to a subpar experience in my opinon. It's much easier to focus on one language, and make it shine.

CJI said:
I had to think for a while. At first I was like "why is sandboxing even necessary?" because I never ran any script not written by me. And then I realised that it wouldn't be nice to run some downloaded plugin and have all my files de-leted, for example.

Exactly. And there's almost no sandboxing with the WSH languages, and I honestly don't know where to even begin adding it with a custom WSH-like design. Most of these languages come with extensive standard libraries already. With Lua, it's a downright breeze! You simply disable the standard libraries that pose a risk to your application, and provide your own API functions that expose the specific functionality that scripters will use to interact with the client.

CJI said:
I had to look up 'detriment' in dictionary, and that's only thing that irritated me (joking) in this piece of text ;)

It's certainly true that there are different tools for different problems, I accept that. That said - I use Python mainly to write standalone apps or with web development frameworks (Django, Plone) and I'm just not used to working under restrictions found in MUSHClient.

They are workable around, of course, and I'm doing it right now. But I'm a little disturbed now - I mean your suggestion that 'everything starts to look like a nail'. I'm not offended or anything, but I started to wonder whether my goal is the right thing to do in scripting environment of MUSHClient. I'll write separate post (and topic) explaining what I want to do and I hope that you'll comment there, ok?

That's exactly my point. :) Python and the like are really good standalone languages, and they're very good when that's what you're writing. But I'm not sure it's the right tool when you want an embedded language. And just to clarify my "WSH is a detriment" statement, WSH makes everything look like nails, by providing a suite of languages but exposing exactly the same things to them. It puts the choice of language almost entirely down to taste!

David Haley USA Wed 14 Jul 2010 05:18 PM #13

Twisol said:
Python and the like are really good standalone languages, and they're very good when that's what you're writing. But I'm not sure it's the right tool when you want an embedded language.

Eh, Python isn't nearly as bad to embed as some of the other languages. And many applications (e.g., EvE Online) have very, very successfully embedded Python.

The Python API isn't as nice as Lua's (it's more complicated, really) but it is far, far nicer than, say, Perl.

In principle somebody could do a proper embedding of Python as Nick did for Lua, if you felt like spending the time on it. I think the words about subpar experiences are slightly exaggerated when it comes to this, but you do need an advocate of some sort willing to stand behind it and do the work.

Twisol USA Wed 14 Jul 2010 05:19 PM #14

CJI said:
And to justify myself - I wanted to do in Python same thing you did with CreateMonitoringThread - check whether source code of some module (other than main script file, which above routine takes care of) has changed and automatically restart script engine then. I would do it with timer from the start, but I found this nice (threaded) routine: http://code.djangoproject.com/browser/django/trunk/django/utils/autoreload.py and tried to use it. That was bad idea... Now I use it with timer and it works, so it's not an issue.

Hmm... You can use UdpListen() in MUSHclient to set up a UDP port to listen on, and run a standalone Python program that uses your thread to watch the file. Then when it sees a change, it can notify your MUSHclient script via the UDP port.

Twisol USA Wed 14 Jul 2010 05:32 PM #15

David Haley said:

Eh, Python isn't nearly as bad to embed as some of the other languages. And many applications (e.g., EvE Online) have very, very successfully embedded Python.

Yeah, I'm sure it's just fine to embed. I only think it's not the right tool for the job. I might look up some Eve scripting to see how theirs looks though.

David Haley said:
In principle somebody could do a proper embedding of Python as Nick did for Lua, if you felt like spending the time on it. I think the words about subpar experiences are slightly exaggerated when it comes to this, but you do need an advocate of some sort willing to stand behind it and do the work.

Yeah, it's definitely possible. It would be a bit of a nightmare to maintain multiple languages within the same project, and it's needless bloat when most users will only use one language. Hence the DLL approach, which has its own set of complexities. I tend to think that, overall, it would be a subpar experience with at least one language.

David Haley USA Wed 14 Jul 2010 06:12 PM #16

Twisol said:
I'm sure it's just fine to embed. I only think it's not the right tool for the job.

It's fine to embed, but not the right tool for embedding? ;)

Twisol said:
It would be a bit of a nightmare to maintain multiple languages within the same project, and it's needless bloat when most users will only use one language.

Surely this is a slight exaggeration?

Besides, just because most users use a single language doesn't mean that a single language is used by most users...

Nick Gammon Australia Forum Administrator Wed 14 Jul 2010 08:58 PM #17

CJI said:

Now, where have I seen thread used... Where could it be... Oh, yes, I remember:

void CMUSHclientDoc::ThreadFunc(LPVOID pParam)

Yes indeed. I was going to admit that MUSHclient actually uses threads a bit. That code I did myself. In my defense, that was long before I read about "threads are evil".

Also MFC (the underlying library) also uses threads, in particular modal dialog boxes are not really modal dialogs, but MFC kicks off a new thread, and then locks the underlying window so you can't click on it. The reason is to keep the document "alive" while you stare at the dialog. You can see this in action, because if you edit the world's configuration, you can see text from the MUD scroll by underneath.

In fact this itself caused a major headache, because the configuration editor could not assume that the data structures had not changed while you were looking at them. For example, a trigger that deleted another trigger, might delete a trigger while you were editing it.

However you can live without threads. The stop-start nature of data to and from a MUD would seem to cry out for threads, but instead the client uses asynchronous IO routines (as do many servers) which still give you the ability to handle half-completed actions.

As noted, threads solve one problem, but create another. At least coroutines, which pause at deterministic places, cause far less problems with issue like whether variables change in value from one line of code to the next.

Nick Gammon Australia Forum Administrator Wed 14 Jul 2010 09:04 PM #18

David Haley said:

Eh, Python isn't nearly as bad to embed as some of the other languages.

No doubt Python can be embedded. However as someone who embedded Lua directly, I can report it took quite a lot of work. Basically you have something like 400 script functions which need some sort of "glue" routine to interface them with the underlying functionality they are supposed to do.

And then you need to handle differently things like arrays (which in the WSH end are done with Variant arrays) and convert them into the script equivalent (in the case of Lua it was tables). And booleans are done differently in Lua at least (the number zero being considered true, for example).

Then it all has to be tested. And documented.

And then I look at other games (eg. World of Warcraft client) which *only* supports Lua, and I ask myself "why go to all the effort?".

David Haley USA Wed 14 Jul 2010 09:21 PM #19

Well, I was mostly trying to probe what makes a given tool the right one for this job. I think the WSH approach here is good enough for Python, generally speaking, so I wouldn't ask that you actually do this work unless somebody really really wanted Python -- in which case they should provide the binding themselves. ;)

Twisol USA Wed 14 Jul 2010 10:13 PM #20

David Haley said:

Twisol said:
I'm sure it's just fine to embed. I only think it's not the right tool for the job.

It's fine to embed, but not the right tool for embedding? ;)

Well, if you look at what I responded to, you'll see that by "fine" I meant that the difficulty isn't great. The rest of my point was that I don't think it's the right language to embed, at least in this case.

David Haley said:

Twisol said:
It would be a bit of a nightmare to maintain multiple languages within the same project, and it's needless bloat when most users will only use one language.

Surely this is a slight exaggeration?

Not really. Have you seen all of the glue code one has to maintain for the Lua interface? You'd have to do this again for every additional language.

David Haley said:
Besides, just because most users use a single language doesn't mean that a single language is used by most users...

But having all languages in one program just to satisfy everyone is impractical, especially with static linkage which would, as I mentioned, introduce a lot of bloat, most of which any given user won't ever use (because they use just one of those languages generally). Dynamic linkage is a better solution, but we have that in WSH, and I've voiced my perspective on WSH already. (And, of course, WSH is not cross-platform.)

I think MUSHclient is the only program I've ever seen to support multiple languages this way. WoW uses only Lua, even EvE only uses Python, and I don't see anyone there complaining about a lack of options.

Nick Gammon said:
However you can live without threads. The stop-start nature of data to and from a MUD would seem to cry out for threads, but instead the client uses asynchronous IO routines (as do many servers) which still give you the ability to handle half-completed actions.

I'd like to note here that Aspect uses an excellent event-based library called EventMachine, and I try to avoid threads in general. It's not like I have anything remotely near completion, but so far it's really quite easy to deal with, and I like this model a lot.

Twisol USA Wed 14 Jul 2010 10:21 PM #21

David Haley said:
Well, I was mostly trying to probe what makes a given tool the right one for this job. I think the WSH approach here is good enough for Python, generally speaking, so I wouldn't ask that you actually do this work unless somebody really really wanted Python -- in which case they should provide the binding themselves. ;)

I think there's a lot about Lua that really lends itself to embedding. The big one for me is that it's extremely easy to sandbox. I want control over what scripters can and can't do within my application. If I don't have that control, it's risky both for me and anyone using those scripts. Lua also comes with a minimalist standard library, which you control the presence or absence of. That's probably a "bad thing" for people who want to use Lua for standalone programs, but it makes it a wonderful candidate for embedding.

CJI Poland Thu 15 Jul 2010 12:33 AM #22

I'll reply in reversed order.

Twisol said:

The big one for me is that it's extremely easy to sandbox.

Well, sandboxing, if done well, is secure... But I really don't like the idea. You reduce danger of executing harmful code, but at the same time you greatly reduce functionality available in scripts. I just checked, out of curiosity, and this code worked:


import win32api
world.Note( win32api.GetComputerName() )

It means that almost all WinAPI calls are directly accessible from my python scripts. I could create new window (threading works to some extent - just accessing "world" object crashes MUSHClient, so I could really create another system-level window and do anything with it!), paint on the desktop, register (with pyHook) callback to intercept mouse and keyboard messages - system-wide (which is good, sometimes - and bad, if I want to write keylogger and convince someone to use my script...). And many, many more cool things, if I felt like refreshing my WinAPI skills.

If Python was properly sandboxed, this would (or at least should) be impossible. I'm not sure I really want to trade this functionality for security, especially considering that NOW it works that way and no one complains ;P

Twisol said:

I think MUSHclient is the only program I've ever seen to support multiple languages

I think so too. And that's one of many reasons why I use MUSHClient, and why I think it's simply the best client out there. Please don't talk about dropping support for other languages. Pretty please :)

David Haley said:

I think the WSH approach here is good enough for Python [...] unless somebody really really wanted Python -- in which case they should provide the binding themselves. ;)

I agree. Absolutely. I might - some time in future - give it a try, but - generally speaking - the way it works now is sufficient for me. So maybe some day as an exercise, but not anytime soon.

Nick Gammon said:

And then you need to handle differently things like arrays (which in the WSH end are done with Variant arrays) and convert them into the script equivalent (in the case of Lua it was tables). And booleans are done differently in Lua at least (the number zero being considered true, for example).

There are tools that automate part of the process. I know that they exist, but unfortunately I have no experience in using them... So as I said - maybe some day :)

Nick Gammon said:

Also MFC (the underlying library) also uses threads, in particular modal dialog boxes are not really modal dialogs, but MFC kicks off a new thread, and then locks the underlying window so you can't click on it.

I don't get it - why these dialogs are modal in the first place? My character once DIED! because of this! (It was over before I closed two dialogs that I had open...)

And Microsoft Foundation Classes are evil ;) I programmed some time with pure WinAPI in C, and then, fortunately, switched directly to QT and C++, so I was spared pleasure of working with MFC - but what I remember from quick overview is pure horror I felt. ;)

Nick Gammon said:

In my defense, that was long before I read about "threads are evil".

I really think that there is nothing to defend. Threads are just one of many tools you can use. They are not evil, neither they are some kind of super solution of "panacea". They have their weak points and advantages. Especially now, when mutlicore processors are so common, using threads is worth considering. Synchronization is an issue, but done right - poses no real threat to stability (but debugging is still hard). Threads are not evil. Their misuse - is.

David Haley said:

It's fine to embed, but not the right tool for embedding? ;)

Twisol explained later, but I think that embedding does not necessarily have to heavily restrict, or sandbox, embedded language. Twisol seems to have another opinion.

Twisol said:

You can use UdpListen() in MUSHclient to set up a UDP port to listen on, and run a standalone Python program that uses your thread to watch the file. Then when it sees a change, it can notify your MUSHclient script via the UDP port.

I thought about it for a brief moment, but that would be adding complexity instead of reducing it. If I had to script some kind of socket communication I'd do this with non-blockng socket using TCP from pure Python.

Twisol said:

Most of these languages come with extensive standard libraries already.

And, as I said earlier, I consider this an advantage. I understand your concerns, but really, sandboxing Python or Ruby or (I guess) PERL cuts their usability by half at least. Their main strength are huge standard libraries (yes, yes, elegant and pragmatic design, metaprogramming capabilities, beautiful syntax, etc. too, but if I could choose language based on syntax and design I'd pick haskell or ocaml ;) ).

If security is that big of the concern, you always can sandbox whole app or even system, like with BOCHS or the likes. It will be secure, no matter how inefficient.

On the other hand - I wrote my persistent data storage using SQLAlchemy and MySQL, effectively bypassing MUSHClient variables and never using them. I'd be very unhappy if I had to rewrite it and use variables (or built in sqlite) again... Miniwindows are sweet, but using them with WinAPI-like calls is painfully inefficient (in terms of code complexity) - I prefer generating image with PIL and then rendering it. I couldn't do it in heavily restricted environment...

Phiew. I wrote it all, thanks for reading :)

And one last thing - is it possible to register callback which will be called just before script engine restarts?

PS. Please correct any grammar or spelling mistakes I make. I read a lot in English and I have spell checker on, but I have very few opportunities to write, unfortunately.

Nick Gammon Australia Forum Administrator Thu 15 Jul 2010 01:14 AM #23

CJI said:

I don't get it - why these dialogs are modal in the first place? My character once DIED! because of this! (It was over before I closed two dialogs that I had open...)

Modeless dialog boxes are a special sort of pain themselves. :)

For example, if not closed yet, where do they go? I particularly dislike the Find dialog boxes that hang around even after you have found what you want.

And say you have world configuration up with a list of triggers, and then you edit a trigger, and it is all modeless, and you maybe change a trigger option by typing in a command in the command window, then that's two dialog boxes that need refreshing to show the new information.

Look, it is probably like threads. If you use them properly they can be fine.

Nick Gammon Australia Forum Administrator Thu 15 Jul 2010 01:15 AM #24

CJI said:

PS. Please correct any grammar or spelling mistakes I make. I read a lot in English and I have spell checker on, but I have very few opportunities to write, unfortunately.

Your English is very good. Any minor mistakes (and I haven't noticed any) could easily be made by native English speakers.

Twisol USA Thu 15 Jul 2010 01:28 AM #25

CJI said:
Well, sandboxing, if done well, is secure... But I really don't like the idea. You reduce danger of executing harmful code, but at the same time you greatly reduce functionality available in scripts. I just checked, out of curiosity, and this code worked:


import win32api
world.Note( win32api.GetComputerName() )

Well, yes. That's what makes it a good application language. And if the host program doesn't need to be very secure, sandboxing is completely optional (though I question how easy it would be to sandbox Python, too).

And the functionality issue is all down to what you want your users to be able to do. It's the flip side of sandboxing. The idea with sandboxing Lua is to hide all the directly abusable API calls - which is mostly just the OS and I/O libraries - and add your own APIs for doing what you want to allow. And of course you can reinstate some form of the original API, but perhaps limited to a specific directory rather than the whole computer, to minimize potential damage.

CJI said:
I think so too. And that's one of many reasons why I use MUSHClient, and why I think it's simply the best client out there. Please don't talk about dropping support for other languages. Pretty please :)

At this stage, I don't think it could happen. Too many people are used to it.

CJI said:
There are tools that automate part of the process. I know that they exist, but unfortunately I have no experience in using them... So as I said - maybe some day :)

Like SWIG? The functions being glued are kind of odd, I'm not sure how easy it would be. I've never used SWIG either, though.

CJI said:
I really think that there is nothing to defend. Threads are just one of many tools you can use. They are not evil, neither they are some kind of super solution of "panacea". They have their weak points and advantages. Especially now, when mutlicore processors are so common, using threads is worth considering. Synchronization is an issue, but done right - poses no real threat to stability (but debugging is still hard). Threads are not evil. Their misuse - is.

Well put. Still, I prefer to use threads for jobs that can be truly parallel; if at all possible I'd rather use an event-based approach like EventMachine.

CJI said:
Twisol explained later, but I think that embedding does not necessarily have to heavily restrict, or sandbox, embedded language. Twisol seems to have another opinion.

Quite right, it's not necessary. It really depends on the needs of the host application. When you're dealing with distributable third-party plugins, I'm inclined to prefer at least limited sandboxing. As you noted before, it wouldn't be nice to have a seemingly benign plugin erase all of your files.

CJI said:
I thought about it for a brief moment, but that would be adding complexity instead of reducing it. If I had to script some kind of socket communication I'd do this with non-blockng socket using TCP from pure Python.

Really? UDP is simpler than TCP, and when you're using it over the loopback interface (never leaving the physical computer), it's reliable. You just send packets at will, and receive them on the other end. That's exactly why UdpListen() was added, if I recall correctly. Before the advent of miniwindows, it could be used to communicate with an external program running additional windows.

CJI said:
And, as I said earlier, I consider this an advantage. I understand your concerns, but really, sandboxing Python or Ruby or (I guess) PERL cuts their usability by half at least. Their main strength are huge standard libraries (yes, yes, elegant and pragmatic design, metaprogramming capabilities, beautiful syntax, etc. too, but if I could choose language based on syntax and design I'd pick haskell or ocaml ;) ).

That's why I treat languages as separate tools. :) Lua is easy to sandbox. I'd never try to sandbox Python or Ruby, and I don't have much experience with any of the others you mentioned.

CJI said:
If security is that big of the concern, you always can sandbox whole app or even system, like with BOCHS or the likes. It will be secure, no matter how inefficient.

A bit overkill for the purposes I had in mind, however this is definitely something I will need to do with Aspect.

CJI said:
On the other hand - I wrote my persistent data storage using SQLAlchemy and MySQL, effectively bypassing MUSHClient variables and never using them. I'd be very unhappy if I had to rewrite it and use variables (or built in sqlite) again... Miniwindows are sweet, but using them with WinAPI-like calls is painfully inefficient (in terms of code complexity) - I prefer generating image with PIL and then rendering it. I couldn't do it in heavily restricted environment...

Eeeeergh. I completely agree with the miniwindow API, sadly. They're awesome, but they can be a real pain to use for truly complex things.

CJI said:
PS. Please correct any grammar or spelling mistakes I make. I read a lot in English and I have spell checker on, but I have very few opportunities to write, unfortunately.

I hardly had any idea you might have trouble with English. :)

David Haley USA Thu 15 Jul 2010 05:19 AM #26

Quote:
The rest of my point was that I don't think it's the right language to embed, at least in this case.

Surely, if you find yourself repeating a claim after having it questioned, it's a sign that it's time to say why you believe it. :-)

Quote:
Not really. Have you seen all of the glue code one has to maintain for the Lua interface? You'd have to do this again for every additional language.

I was mainly referring to the claim that it's "needless bloat"; it seems to me that it's needless for you, but then again you're not every potential script developer, either. But see below re: maintenance nightmare.

Quote:
But having all languages in one program just to satisfy everyone is impractical, especially with static linkage which would, as I mentioned, introduce a lot of bloat, most of which any given user won't ever use (because they use just one of those languages generally).

By your reasoning, Lua should not have been introduced because it added lots of needless bloat when people were already using other languages, and most people would not have been using Lua -- this is back when Lua was experimental in MUSHclient. Perhaps it would be used more extensively if it had a first-class binding like Lua did. Had Python been done first, Lua would be in the same boat that you've put Python in now.

Look, you don't want to use Python, but really the only argument against embedding it directly is that it would take time to do so. These arguments about "needless bloat" are entirely subjective and make sense only from the perspective of somebody who doesn't want to use Python in the first place. OK, duh. I don't see why you need to prove any point here... If somebody wants to spend the time to add the required glue, why not? It will hardly be a "maintenance nightmare" -- the maintenance is in fact rather simple, with almost all the cost entirely up-front. And besides, you won't be the one taking care of it. I guess I'm not sure why this is generating so much verbiage...

Twisol USA Thu 15 Jul 2010 05:49 AM #27

David Haley said:

Quote:
The rest of my point was that I don't think it's the right language to embed, at least in this case.

Surely, if you find yourself repeating a claim after having it questioned, it's a sign that it's time to say why you believe it. :-)

I'm sorry, have you read any of my posts? If there's something I've left out, please ask specifically.

David Haley said:

Quote:
Not really. Have you seen all of the glue code one has to maintain for the Lua interface? You'd have to do this again for every additional language.

Again, I don't think it's a good idea to support every possible language, when it's practically a given that each user will use one language and stick with it. All of the other provided languages become loose baggage.

David Haley said:

I'm talking from our present circumstances. I wasn't here when Lua was introduced. At this moment, I think Lua is the better choice. That's a highly subjective statement, but as should be extremely obvious, it is only a thought. And I've explained why I think that already.

And again, I wouldn't change it now, because people are used to it and they want to keep it. No, I'd prefer to use Lua and only Lua from the start. I can't speak directly for Nick, but I remember him saying things previously to the same effect.

David Haley USA Thu 15 Jul 2010 02:26 PM #28

Quote:
Again, I don't think it's a good idea to support every possible language, when it's practically a given that each user will use one language and stick with it. All of the other provided languages become loose baggage.

I still don't really understand why this is an argument in favor of Lua or anything really. It means that a given person will pick a given language and stick with it. OK. How is this supposed to be relevant to which languages should be added?

And by the way no need to go into hyperbole about supporting every possible language here; that's not what we're talking about after all.

Quote:
That's a highly subjective statement, but as should be extremely obvious, it is only a thought.

Well, that clears things up for me then. Your initial posts sounded like you were trying to make an objective point; now I'm satisfied. :)

By the way, yes, I actually have been reading your posts; no need to get snippy about that. :( (If I ask the question, it means that I remain unconvinced...!)

Twisol USA Thu 15 Jul 2010 07:39 PM #29

David Haley said:
I still don't really understand why this is an argument in favor of Lua or anything really. It means that a given person will pick a given language and stick with it. OK. How is this supposed to be relevant to which languages should be added?

It was more of an argument in favor of picking one language and sticking with it for an application. Even though Lua has the glue code, actual close-up integration is severely lacking. We still pass function names rather than functions to serve as callbacks, as a recent poster in another topic pointed out. It's to keep things "consistent" with the other languages, but it becomes a sub-par experience.

David Haley said:
And by the way no need to go into hyperbole about supporting every possible language here; that's not what we're talking about after all.

Meh, true, but IMHO one is enough. Begin to support more, and that opens the door for people requesting their own pet languages. And if you want to keep the API "consistent" between the languages - see above - you have to settle for the least common denominator. If you don't want to keep the API consistent, it's more work to do for each language "adapter", and a script in one language doesn't translate directly to a script in the other, at least in regards to the API in question.

David Haley said:
Well, that clears things up for me then. Your initial posts sounded like you were trying to make an objective point; now I'm satisfied. :)

*shake* It's pretty much all opinion, but based on my experience (YMMV, of course). I did explain very early on that I'm not a language fanboy, and I treat my languages as tools. I'm familiar with a good number of languages (though knowing some people, I wouldn't say "a lot"), and I have my own informal criteria for picking one for a new project.

In other words, I try not to look at everything like a nail, but that means that each language has a different place. All I'm trying to say is that I'm not sure Python is the right tool for this particular job. And I could very well be wrong!

David Haley said:
By the way, yes, I actually have been reading your posts; no need to get snippy about that. :( (If I ask the question, it means that I remain unconvinced...!)

Well, you were implying that I hadn't said anything about why I believed it, which rather bothered me as most of my posts have contained at least a small note about why I prefer Lua over other languages for embedding. No worries.