Garbage collection way too lazy?

I've got a ton of RAM, so I didn't notice this at first, but a friend using my same scripts with a lot less RAM has so many problems with memory usage that he has to reload the system every 15 minutes or so.

After some investigating with a clean world and a couple of simple functions, I determined that Lua just wasn't collecting my garbage. I could set the variable to an empty table, set it to nil, make it local inside a function that returned (making it go out of scope), and the memory continued to go up, never down. Running collectgarbage("collect") manually does make it drop back down to what it was before my function to allocate, so I'm wondering if the automatic garbage collection is just not working in MUSHclient.

Anyone else have this sort of problem? If it's just me, I suppose I'll keep doing the manual garbage collection as a workaround, unless someone give me pointers on returning tables from functions and having them deallocated every time when they're not needed.

Any chance you could post your test scripts?

And then I just executed "/testleak()" on the command line a couple times and watched the memory shoot up. Now, if I put a "collectgarbage('collect')" in that function, the memory remains the same.

You're correct, but that's the behavior I'd expect. The GC doesn't run when the script ends; it runs when it needs to. Try this and watch the memory usage:



It goes up, then down, then up, etc., showing that Lua is perfectly capable of collecting garbage automatically. Now, I'd expect something different if MUSHclient itself called lua_gc() every time a script executed, but that's unnecessary overhead IMHO.

By way of analogy: The garbage trucks here only come on Wednesdays, not every time I put something in the trash.

Which function? You posted two.

There are a number of points here. For one thing, the Lua garbage collection tries to adapt, I think. The first time it collects after a short time, then it collects later, and so on. It tries to balance time taken collecting with the need to reclaim memory.

Second, collecting garbage doesn't necessarily make your Windows memory go down. The C runtime libraries grab memory from a Windows pool when needed. Even if the C runtime libraries find they have some "spare" memory they don't necessarily give it back to Windows, particularly if it is fragmented.

I would be looking at your design, in particular can you re-use tables rather than always creating new ones?



Watching Windows memory usage isn't a reliable way of concluding that Lua isn't collecting garbage.

Anyway, the Lua garbage collector has parameters you can tweak.

Also it might be worth collecting garbage manually at a logical point (eg. in a mapper, after you draw the map). That way you always have a small overhead rather than a big pause from time to time that you notice.



The Lua that comes with MUSHclient is basically standard Lua, maybe with a couple of compile options tweaked. I don't know why I would change it to disable garbage collection, and indeed I haven't.

I didn't mean to imply you had messed it up on purpose, but my experience lately has been that the garbage collection never reduces my memory unless I run it manually, and I have no idea why that might be.

I can run the testleak function a dozen times and watch the memory allocation climb by several megabytes, but it never cleans it up. Is the threshold really that high?

And, if there's a better way to handle this sort of thing, I'm open to suggestions. I just don't want to have to make some sort of ugly hack to circumvent this problem when it could be handled more elegantly, I guess.

My MUSHclient process started at 9,000KB or so. It took about 50,000KB more before it stopped and collected (though of course the Task Manager only has 1 second of resolution).

I also found an old Lua-L email from 2003 mentioning a "used x 2" threshold [1].

[EDIT]: It seems that Lua 5.0 had C API functions for setting the threshold, but I can't find any trace of them in 5.1...

[1]: http://lua-users.org/lists/lua-l/2003-02/msg00540.html

That's not so much on a system with gigabytes of RAM, but my friend has 512MB, and he runs out faster than me. My MUSHclient usually sits around 90MB with my one world open (I have a lot of settings), and I suppose I expected it to clean up a little more aggressively than every 40-50MB.

Anyway, since I have access to the Lua garbage collector, I'm fine with just doing the collection a bit more often myself now. No big deal. I was wondering if others had bumped into the same sort of issue, like maybe it's a scripting inefficiency on my part.

Thanks.

It might also be useful to note that MUSHclient gives each plugin (as well as the world itself) its own Lua VM, meaning none of them know about eachothers' memory usage. If they all sit just below the threshold, you could definitely have a memory problem.

I only use plugins when that's the only way to get access to certain functionality, like GMCP telnet suboptions. My mapper is entirely main world script, and it's definitely the big memory hog.

OK, if you run "testleak ()" there is no reason, yet, for it to collect the garbage. Inside leaker it is validly using a lot of memory. After you run testleak the variable "a" goes out of scope, but the script ends. You would need to do something else afterwards to trigger off garbage collection, and I'm not sure what that would be.

Some tests seem to indicate that just running some simple scripts don't do it (of course, doing collectgarbage does).

Judging by what little I have read, there would be a trigger point. In your case you may have used 1 Mb of memory, the trigger point might be when another 1 Mb is used.

I seem to recall, but can't find it right now, that in my mapper I just collected garbage after drawing the map. All those temporary tables got cleaned up then. I don't see any harm in that, as you know you have used temporary things, you know you are finished with them, and a slight extra delay after drawing the map won't annoy people too much. However running out of memory later on, will.

I suspect that you wouldn't actually run out of memory; the GC would kick in if it had to (but the penalty in delay would be higher). However, you would definitely get the perception of wasteful, ever-growing memory usage, which freaks some people out.

I believe what my friend was seeing was overall system memory usage growing beyond the capacity for his physical memory, so he was really bogged down when it started paging to virtual memory all the time.

What version of MUSHclient did he have? Some older versions had a bug about normal memory leaking.