[Home] [Downloads] [Search] [Help/forum]

Gammon Software Solutions forum

See www.mushclient.com/spam for dealing with forum spam. Please read the MUSHclient FAQ!

[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  Lua
. . -> [Subject]  Garbage collection way too lazy?
Home  |  Users  |  Search  |  FAQ
Username:
Register forum user name
Password:
Forgotten password?
Garbage collection way too lazy?

[Reply to this subject]  Reply to this subject   [New subject]  Start a new subject   [Refresh] Refresh page

Posted by Larkin   (278 posts)  [Biography] bio
Date Sat 12 Feb 2011 03:55 PM (UTC)  [ quote  ]
Message
I've got a ton of RAM, so I didn't notice this at first, but a friend using my same scripts with a lot less RAM has so many problems with memory usage that he has to reload the system every 15 minutes or so.

After some investigating with a clean world and a couple of simple functions, I determined that Lua just wasn't collecting my garbage. I could set the variable to an empty table, set it to nil, make it local inside a function that returned (making it go out of scope), and the memory continued to go up, never down. Running collectgarbage("collect") manually does make it drop back down to what it was before my function to allocate, so I'm wondering if the automatic garbage collection is just not working in MUSHclient.

Anyone else have this sort of problem? If it's just me, I suppose I'll keep doing the manual garbage collection as a workaround, unless someone give me pointers on returning tables from functions and having them deallocated every time when they're not needed.
[Go to top] top
Posted by Twisol   USA  (2,230 posts)  [Biography] bio
Date Reply #1 on Sat 12 Feb 2011 11:41 PM (UTC)  [ quote  ]
Message
Any chance you could post your test scripts?
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Larkin   (278 posts)  [Biography] bio
Date Reply #2 on Sun 13 Feb 2011 12:08 AM (UTC)  [ quote  ]
Message 

function leaker()
  local something = {}
  for i=1,50000 do
    table.insert(something, "blahblahblah")
  end
  return something
end

function testleak()
  local a = leaker()
end


And then I just executed "/testleak()" on the command line a couple times and watched the memory shoot up. Now, if I put a "collectgarbage('collect')" in that function, the memory remains the same.
[Go to top] top
Posted by Twisol   USA  (2,230 posts)  [Biography] bio
Date Reply #3 on Sun 13 Feb 2011 03:47 AM (UTC)  [ quote  ]

Amended on Sun 13 Feb 2011 03:50 AM (UTC) by Twisol

Message
You're correct, but that's the behavior I'd expect. The GC doesn't run when the script ends; it runs when it needs to. Try this and watch the memory usage:

function testleak()
  for i=1, 1000 do
    local a = leaker()
  end
end


It goes up, then down, then up, etc., showing that Lua is perfectly capable of collecting garbage automatically. Now, I'd expect something different if MUSHclient itself called lua_gc() every time a script executed, but that's unnecessary overhead IMHO.

By way of analogy: The garbage trucks here only come on Wednesdays, not every time I put something in the trash.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Nick Gammon   Australia  (18,800 posts)  [Biography] bio   Forum Administrator
Date Reply #4 on Sun 13 Feb 2011 06:39 AM (UTC)  [ quote  ]
Message
Larkin said:


function leaker()
 ...
end

function testleak()
...
end


Now, if I put a "collectgarbage('collect')" in that function, the memory remains the same.


Which function? You posted two.

There are a number of points here. For one thing, the Lua garbage collection tries to adapt, I think. The first time it collects after a short time, then it collects later, and so on. It tries to balance time taken collecting with the need to reclaim memory.

Second, collecting garbage doesn't necessarily make your Windows memory go down. The C runtime libraries grab memory from a Windows pool when needed. Even if the C runtime libraries find they have some "spare" memory they don't necessarily give it back to Windows, particularly if it is fragmented.

I would be looking at your design, in particular can you re-use tables rather than always creating new ones?

Quote:

I determined that Lua just wasn't collecting my garbage.


Watching Windows memory usage isn't a reliable way of concluding that Lua isn't collecting garbage.

Anyway, the Lua garbage collector has parameters you can tweak.

Also it might be worth collecting garbage manually at a logical point (eg. in a mapper, after you draw the map). That way you always have a small overhead rather than a big pause from time to time that you notice.

Quote:

I'm wondering if the automatic garbage collection is just not working in MUSHclient ...


The Lua that comes with MUSHclient is basically standard Lua, maybe with a couple of compile options tweaked. I don't know why I would change it to disable garbage collection, and indeed I haven't.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Larkin   (278 posts)  [Biography] bio
Date Reply #5 on Sun 13 Feb 2011 02:47 PM (UTC)  [ quote  ]
Message
I didn't mean to imply you had messed it up on purpose, but my experience lately has been that the garbage collection never reduces my memory unless I run it manually, and I have no idea why that might be.

I can run the testleak function a dozen times and watch the memory allocation climb by several megabytes, but it never cleans it up. Is the threshold really that high?

And, if there's a better way to handle this sort of thing, I'm open to suggestions. I just don't want to have to make some sort of ugly hack to circumvent this problem when it could be handled more elegantly, I guess.
[Go to top] top
Posted by Twisol   USA  (2,230 posts)  [Biography] bio
Date Reply #6 on Sun 13 Feb 2011 08:44 PM (UTC)  [ quote  ]

Amended on Sun 13 Feb 2011 08:53 PM (UTC) by Twisol

Message
Larkin said:
I can run the testleak function a dozen times and watch the memory allocation climb by several megabytes, but it never cleans it up. Is the threshold really that high?

My MUSHclient process started at 9,000KB or so. It took about 50,000KB more before it stopped and collected (though of course the Task Manager only has 1 second of resolution).

I also found an old Lua-L email from 2003 mentioning a "used x 2" threshold [1].

[EDIT]: It seems that Lua 5.0 had C API functions for setting the threshold, but I can't find any trace of them in 5.1...

[1]: http://lua-users.org/lists/lua-l/2003-02/msg00540.html
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Larkin   (278 posts)  [Biography] bio
Date Reply #7 on Sun 13 Feb 2011 08:49 PM (UTC)  [ quote  ]
Message
That's not so much on a system with gigabytes of RAM, but my friend has 512MB, and he runs out faster than me. My MUSHclient usually sits around 90MB with my one world open (I have a lot of settings), and I suppose I expected it to clean up a little more aggressively than every 40-50MB.

Anyway, since I have access to the Lua garbage collector, I'm fine with just doing the collection a bit more often myself now. No big deal. I was wondering if others had bumped into the same sort of issue, like maybe it's a scripting inefficiency on my part.

Thanks.
[Go to top] top
Posted by Twisol   USA  (2,230 posts)  [Biography] bio
Date Reply #8 on Sun 13 Feb 2011 08:54 PM (UTC)  [ quote  ]
Message
It might also be useful to note that MUSHclient gives each plugin (as well as the world itself) its own Lua VM, meaning none of them know about eachothers' memory usage. If they all sit just below the threshold, you could definitely have a memory problem.
'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
[Go to top] top
Posted by Larkin   (278 posts)  [Biography] bio
Date Reply #9 on Sun 13 Feb 2011 08:56 PM (UTC)  [ quote  ]
Message
I only use plugins when that's the only way to get access to certain functionality, like GMCP telnet suboptions. My mapper is entirely main world script, and it's definitely the big memory hog.
[Go to top] top
Posted by Nick Gammon   Australia  (18,800 posts)  [Biography] bio   Forum Administrator
Date Reply #10 on Sun 13 Feb 2011 09:28 PM (UTC)  [ quote  ]

Amended on Sun 13 Feb 2011 09:29 PM (UTC) by Nick Gammon

Message
Larkin said:


function leaker()
  local something = {}
  for i=1,50000 do
    table.insert(something, "blahblahblah")
  end
  return something
end

function testleak()
  local a = leaker()
end




OK, if you run "testleak ()" there is no reason, yet, for it to collect the garbage. Inside leaker it is validly using a lot of memory. After you run testleak the variable "a" goes out of scope, but the script ends. You would need to do something else afterwards to trigger off garbage collection, and I'm not sure what that would be.

Some tests seem to indicate that just running some simple scripts don't do it (of course, doing collectgarbage does).

Judging by what little I have read, there would be a trigger point. In your case you may have used 1 Mb of memory, the trigger point might be when another 1 Mb is used.

I seem to recall, but can't find it right now, that in my mapper I just collected garbage after drawing the map. All those temporary tables got cleaned up then. I don't see any harm in that, as you know you have used temporary things, you know you are finished with them, and a slight extra delay after drawing the map won't annoy people too much. However running out of memory later on, will.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by David Haley   USA  (3,881 posts)  [Biography] bio   Moderator
Date Reply #11 on Tue 15 Feb 2011 08:48 PM (UTC)  [ quote  ]
Message
I suspect that you wouldn't actually run out of memory; the GC would kick in if it had to (but the penalty in delay would be higher). However, you would definitely get the perception of wasteful, ever-growing memory usage, which freaks some people out.
David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top
Posted by Larkin   (278 posts)  [Biography] bio
Date Reply #12 on Tue 15 Feb 2011 08:53 PM (UTC)  [ quote  ]
Message
I believe what my friend was seeing was overall system memory usage growing beyond the capacity for his physical memory, so he was really bogged down when it started paging to virtual memory all the time.
[Go to top] top
Posted by Nick Gammon   Australia  (18,800 posts)  [Biography] bio   Forum Administrator
Date Reply #13 on Tue 15 Feb 2011 08:56 PM (UTC)  [ quote  ]
Message
What version of MUSHclient did he have? Some older versions had a bug about normal memory leaking.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

3,565 views.

[Reply to this subject]  Reply to this subject   [New subject]  Start a new subject   [Refresh] Refresh page

Go to topic:           Search the forum

[Go to top] top

[Home]

Written by Nick Gammon - 5K

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( http://www.gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Internet Contents Rating Association (ICRA) - 2K]    [Web site powered by FutureQuest.Net]