[Home] [Downloads] [Search] [Help/forum]

Gammon Software Solutions forum

See www.mushclient.com/spam for dealing with forum spam. Please read the MUSHclient FAQ!

[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  General
. . -> [Subject]  Freezing Mushclient
Home  |  Users  |  Search  |  FAQ
Username:
Register forum user name
Password:
Forgotten password?
(New message)
Subject: Freezing Mushclient
Name:
Your forum user name.
Register forum user name
Password:
Your forum password.
Forgotten password?
Message:
Message to be posted (in English, please).
Forum codes:
Check this if your message uses 'forum codes' or templates (auto-detected for new posts).
Forum codes Templates

Save this message ...

Subject review (reverse sequence)

Pages: 1 2  

Posted by Worstje   Netherlands  (867 posts)  [Biography] bio
Date Sun 05 Aug 2007 01:29 PM (UTC)  [ quote  ]

Amended on Sun 05 Aug 2007 01:32 PM (UTC) by Worstje

Message
First of all, oops. I did mean script-file in that post far above. (Edit: I keep writing down the wrong one, bad addiction, gah!)

Second, I don't have too much experience with Python globbing memory like an insane beast. I have a plugin with 650 triggers in it (and growing), and I can reload it to my hearts content. While loading, it temporarily jumps up by around 1mb in memory, but immediately drops again. I tried kicking different triggers into action (both send to script ones and script-file variety), and I can easily say that neither makes a difference for me.

Then again; this computer uses a pretty old version of Python (2.4.2) and a pretty old version of pywin32. Maybe it is a more recent bug in either of those that is causing it?
[Go to top] top
Posted by Isthiriel   (111 posts)  [Biography] bio
Date Sun 05 Aug 2007 12:15 PM (UTC)  [ quote  ]

Amended on Sun 05 Aug 2007 03:38 PM (UTC) by Isthiriel

Message
Ah, I meant for every call from MushClient to Python. And I got the 4KB number from running a 0.1s timer for a minute and noticing the memory footprint increased by about 2.5MB

I added:
def test_memory():
    for i in range(1000):
        world.GetInfo(1)


to my script file and executed it. The first run seemed to add 4KB. Subsequent runs added 0KB to the footprint. (I should note that 4KB seems to be Task Manager's minimum granularity for memory usage.)

Recompiling the script randomly added from 20KB to 280KB :( Which smells like a bug in the WSH interface that the pywin32 extension implements.



Checking the result I mentioned in the first paragraph I watched the footprint increase by 3.8MB over a minute ... and then noticed that the timer was doing a Send-to-Script rather than a straight function call (because I defined my poll() function without arguments in the script, *sigh*). Changing it to a function call stopped the leak (memory usage stable at 28636K and script calls from timer > 4000 while I typed this post).

HTH.

EDIT: (I'm using Python 2.5.1 [2007-Apr-18] and pywin32 build 210 [2006-Sep-22], which are the latest versions of each.)

EDIT[2]: (Python 2.5 because I'm a neophile and the "with" statement.)
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Sat 04 Aug 2007 12:40 AM (UTC)  [ quote  ]
Message
I tried running this small script:


for i = 1, 1000 do 
  GetInfo (1)
end -- for


This effectively calls a function 1000 times, but when using Lua and doing this, the memory footprint seemed to rise about 40 Kb.

Doing this under VBscript (which uses COM) seemed to consume about 24 Kb. This is a lot less than then 4 Kb "per call" you reported.

You could probably make a case that the memory should not rise at all, so there may be a problem here.

In the case of GetInfo (1) a string is being created, so perhaps it is not being disposed of correctly. Looking at what is happening here, GetInfo (1) does this:


VARIANT CMUSHclientDoc::GetInfo(long InfoType) 
{
VARIANT vaResult;
VariantInit(&vaResult);

  vaResult.vt = VT_EMPTY;   // in case dates are zero

  switch (InfoType)
    {
    case    1: SetUpVariantString (vaResult, m_server); break;

// blah blah for other cases

    default:
      vaResult.vt = VT_NULL;
      break;

    } // end of switch

	return vaResult;
}  // GetInfo


And SetUpVariantString does this:


void CMUSHclientDoc::SetUpVariantString (VARIANT & tVariant, const CString & strContents)
  {
  VariantClear (&tVariant);
  tVariant.vt = VT_BSTR;
  tVariant.bstrVal = strContents.AllocSysString (); 
  }   // end of CMUSHclientDoc::SetUpVariantString


So, clearly a string is being allocated, and the allocation of around 1000 strings might account for a lot of the memory usage I am seeing.

However as that string is being passed back to the script engine (the one that did the GetInfo) it seems to me that the script engine is responsible for freeing the string up when it is done with it.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Isthiriel   (111 posts)  [Biography] bio
Date Fri 03 Aug 2007 04:43 PM (UTC)  [ quote  ]
Message
Lua's garbage collector is mark-and-sweep.

AFAIK, so is Javascript's, VBScript's and PerlScript's.

If the world object is being handed to Python with a cycle and on each script invocation the world object doesn't reuse the exact same Python objects... I think that's enough to make it leak.

One of these days I'll open up the MushClient source and see if I can work out what the problem is. It's definitely an integration/interaction problem rather than something that can be laid at the feet of either side :(
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Fri 03 Aug 2007 11:30 AM (UTC)  [ quote  ]
Message
I don't create the world object each time, so I don't see how MUSHclient is doing it. If there is a fundamental problem, since all script engines, except Lua, are called using the Windows Script Interface, all languages would have a leak.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Isthiriel   (111 posts)  [Biography] bio
Date Fri 03 Aug 2007 09:21 AM (UTC)  [ quote  ]
Message
I have a similar problem with my MushClient/Python, though I don't use Send To Script at all; everything is script calls.

Each function call on the script adds about 4KB to MushClient's footprint. Python itself doesn't leak memory in my experience. I use it for automation all the time.

However, Python's garbage collector is a refcounter, not mark-and-sweep. So it can't detect cycles.

My assumption has been that the World object handed to Python is not being destructed properly because it contains a cycle.
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Fri 27 Jul 2007 09:21 PM (UTC)  [ quote  ]
Message
Quote:

Using send to script, the code is only read once ...


You mean, using the script file.

I am glad that solves the problem. "Send to script" shouldn't really consume lots of memory, but it sounds like Python is using something internally that it doesn't release.

Worstje is right, using a script file will incur less overhead. Doing "send to script" in triggers (etc.) is easier because you can see the code in front of your eyes, but less efficient.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Worstje   Netherlands  (867 posts)  [Biography] bio
Date Fri 27 Jul 2007 10:31 AM (UTC)  [ quote  ]
Message
Definately. With send to script, MUSHclient needs to parse the stuff in the Send field each time the trigger is fired (to take care of %1 and all that). After that, the code is being fed to the script engine of your world. Then the script engine has to translate it again.

Using send to script, the code is only read once and then stays within the script engine. MUSH simply tells the script engine 'execute function PromptCapture', which is definately less overhead.
[Go to top] top
Posted by Candido   USA  (78 posts)  [Biography] bio
Date Fri 27 Jul 2007 09:25 AM (UTC)  [ quote  ]
Message
I moved my prompt script to the script file and now once I fill up my output buffer, Mushclient almost completely stops taking new memory! Is calling the script file generally more efficient than using "send to script"?

Thanks for the help.
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Fri 27 Jul 2007 08:27 AM (UTC)  [ quote  ]
Message
Minimizing, I believe, flags Windows to release the actual memory, and leave the copy in the page file. Thus I would expect the page file to stay the same, but the use of "real" memory to diminish.

You could try moving a trigger that is called a lot into the script file, rather than "send to script" - this may lower whatever overhead doing the "send to script" has.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Candido   USA  (78 posts)  [Biography] bio
Date Fri 27 Jul 2007 08:17 AM (UTC)  [ quote  ]
Message
Garbage collection doesn't seem to help. In fact, Mushclient takes another few mb of memory when I collect.

Yes, I'm using "send to script".

One other thing I noticed that might be relevant. In the task manager's performance menu, there's a gauge for page file usage right below cpu usage. I'm not exactly sure what page files are, but as Mushclient's memory use increases, so does page file usage. If I minimize Mushclient, it's memory use will drop dramatically, but page file usage will remain where it was.
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Fri 27 Jul 2007 07:58 AM (UTC)  [ quote  ]
Message
There are potentially a number of places where memory could be wasted:


  • MUSHclient allocating space for a string when calling a script and it not being freed - this *may* be happening, but I am using C++ classes for this and they should be freed when the destructor is called.

  • The script engine making a copy of the strings (eg. in its own format), and not necessarily freeing it as it should.

  • Some fixed overhead incurred by the script engine - erroneously - so that it gradually uses memory

  • The script itself doing something (like storing a variable into a table) in such a way that it consumes memory.


It could make a difference the way you are calling the script - are you using "send to script", or putting a script function name in the script box, and using a script file? The "send to script" method incurs a compilation step each time, and conceivably this is wasting memory in some way.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top
Posted by Worstje   Netherlands  (867 posts)  [Biography] bio
Date Fri 27 Jul 2007 07:18 AM (UTC)  [ quote  ]
Message
Try and see what happens when you force a manual garbage collection:

/import gc
/gc.enable()
/gc.collect()

If that drops it, it just means Pythons garbage collector is not freeing up the memory. Of course, doing this from command line only affects the main script but not the plugins that may use python since those are seperate environments.
[Go to top] top
Posted by Candido   USA  (78 posts)  [Biography] bio
Date Fri 27 Jul 2007 07:17 AM (UTC)  [ quote  ]
Message
Once the buffer fills up, Mushclient stops taking memory if triggers and/or scripting are disabled. So, I guess I have a trigger that's causing this, and considering the nature of my tests (and the nature of the problem itself, actually), it would make sense if it was a prompt trigger.

That narrows it down quite a bit, but I still have to find the 'resource'. Looking at my prompt trigger, I see nothing obvious that would cause the problem, but I'm not completely sure what I'm looking for. It would be something like a line that appends to a list every match, right?
[Go to top] top
Posted by Nick Gammon   Australia  (18,770 posts)  [Biography] bio   Forum Administrator
Date Fri 27 Jul 2007 06:53 AM (UTC)  [ quote  ]
Message
One possible explanation is that something (eg. a trigger) that matches is consuming some resource when a Python script is called, that is never freed.

It would be interesting to turn off scripting and rerun your test - that would isolate it to MUSHclient itself consuming resources, or the script consuming them.

However the fact that reloading your script seems to clear things points to the script engine (or the script itself) consuming something. I am not familiar enough with Python to suggest a solution.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

7,623 views.

This is page 1, subject is 2 pages long: 1 2  [Next page]

[Reply to this subject]  Reply to this subject   [New subject]  Start a new subject   [Refresh] Refresh page

Go to topic:           Search the forum

[Go to top] top

[Home]

Written by Nick Gammon - 5K

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( http://www.gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Internet Contents Rating Association (ICRA) - 2K]    [Web site powered by FutureQuest.Net]