Unit testing?

Considering the complexity of the codebase and the ease of which one can introduce bugs, I think it could be worthwhile to look into setting up a unit testing suite of some sort. I found one, UnitTest++ [1], that looks pretty nice. I haven't looked into any others.

Unit testing could make producing features and fixes easier, as well as doing refactorings(!), because we don't necessarily have to open up MUSHclient and tinker around and maybe miss something. It can be automated too, and the UnitTest++ documentation shows how it can be run automatically after every build. What are your thoughts on this?

[1] http://unittest-cpp.sourceforge.net/

OK. Let me explain it like this. For unit testing to work you need to anticipate the problems, in order to catch them.

For example, in the recent thread you started, you found a problem (that had not been previously reported) where someone might set "debug = nil".

So to detect that issue, you needed to set up a unit test where you set debug to nil, and then tried a script. Otherwise it would have slipped through.

But how far does that go? Do you also need to test "debug = 42", or "debug = false"? What about "debug = print"?

And in the case of MUSHclient, the "valid output" is visible on the screen. So how do you detect if colours are wrong? Or the occasional line is skipped? How do you detect that when you enter an alias the wrong response is sent?

I think this just replaces one problem with another one.

Well, I guess that's the problem. Everything in MUSHclient is very coupled. Unit testing works best when you can isolate a single component on all sides. Given the telnet component, for example, one might pass in a variety of different input - probably clustering MXP, ANSI colors, etc. into their own groups - and catch the processed output on the other end, inspecting the results to ensure correctness.

I guess it would be too much work to re-architect MUSHclient this way, which is a pity, because it really helps maintain correctness.

As Nick said, Unit Testing is useful on tier 2/tier 3 components... not clients.

Mushclient is a far too established codebase, and, as you mentioned, to tightly coupled, to easily start adding unit tests too. You could try for some NUTs (Not a Unit Test), but the benefit would be incredibly marginal compared to the gain. +1 for enthusiasm though :P

In other words... enterprise software? Would you care to explain the prevalence of unit testing elsewhere, such as Ruby? TDD (Test-Driven Development) is an actual methodology for developing software. It's not just some afterthought.


Yeah, I get that MUSHclient is too complex for this at this point. Oh well.

You can't really write unit tests for end-user interaction like in GUIs. It's not a question of language or of enterprise software, nor is it a question of the codebase being big. It's a question of what the code is doing.

It's easy to write unit tests for code that does manipulation on data structures because the inputs and outputs are clear. For example, you can verify that a container contains what it should after several pokes at it, or even that a hash table maintains a certain distribution of elements across buckets (to make sure that new hash code algorithms aren't screwing up). It's still pretty easy to write unit tests on image manipulation code (like rotations, or color shifting), as long as you know the beginning and end images.

It's very hard to write a unit test that guarantees that a free-form scripting environment will always do "the right thing" because as Nick pointed out the space of possible inputs is basically unbounded.

If you can show me some large piece of GUI or other user-oriented software that uses automated unit tests for its user-sensitive code, that would be nice. In the meantime I think that, no offense, you are using a buzzword and a fad and applying it everywhere without having seen it actually work.

Believe me, where I work we have -ton- of unit tests for extremely large GUIs. No need to be so harsh, unit tests are always exciting the first time around :)

Saying unit testing is not useful on clients is also ridiculous. Clients should be architected and componentitized like any other software. It is harder to apply unit tests to top level GUI components, yes, but every client has lower level building blocks. Ie, for MUSHclient, a couple that spring to mind are scripting, alias/trigger handling, buffer handling, etc.

In any case, unit tests are much easier to apply when writing new code, and frankly aren't worth the effort in this case. I've done TDD before, and while it's a useful learning tool it's not something I would apply 100% in practice. It's a great way to teach where you will get the best bang for your buck when applying tests, how to architect components in a testable way, and what the correct tradeoffs between correctness and testing are.

For a piece of software like SQLite, reliability and regressions are of major concern. For MUSHclient, not as much.

Well, I'm glad that I didn't actually say that, then...!


This is exactly the kind of stuff that I said unit tests are appropriate for: lower-level data structures and other cases where input and output conditions are very clear. I get the impression that you might have read my post a little too quickly. :-)

Indeed. I never explicitly suggested unit-testing GUIs. Rather, I suggest integrating unit tests where they make sense, such as the output buffer code or the telnet layer code. That's assuming architectural issues are overcome, which isn't likely without a lot of work.


Sure. The point is, if you can isolate singular components on all sides, you can test the behavior of that specific component without also relying on side effects from other parts of the code. With the scripting engine, you might write a TestScriptEngine that implements no scripting engine, but simply calls the scripting interface methods and checks that they work with a variety of input.


Yeah, it's certainly possible. I don't have much hands-on experience with unit testing. But take into consideration that this isn't out of the blue: I have (recent) experience with accidentally creating bugs, with no easy way to tell if I broke something. I wanted to figure out a way to make it easier to make reliable changes. Unit testing seemed very appropriate. I am not - let me repeat that - I am not throwing things at random against a wall to see what sticks.


Indeed; WillFa did.

I certainly see unit testing as being useful (and I think it is in fact used) for things like PCRE or SQLite3. This is because you can set up test cases (eg. regular expressions and their data, and expected results). It would also be very useful there because you would want to ensure that internal changes to not break something you didn't expect.

A quick read here:

http://en.wikipedia.org/wiki/Unit_testing

... reveals amongst other things:


Unit testing limitations

Testing cannot be expected to catch every error in the program: it is impossible to evaluate every execution path in all but the most trivial programs. The same is true for unit testing. Additionally, unit testing by definition only tests the functionality of the units themselves. Therefore, it will not catch integration errors or broader system-level errors (such as functions performed across multiple units, or non-functional test areas such as performance). Unit testing should be done in conjunction with other software testing activities. Like all forms of software testing, unit tests can only show the presence of errors; they cannot show the absence of errors.

Software testing is a combinatorial problem. For example, every boolean decision statement requires at least two tests: one with an outcome of "true" and one with an outcome of "false". As a result, for every line of code written, programmers often need 3 to 5 lines of test code.[


A quick scan of the MUSHclient source shows that there are about 116,000 lines of C++ code. Now not all that is procedural code (I excluded the .h files), some would be comments.

However you can see that even if it could be done, doing 3 to 5 lines of test code, for 100,000 lines of target code, would be a massive undertaking.

And again, in the case that kicked off this thread, there was a bug in the case of "debug = nil" which wasn't handled correctly. However you need to think to make that test case. If you don't think to test it, you won't catch it. Next time someone might do "package = nil" and trigger a different bug.

Well, unit tests aren't exactly write-once. You're always going to miss some cases. When you find them, you add them. The ones you do think of provide good regression testing, too.

I'm not suggesting we add this to all of MUSHclient. I just think it might be a good idea for code that's touched a lot. If a refactoring is being done somewhere, that's a good place to do it too.

He didn't say it when he replied to WillFa, and he was replying to me, and other comments were directed at me, so it sounded like that was too...
I'll reply more later if necessary, for now I have to run.

Guilty of being lazy... I was responding to you and WillFa at the same time, and I probably should have quoted :)