In principle, yes, you can use whatever .dll you wish, as long as the entry points are all the same. So if you recompile Lua, and the exported API is the same, then yes, you should be able to use it instead of the version Nick ships.
Why, though? AFAIK the version Nick has is 5.0.2 which is the most recent stable version.
I see no objection to compiling your own lua.dll and using that instead. Note that MUSHclient uses lua.dll and lualib.dll which are the standard dlls made by Lua - lua.dll is the core, and lualib.dll implements the standard lua libraries.
However version 3.67 onwards of MUSHclient uses lua50.dll, a single DLL, which is the exact DLL downloaded from the SourceForge Win32 binaries page.