Patch to serialize.save

Posted by Twisol on Thu 22 Apr 2010 08:26 AM — 13 posts, 49,469 views.

USA #0
I was explaining how to use serialize.lua to a friend, and I found myself somewhat befuddled by the output of serialize.save. Specifically, I don't understand why it must declare the table globally. Here's an example output now:

tbl = {}
  tbl[1] = 1
  tbl[2] = 2
  tbl[3] = 3


The expected use of this seems to be something like:

assert(loadstring(serialized_stuff))()

That's fine, but 'tbl' is now defined globally, potentially overwriting something in the global namespace. I might also only want to use it locally within a function, also. You might well say that I should be able to trust my own serialized data, but there's no guarantee that this is strictly staying within one plugin, or that I might change a global variable name in the future, and updated plugins will use the old state and overwrite that global.

My suggested patch would output:

local tbl = {}
  tbl[1] = 1
  tbl[2] = 2
  tbl[3] = 3
return tbl


This would allow you to directly control how the data is managed:

-- if you want a global:
data = assert(loadstring(serialized_stuff))()
-- or a local:
local data = assert(loadstring(serialized_stuff))()


It also means that serialize.save doesn't need to know the name of the table to use. Since the loaded chunk has its own inner scope, you can use a specific name (in this case, tbl), and it will never clash, because the serialized table uses the "local" keyword. Thus, serialize.save can remove the "what" argument entirely.


Here's the patch. I don't have a diff tool handy, so I'll just paste by function.

function save (v)
  assert (v, "Must provide a valid variable for the 1st parameter!")

  local out = {}  -- output to this table
  save_item ("tbl", v, out, 0, {})   -- do serialization
  out[1] = "local " .. out[1]
  table.insert(out, "return tbl")
  return table.concat (out, "\n")  -- turn into a string
end -- serialize.save



Alternatively, if you want to keep backwards compatibility (I'm sure you do), ignore the above patch and just provide a serialize.load function like below

function load(name, str)
  local old  = _G[name]
  
  assert(type(name) == "string", "Argument 1 must be a string!")
  assert(type(str) == "string", "Argument 2 must be a string!")
  
  assert(loadstring(str))()
  
  local val = _G[name]
  _G[name] = old
  return val
end


I'd prefer the serialize.save improvement, but this would at least make it more natural to decode:

-- if you want a global:
data = serialize.load("name", serialized_stuff)
-- or a local:
local data = serialize.load("name", serialized_stuff)


You unfortunately still have to know the associated name ahead of time.
Australia Forum Administrator #1
You don't need to worry about that. The recommended way of loading the serialized data is into a table that has its environment changed, thus the "global" table you refer to in your post is only global to the environment. For example:


local t = {}
setfenv (assert (loadstring (serialized_stuff)), t) ()


Since you are writing a plugin, you presumably can control the way the data is read back in, and that method is the recommended way.

Your suggested patch of "local" would defeat that BTW.
Amended on Thu 22 Apr 2010 09:37 AM by Nick Gammon
USA #2
I also prefer the environment method, although I understand there's some kind of big change with how that works in Lua 5.2 (haven't looked into the details yet).
USA #3
Right then, that's a much better solution. It's not immediately obvious though, nor was it described anywhere I can see.

I'd still be in favor of adding a serialize.load convenience function like below:

function load(str)
  local t = {}
  setfenv(assert(loadstring(str)), t)()
  return select(2, next(t))
end



Nick Gammon said:
Your suggested patch of "local" would defeat that BTW.

In favor of something simpler, I feel. *shrug* It creates a local within the chunk, and returns it. That's more or less how serialize.save_simple works, too; you just have to tack on a "return " before you loadstring it. (It would be convenient to add that directly to the save_simple method instead, or provide a load_simple function for convenience)


David Haley said:
I also prefer the environment method, although I understand there's some kind of big change with how that works in Lua 5.2 (haven't looked into the details yet).

In Lua 5.2, you'd do this:

local t = {}
assert(loadin(t, serialized_data))()



EDIT: Here's the serialize.load_simple function as well.

function load_simple(v)
  return assert(loadstring("return " .. (v or "")))()
end
Amended on Thu 22 Apr 2010 06:15 PM by Twisol
USA #4
There's already a save_simple function in serialize.lua. It has the advantage of only pickling what's in between the braces


t={}
foo = serialize.save_simple(t)

print(foo)   -->  "{ }"


Note you're passing in a table and not the name of a table. The drawback of save_simple is that


t={}
t.me = t

print ( serialize.save_simple(t) )


recurses forever or until stack overflow, which ever comes first. ;)




local t,foo
t={}


foo = serialize.save_simple(t)

local newtbl
newtbl = assert(loadstring(foo))()


gives your intended result.
Australia Forum Administrator #5
Twisol said:


function load(str)
  local t = {}
  setfenv(assert(loadstring(str)), t)()
  return select(2, next(t))
end



Why not just return t?
USA #6
WillFa said:

There's already a save_simple function in serialize.lua. It has the advantage of only pickling what's in between the braces

And the disadvantage that it doesn't work with tables with cyclical references.

To the rest: I know how save_simple works, but thanks.

Nick Gammon said:

Twisol said:


function load(str)
  local t = {}
  setfenv(assert(loadstring(str)), t)()
  return select(2, next(t))
end



Why not just return t?


Because then you don't get out exactly what you put in. You get a table containing the one you put in, and you have to either know its name or use next() to get it back.


EDIT: Actually, there is a benefit to returning 't', and it's for the same reason you use your 'saved' table in serialize.save. But then you lose the advantage I just gave above, because there's no simple way to track what was just added to the table.
Amended on Thu 22 Apr 2010 10:56 PM by Twisol
Australia Forum Administrator #7
It's true that the serialize.save seems to create another level, which is in fact the thing you are serializing, eg.


mobs = {}  -- create mobs table

mobs.kobold = {
  name = 'killer',
  hp = 22,
  gold = 5,
  location = 'city square',
  treasure = { "sword", "gold", "helmet" } -- sub table
  }

-- and another one ...

mobs.worm = {
  name = 'gordon',
  hp = 4,
  gold = 15,
  location = 'underground',
  treasure = { "food", "knife" },
  attacks = { "bite", "poison" }
  }


require "serialize"
require "tprint"

tprint (mobs)

s = serialize.save ('mobs')

print (s)

local t = {}
setfenv (assert (loadstring (s)), t) ()

tprint (t)


In the case of a plugin, you serialize (say) the mobs table, and then next time around, in global namespace or somewhere else, you loadstring it. So it creates the mobs table again for you. Your objection may have some weight if you use the setfenv method, because now things are in t.mobs, but if you just do a loadstring, you don't want all the mobs data in _G you want it in mobs.

Please, enough. It is working fine. You just need to be aware of the way it works.
USA #8
Nick Gammon said:
Please, enough. It is working fine. You just need to be aware of the way it works.


Yes, it works fine... I was just trying to make it more convenient. At least make a note of the setfenv() technique somewhere, or point me to where it says it already? I couldn't find it mentioned.

EDIT: method -> technique

EDIT 2: Thanks, I appreciate it.
Amended on Thu 22 Apr 2010 11:52 PM by Twisol
Australia Forum Administrator #9
http://www.gammon.com.au/scripts/doc.php?lua=setfenv

The exact method using loadstring was mentioned in the first post here:

Template:post=10043
Please see the forum thread: http://gammon.com.au/forum/?id=10043.


I also mentioned the method more than once in the mudstandards forum.

I have added a clarifying post into the main post about serializing variables into a string:

Template:post=4960
Please see the forum thread: http://gammon.com.au/forum/?id=4960.
Amended on Fri 23 Apr 2010 01:14 AM by Nick Gammon
USA #10
Nick Gammon said:
(function=setfenv)

Says "Function setfenv does not exist.". :P

Nick Gammon said:
The exact method using loadstring was mentioned in the first post here:

(post=10043)

I also mentioned the method more than once in the mudstandards forum.

I meant specifically using setfenv() with the output from serialize.save, since serializing is a rather more common task than dealing with telopts.

Nick Gammon said:
I have added a clarifying post into the main post about serializing variables into a string:

(post=4960)

I appreciate that a lot, thanks!
Amended on Thu 22 Apr 2010 11:58 PM by Twisol
Australia Forum Administrator #11
Modified my post. It was a Lua function, not a normal function.

As for the other forum, I specifically mentioned that method for un-serializing Lua data, however that data might have been received.
USA #12
I don't think most users of MUSHclient frequent MudStandards. ;) Anyways, I more or less got what I wanted, so I'm happy.