[Home] [Downloads] [Search] [Help/forum]

Gammon Software Solutions forum

See www.mushclient.com/spam for dealing with forum spam. Please read the MUSHclient FAQ!

[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  Lua
. . -> [Subject]  Serializing table data into a string
Home  |  Users  |  Search  |  FAQ
Username:
Register forum user name
Password:
Forgotten password?

Serializing table data into a string

This subject is now closed.   [New subject]  Start a new subject   [Refresh] Refresh page


Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Thu 02 Dec 2004 09:49 PM (UTC)  quote  ]

Amended on Sun 13 Nov 2005 10:03 PM (UTC) by Nick Gammon

Message
One of the nice things about using Lua is the ability to create nested tables. Here is an example:


mobs = {}  -- create mobs table

mobs.kobold = {
  name = 'killer',
  hp = 22,
  gold = 5,
  location = 'city square',
  treasure = { "sword", "gold", "helmet" } -- sub table
  }

-- and another one ...

mobs.worm = {
  name = 'gordon',
  hp = 4,
  gold = 15,
  location = 'underground',
  treasure = { "food", "knife" },
  attacks = { "bite", "poison" }
  }



The example above creates a single table (mobs) with sub-tables (kobold and worm) and inside those some other sub-tables again (treasure and attacks).

Such tables are all very well in Lua, but how do you save them from one MUSHclient session to another? The least tedious way is to write a recursive serializer, an example of which is below. This is based on chapter 12.1.2 of the Lua manual, however it has been adapted to generate a string rather than writing to a file.


-- ----------------------------------------------------------
-- serializer
-- See "Programming In Lua" chapter 12.1.2.
-- Also see forum thread:
--   http://www.gammon.com.au/forum/bbshowpost.php?bbsubject_id=4960
-- ----------------------------------------------------------
function basicSerialize (o)
  if type(o) == "number" or type(o) == "boolean" then
    return tostring(o)
  else   -- assume it is a string
    return string.format("%q", o)
  end
end -- basicSerialize 

--
-- Lua keywords might look OK to not be quoted as keys but must be.
-- So, we make a list of them.
--

lua_reserved_words = {}

for _, v in {
    "and", "break", "do", "else", "elseif", "end", "false", 
    "for", "function", "if", "in", "local", "nil", "not", "or", 
    "repeat", "return", "then", "true", "until", "while"
            } do lua_reserved_words [v] = true end

-- ----------------------------------------------------------
-- save one variable (calls itself recursively)
-- ----------------------------------------------------------
function save (name, value, out, indent, saved)
  saved = saved or {}       -- initial value
  indent = indent or 0      -- start indenting at zero cols
  local iname = string.rep (" ", indent) .. name -- indented name
  if type(value) == "number" or 
     type(value) == "string" or
     type(value) == "boolean" then
    table.insert (out, iname .. " = " .. basicSerialize(value))
  elseif type(value) == "table" then
    if saved[value] then    -- value already saved?
      table.insert (out, iname .. " = " .. saved[value])  -- use its previous name
    else
      saved[value] = name   -- save name for next time
      table.insert (out, iname .. " = {}")   -- create a new table
      for k,v in pairs(value) do      -- save its fields
        local fieldname 
        if type (k) == "string"
           and string.find (k, "^[_%a][_%a%d]*$") 
           and not lua_reserved_words [k] then
          fieldname = string.format("%s.%s", name, k)
        else
          fieldname  = string.format("%s[%s]", name,
                                        basicSerialize(k))  
        end
        save(fieldname, v, out, indent + 2, saved)
      end
    end
  else
    error("cannot save a " .. type(value))
  end
end  -- save 

-- ----------------------------------------------------------
-- Serialize a variable or nested set of tables:
-- ----------------------------------------------------------

--[[

  Example of use:

  SetVariable ("mobs", serialize ("mobs"))  --> serialize mobs table
  loadstring (GetVariable ("mobs")) ()  --> restore mobs table 

--]]

function serialize (what, v)
  v = v or _G [what]  -- default to "what" in global namespace

  assert (type (what) == "string", 
          "Argument to serialize should be the *name* of a variable")
  assert (v, "Variable '" .. what .. "' does not exist")

  local out = {}  -- output to this table
  save (what, v, out)   -- do serialization
  return table.concat (out, "\r\n")  -- turn into a string
end -- serialize



Most of the work is done in the "save" function, which recursively saves variables. To package the results into a string, we initially save to a table (out) which is then concatenated into a single string, with carriage-return, linefeeds between each line.



Example of saving a table

Assuming we have the "mobs" table shown at the top, all we have to do is this to save it:


SetVariable ("mobs", serialize ("mobs"))


After doing that, we can edit the "mobs" variable in MUSHclient:


mobs = {}
  mobs.worm = {}
    mobs.worm.attacks = {}
      mobs.worm.attacks[1] = "bite"
      mobs.worm.attacks[2] = "poison"
    mobs.worm.treasure = {}
      mobs.worm.treasure[1] = "food"
      mobs.worm.treasure[2] = "knife"
    mobs.worm.name = "gordon"
    mobs.worm.gold = 15
    mobs.worm.location = "underground"
    mobs.worm.hp = 4
  mobs.kobold = {}
    mobs.kobold.treasure = {}
      mobs.kobold.treasure[1] = "sword"
      mobs.kobold.treasure[2] = "gold"
      mobs.kobold.treasure[3] = "helmet"
    mobs.kobold.name = "killer"
    mobs.kobold.gold = 5
    mobs.kobold.location = "city square"
    mobs.kobold.hp = 22



The serialize function will handle shared sub-tables, or even tables that refer to themselves. It does that by keeping track of which tables have been created, and not creating them again.



Example of loading a table

Once we have the table saved (which we might do in a plugin's OnPluginSaveState function), then we need to be able to restore the table next time we want to use it.


loadstring (GetVariable ("mobs"))




Serializing local variables

The serialize function has an optional second argument, which is the contents of the variable. If the variable is a global variable this is not needed, as using the global variable of the supplied name is the default. However if you want to serialize a local variable then you would need to specify its value as well as its name.

eg.


do
local test = { "a", "b", "c" }

print (serialize ("test", test))
end


In the example above "test" is a variable that is not in the global namespace, and thus its name and value need to be supplied to the serialize function.


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #1 on Mon 13 Dec 2004 10:21 PM (UTC)  quote  ]
Message
One problem with serializing is when to do it. MUSHclient version 3.60 introduces some help in this respect.

There is a new script callback (in the scripting configuration) - World Save.

You can put a function there that is called just before saving a world to disk. This is a good time to serialize internal variables into MUSHclient variables. For example:


function OnWorldSave ()
  SetVariable ("skills", serialize ('skills'))
end -- function


However another problem arises. If you haven't changed anything else in the world, MUSHclient will not offer to save it, and thus the serializing will not be done. Thus, you need to tell it when you have changed a variable that needs to be saved. You can now do this:


SetChanged (true)  -- world has changed


You might do that in a trigger that updates internal variables. This will not cause the world to be saved, it will simply set a flag that causes it to be saved at world close time (assuming you go ahead and save it).

You can test this flag with GetInfo (111).

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by RasmusKL   (15 posts)  [Biography] bio
Date Reply #2 on Sat 16 Dec 2006 11:48 AM (UTC)  quote  ]
Message
Hey Nick,

The serialization code works fine, except when I try to use it with saving actual nested tables (I can save the whole thing though).

Example:

t = { }
t.sub = { }

It's not possible to serialize("t.sub"), because it can't look it up in the global namespace, should I split the string over . and look them up individually, or is there a smarter way?

Thanks,
- Rasmus.

[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #3 on Sat 16 Dec 2006 07:00 PM (UTC)  quote  ]
Message
The newer version of the serialization code handles this, because you can supply an argument to it, which is the actual table to be serialized. This ships as a separate file (serialize.lua) in recent versions of MUSHclient. From the instructions inside that file:


You can also supply the actual variable if the variable to be serialized does
  not exist in the global namespace (for instance, if the variable is a local 
  variable to a function). eg.

     require "serialize"
     do
      local myvar = { 1, 2, 8, 9 }
      print (serialize.save ("myvar", myvar))
    end

  In this example, without supplying the location of "myvar" the serialize would fail
  because it would not be found in the _G namespace.


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #4 on Thu 17 May 2007 06:18 AM (UTC)  quote  ]
Message
There is an example of loading and saving a Lua table in a plugin in this post:

http://www.gammon.com.au/forum/?id=7855

In the OnPluginInstall function the MUSHclient variable is fetched and converted into the appropriate Lua table by using loadstring.

In the OnPluginSaveState function (which is called when the state file is about to be written), the Lua table is converted back into a string by calling serialize.save.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #5 on Thu 19 Jul 2007 01:07 AM (UTC)  quote  ]

Amended on Thu 19 Jul 2007 01:08 AM (UTC) by Nick Gammon

Message
The serialize.lua file distributed with MUSHclient is handy for turning tables into strings, but can get a bit wordy for simple tables.

For example, with the mobs table shown in the original post in this thread, calling serialize.save produces fairly wordy output, like this:


print ((serialize.save ("mobs")))

--> Output

mobs = {}
  mobs.kobold = {}
    mobs.kobold.treasure = {}
      mobs.kobold.treasure[1] = "sword"
      mobs.kobold.treasure[2] = "gold"
      mobs.kobold.treasure[3] = "helmet"
    mobs.kobold.name = "killer"
    mobs.kobold.gold = 5
    mobs.kobold.location = "city square"
    mobs.kobold.hp = 22
  mobs.worm = {}
    mobs.worm.attacks = {}
      mobs.worm.attacks[1] = "bite"
      mobs.worm.attacks[2] = "poison"
    mobs.worm.treasure = {}
      mobs.worm.treasure[1] = "food"
      mobs.worm.treasure[2] = "knife"
    mobs.worm.name = "gordon"
    mobs.worm.gold = 15
    mobs.worm.location = "underground"
    mobs.worm.hp = 4


For tables that do not have cycles (that is they do not refer to themselves), or that do not refer to other tables, a simpler serialization is all that is required.


print ("mobs = " .. serialize.save_simple (mobs))

--> Output

mobs = {
  kobold = {
    treasure = {
      [1] = "sword",
      [2] = "gold",
      [3] = "helmet",
      },
    name = "killer",
    gold = 5,
    location = "city square",
    hp = 22,
    },
  worm = {
    attacks = {
      [1] = "bite",
      [2] = "poison",
      },
    treasure = {
      [1] = "food",
      [2] = "knife",
      },
    name = "gordon",
    gold = 15,
    location = "underground",
    hp = 4,
    },
  }


The function serialize.save_simple is simply passed a variable (not the name of a variable), and thus it doesn't know the top-level variable name. Hence I added "mobs = " to the print statement.

I think this looks neater for simple tables, and you can always use the standard serialization for more complex ones.

Below is the new serialize.lua file contents:


-- serialize.lua

module (..., package.seeall)

-- ----------------------------------------------------------
-- serializer
-- See "Programming In Lua" chapter 12.1.2.
-- Also see forum thread:
--   http://www.gammon.com.au/forum/?id=4960
-- ----------------------------------------------------------

--[[

  Example of use:

  require "serialize"
  SetVariable ("mobs", serialize.save ("mobs"))  --> serialize mobs table
  loadstring (GetVariable ("mobs")) ()  --> restore mobs table 

  If you need to serialize two tables where subsequent ones refer to earlier ones
  you can supply your own "saved tables" variable, like this:

    require "serialize"
    result, t = serialize.save ("mobs")
    result = result .. "\n" .. serialize.save ("quests", nil, t)

  In this example the serializing of "quests" also knows about the "mobs" table
  and will use references to it where necessary.  

  You can also supply the actual variable if the variable to be serialized does
  not exist in the global namespace (for instance, if the variable is a local 
  variable to a function). eg.

     require "serialize"
     do
      local myvar = { 1, 2, 8, 9 }
      print (serialize.save ("myvar", myvar))
    end

  In this example, without supplying the location of "myvar" the serialize would fail
  because it would not be found in the _G namespace.

  ----- Added on 19 July 2007:
  
  You can now do a "simple save" which is intended for tables without cycles. That is,
  tables, that do not refer to other tables. This is appropriate for "simple" data, like
  a straight table of keys/values, including subtables.
  
  For a simple save, all you need to do is supply the value, like this:
  
  print (serialize.save_simple ({ foo = 22, bar = "hi", t = { s = 9, k = 22 } }))
  
  This produces:
  
  {
  t = {
    s = 9,
    k = 22,
    },
  bar = "hi",
  foo = 22,
  }
  
--]]

local save_item  -- forward declaration, function appears near the end
local save_item_simple

function save (what, v, saved)

  saved = saved or {} -- initial table of tables we have already done
  v = v or _G [what]  -- default to "what" in global namespace

  assert (type (what) == "string", 
          "1st argument to serialize.save should be the *name* of a variable")
  
  assert (v, "Variable '" .. what .. "' does not exist")

  assert (type (saved) == "table" or saved == nil, 
          "3rd argument to serialize.save should be a table or nil")

  local out = {}  -- output to this table
  save_item (what, v, out, 0, saved)   -- do serialization
  return table.concat (out, "\n"), saved  -- turn into a string (also return our table)
end -- serialize.save

function save_simple (v)
  local out = {}  -- output to this table
  save_item_simple (v, out, 2)   -- do serialization
  return table.concat (out)   -- turn into a string 
end -- serialize.save_simple

--- below are local functions for this module -------------

local function basicSerialize (o)
  if type(o) == "number" or type(o) == "boolean" then
    return tostring(o)
  else   -- assume it is a string
    return string.format("%q", o)
  end
end -- basicSerialize 

--
-- Lua keywords might look OK to not be quoted as keys but must be.
-- So, we make a list of them.
--

local lua_reserved_words = {}

for _, v in ipairs ({
    "and", "break", "do", "else", "elseif", "end", "false", 
    "for", "function", "if", "in", "local", "nil", "not", "or", 
    "repeat", "return", "then", "true", "until", "while"
            }) do lua_reserved_words [v] = true end

-- ----------------------------------------------------------
-- save one variable (calls itself recursively)
-- 
-- Modified on 23 October 2005 to better handle keys (like table keys)
-- ----------------------------------------------------------
function save_item (name, value, out, indent, saved)  -- important! no "local" keyword
  local iname = string.rep (" ", indent) .. name -- indented name

  -- numbers, strings, and booleans can be simply serialized

  if type (value) == "number" or 
     type (value) == "string" or
     type (value) == "boolean" then
    table.insert (out, iname .. " = " .. basicSerialize(value))

  -- tables need to be constructed, unless we have already done it

  elseif type (value) == "table" then
    if saved[value] then    -- value already saved?
      table.insert (out, iname .. " = " .. saved[value])  -- use its previous name
    else

  -- remember we have created this table so we don't do it twice

      saved [value] = name   -- save name for next time

  -- make the table constructor, and recurse to save its contents
  

      assert (string.find (name, '^[_%a][_%a%d%.%[%]" ]*$') 
              and not lua_reserved_words [name], 
              "Invalid name '" .. name .. "' for table")

      
      table.insert (out, iname .. " = {}")   -- create a new table

      for k, v in pairs (value) do      -- save its fields
        local fieldname 

        -- if key is a Lua variable name which is not a reserved word
        -- we can express it as tablename.keyname

        if type (k) == "string"
           and string.find (k, "^[_%a][_%a%d]*$") 
           and not lua_reserved_words [k] then
          fieldname = string.format("%s.%s", name, k)

        -- if key is a table itself, and we know its name then we can use that
        --  eg. tablename [ tablekeyname ]

        elseif type (k) == "table" and saved[k] then
          fieldname = string.format("%s[%s]", name, saved [k]) 

        -- if key is an unknown table, we have to raise an error as we cannot
        -- deduce its name
 
        elseif type (k) == "table" then
          error ("Key table entry " .. tostring (k) .. 
                 " in table " .. name .. " is not known")

        -- if key is a number or a boolean it can simply go in brackets,
        -- like this:  tablename [5] or tablename [true]

        elseif type (k) == "number" or type (k) == "boolean" then
          fieldname = string.format("%s[%s]", name, tostring (k))

        -- now key should be a string, otherwise an error
 
        elseif type (k) ~= "string" then
          error ("Cannot serialize table keys of type '" .. type (k) ..
                 "' in table " .. name)

        -- if key is a non-variable name (eg. "++") then we have to put it
        -- in brackets and quote it, like this:  tablename ["keyname"]

        else
          fieldname  = string.format("%s[%s]", name,
                                        basicSerialize(k))  
        end

        -- now we have finally worked out a suitable name for the key,
        -- recurse to save the value associated with it

        save_item(fieldname, v, out, indent + 2, saved) 
      end
    end

  -- cannot serialize things like functions, threads

  else
    error ("Cannot serialize '" .. name .. "' (" .. type (value) .. ")")
  end  -- if type of variable
end  -- save_item 

-- saves tables *without* cycles (produces smaller output)
function save_item_simple (value, out, indent)
  -- numbers, strings, and booleans can be simply serialized

  if type (value) == "number" or 
     type (value) == "string" or
     type (value) == "boolean" then
    table.insert (out, basicSerialize(value))
  elseif type (value) == "table" then
    table.insert (out, "{\n")

    for k, v in pairs (value) do      -- save its fields
      table.insert (out, string.rep (" ", indent))
      if not string.find (k, '^[_%a][_%a%d]*$') 
         or lua_reserved_words [k] then
           table.insert (out, "[" .. basicSerialize (k) .. "] = ")
      else
        table.insert (out, k .. " = ")
      end -- if
      save_item_simple (v, out, indent + 2)   
      table.insert (out, ",\n")
    end -- for each table item
      
    table.insert (out, string.rep (" ", indent) .. "}")
    
  -- cannot serialize things like functions, threads

  else
    error ("Cannot serialize " .. type (value))
  end  -- if type of variable
  
end -- save_item_simple


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #6 on Wed 13 Feb 2008 04:29 AM (UTC)  quote  ]

Amended on Wed 13 Feb 2008 04:33 AM (UTC) by Nick Gammon

Message
To summarize, here is some sample code that could be used in a plugin to serialize a table called "my_variables".

There are two parts to this:


  • In OnPluginSaveState we convert the Lua table into a string, which is saved in a MUSHclient variable.

  • In OnPluginInstall (which is called when the plugin is loaded) we use loadstring to re-parse the Lua table and convert it back from the string variable into an actual Lua table.

    In case the variable does not exist (eg. the first time the plugin is used) the table is set to an empty table first.

    We use "assert" to check for any parsing errors when scanning the Lua table.



require "serialize"  -- needed to serialize table to string

my_variables = {}  -- ensure table exists, if not loaded from variable

-- on plugin install, convert variable into Lua table

function OnPluginInstall ()
  assert (loadstring (GetVariable ("my_variables") or "")) ()
end -- function OnPluginInstall

-- on saving state, convert Lua table back into string variable

-- save_simple is for simple tables that do not have cycles (self-reference)
-- or refer to other tables

function OnPluginSaveState ()
  SetVariable ("my_variables", 
               "my_variables = " .. serialize.save_simple (my_variables))
end -- function OnPluginSaveState


The essential part is shown in bold - if you wanted to save/restore more tables, you would just repeat those parts.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (19,534 posts)  [Biography] bio   Forum Administrator
Date Reply #7 on Thu 22 Apr 2010 11:51 PM (UTC)  quote  ]

Amended on Thu 22 Apr 2010 11:55 PM (UTC) by Nick Gammon

Message
If you want to un-serialize the string into a local table (not the global namespace) you can do this:


local t = {}
setfenv (assert (loadstring (my_variables)), t) () 


Now whatever was serialized in the string "my_variables" is now "inside" the table t.

This method has the advantage (quite possibly a considerable advantage) that the execution environment (that is, the table t) is "sandboxed" against any attempts to make unexpected changes to the global environment in which it is running.

The setfenv effectively produces an empty global environment in which loadstring runs, so anything inside my_variables which might attempt to do something nasty (eg. os.remove "mushclient.exe") will fail, because the os table will not be present.

Since the table t contains the top-level table that was serialized, to access that table you would index into:


t.my_variables


(or whatever the name of the table was, that you serialized at serialization time). If you have no idea what that top-level table name was, you could use the next function to find it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


22,103 views.

This subject is now closed.   [New subject]  Start a new subject   [Refresh] Refresh page

Go to topic:           Search the forum


[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

[Home]

Written by Nick Gammon - 5K

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( http://www.gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Web site powered by FutureQuest.Net]