[Home] [Downloads] [Search] [Help/forum]


Register forum user name Search FAQ

Gammon Forum

[Folder]  Entire forum
-> [Folder]  Programming
. -> [Folder]  General
. . -> [Subject]  Writing dynamic web pages using Lua

Writing dynamic web pages using Lua

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page


Pages: 1 2  

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Fri 28 Apr 2006 03:26 AM (UTC)

Amended on Fri 28 Apr 2006 04:20 AM (UTC) by Nick Gammon

Message
So far my dynamic web pages have been written using PHP, see:


http://www.php.net/


However recently I have become interested in achieving much the same results using Lua.

There is already a Lua add-on called luacgi, however I found it a bit confusing and poorly documented, so this posting is an attempt to get you started at doing web pages in Lua from first principles.



Install Lua at your web host site

The first step would be to install Lua at your web hosting site, if it is not already installed. If necessary, obtain a copy of the source from:


http://www.lua.org/download.html


I used Lua 5.0.2, however Lua version 5.1 has now been released.

In my case, after compiling and installing, my copy of Lua ended up in:


/usr/local/bin/lua


This is important to know, as the path to the Lua executable has to be mentioned in the first line of your web script.

If it is already installed you could find its path by typing:


whereis lua




The basic components of a web page

Web pages sent from a web server to a web browser consist of two major parts, separated by a blank line:


  1. The HTTP header
  2. The body of the page


The header contains things like the type of body (text, html, etc.), cookies, date, server type and so on. This is an example header:


HTTP/1.1 200 OK
Date: Fri, 28 Apr 2006 03:14:58 GMT
Server: Apache/2.0.46 (Unix) PHP/4.3.2
Set-Cookie: foo=bar
Set-Cookie: food=apples
Connection: close
Content-Type: text/html; charset=iso-8859-1


The body of the page is the actual data the user sees (generally, the HTML code).



First example, straight text

Let's start off with a simple example dynamic page. The nice thing about using Apache is that it appears to supply most of the headers itself, so for a simple page the only header we need is:


Content-Type: text/plain


This tells the brower to interpret the body of the web page as pure text (not HTML). Thus, a minimal dynamic web page written in Lua could be this:


#! /usr/local/bin/lua

print [[
Content-Type: text/plain

Hello, world
]]



Here we are printing (sending to the browser) 3 lines:


  1. The header (content-type)
  2. The blank line which separates the header from the body)
  3. The body of the page


The first line of the file indicates the path to the Lua executable.

I saved this file into my "cgi-bin" directory, specifically to:


/usr/local/httpd/cgi-bin/test1.lua


An important step is to make the file executable, like this:


chmod a+x test1.lua


Having done that, I can now enter this URL into my web browser:


http://10.0.0.2/cgi-bin/test1.lua


My internal web server is at the local address of 10.0.0.2 - you would replace that with the address of wherever your server is located.

This may not look very exciting, but a small modification, and we can do some maths on our page. Let's make a 2 times table:


#! /usr/local/bin/lua

-- HTTP header
print [[
Content-Type: text/plain

]]

-- body of page

for i = 1, 10 do
  print (i, i * 2)
end -- for



If I run this I see a page appear in my browser with this in it:


1	2
2	4
3	6
4	8
5	10
6	12
7	14
8	16
9	18
10	20


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #1 on Fri 28 Apr 2006 04:06 AM (UTC)

Amended on Sat 01 Jan 2011 04:44 AM (UTC) by Nick Gammon

Message
Getting data from a web page

The simple example above will always do the same thing, however often we want to let the user specify further information. For example in the forum here, we specify which forum posting we want to see, or when we make a new posting, the new message has to be sent to the web server.

There are four ways we can send data to and from web servers (apart from the things in the header like the date):


  1. The URL path itself

  2. The query string

  3. Cookies

  4. Form data


Let's examine each of these to see how they are used and how they work. To do that we'll make a new Lua dynamic page (test2.lua), that will show the contents of all the environment variables:


#! /usr/local/bin/lua

-- HTTP header
print [[
Content-Type: text/plain
Set-Cookie: foo=bar
Set-Cookie: wonder=always

]]

-- body of page

-- find all environment variables using bash and a temporary file

fname = os.tmpname ()
os.execute ("/bin/bash -c set > " .. fname)
f = io.open (fname, "r")  -- open it
s = f:read ("*a")  -- read all of it
print (s)
f:close ()  -- close it
os.remove (fname)



This example also uses "Set-Cookie" twice to set two cookies. We connect to this page twice (once to set the cookie, once to get the value returned). I used this URL in my web browser:


http://10.0.0.2/cgi-bin/test2.lua/sausage/machine?type=meat&date=Friday


(Note the trailing parts of the path after test2.lua).

I see this in the browser:


BASH=/bin/bash
BASH_VERSINFO=([0]="2" [1]="05b" [2]="0" [3]="1" [4]="release" [5]="i386-redhat-linux-gnu")
BASH_VERSION='2.05b.0(1)-release'
DIRSTACK=()
DOCUMENT_ROOT=/usr/local/httpd/htdocs
EUID=99
GATEWAY_INTERFACE=CGI/1.1
GROUPS=()
HOSTNAME=bacall
HOSTTYPE=i386
HTTP_ACCEPT='text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET='ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING=gzip,deflate
HTTP_ACCEPT_LANGUAGE='en-gb,en;q=0.5'
HTTP_CONNECTION=keep-alive
HTTP_COOKIE='foo=bar; wonder=always'
HTTP_HOST=10.0.0.2
HTTP_KEEP_ALIVE=300
HTTP_USER_AGENT='Mozilla/5.0 (Windows; U; WinNT4.0; en-GB; rv:1.7.12) Gecko/20050919 Firefox/1.0.7'
IFS=$' \t\n'
MACHTYPE=i386-redhat-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin
PATH_INFO=/sausage/machine
PATH_TRANSLATED=/usr/local/httpd/htdocs/sausage/machine
PPID=2192
PS4='+ '
PWD=/usr/local/httpd/cgi-bin
QUERY_STRING='type=meat&date=Friday'
REMOTE_ADDR=10.0.0.3
REMOTE_PORT=2433
REQUEST_METHOD=GET
REQUEST_URI='/cgi-bin/test2.lua/sausage/machine?type=meat&date=Friday'
SCRIPT_FILENAME=/usr/local/httpd/cgi-bin/test2.lua
SCRIPT_NAME=/cgi-bin/test2.lua
SERVER_ADDR=10.0.0.2
SERVER_ADMIN=you@your.address
SERVER_NAME=10.0.0.2
SERVER_PORT=80
SERVER_PROTOCOL=HTTP/1.1
SERVER_SIGNATURE=$'<address>Apache/2.0.46 (Unix) PHP/4.3.2 Server at 10.0.0.2 Port 80</address>\n'
SERVER_SOFTWARE='Apache/2.0.46 (Unix) PHP/4.3.2'
SHELL=/sbin/nologin
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=2
TERM=dumb
UID=99
_=/bin/bash





The important lines I have made bold.

We can see these things:


  • The cookies: HTTP_COOKIE='foo=bar; wonder=always'

    We can see both cookies in a single enviroment variable, we will want a way to split those up.

  • Extra path information: PATH_INFO=/sausage/machine

    This was extra information added after "test2.lua" in my URL path. It appears that Apache has realised that test2.lua is the executable file, and that the rest of the path is collected into PATH_INFO.

  • The query string: QUERY_STRING='type=meat&date=Friday'

    This is the part following the "?" in the URL. Both of the variables here (meat and date) are in a single line, we need to split those up as well.

  • How the page was sent: REQUEST_METHOD=GET

    This tells us that the URL was entered into the Browser (or a link was clicked on), rather than a form being filled in. (If a form was filled in then the request method would be "POST" rather than "GET").



- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #2 on Fri 28 Apr 2006 04:16 AM (UTC)

Amended on Fri 12 May 2006 12:18 AM (UTC) by Nick Gammon

Message
Some simple utilities for use with dynamic web pages

To help process and write dynamic web pages I have developed a few routines in Lua which I will place in a separate file, which can be easily included into our main page.


-- Lua utilities for CGI web use

-- split a string with delimiters into a table (reverse of table.concat)

function split (s, delim)

  assert (type (delim) == "string" and string.len (delim) > 0,
          "bad delimiter")

  local start = 1
  local t = {}  -- results table

  -- find each instance of a string followed by the delimiter

  while true do
    local pos = string.find (s, delim, start, true) -- plain find

    if not pos then
      break
    end

    table.insert (t, string.sub (s, start, pos - 1))
    start = pos + string.len (delim)
  end -- while

  -- insert final one (after last delimiter)

  table.insert (t, string.sub (s, start))

  return t
 
end -- function split

-- trim leading and trailing spaces from a string
function trim (s)
  return (string.gsub (s, "^%s*(.-)%s*$", "%1"))
end -- trim

mysql_replacements = { 
   ["\0"] = "\\0",
   ["\n"] = "\\n",
   ["\r"] = "\\r",
   ["\'"] = "\\\'",
   ["\""] = "\\\"",
   ["\026"] = "\\Z",
   ["\b"] = "\\b",
   ["\t"] = "\\t",
   }

-- Fix SQL text by converting various characters to the format MySQL 
--  will recognise in its string processor
--
-- Note that not all the escapes are necessary for internal SQL use, 
-- however if data is being dumped to disk (eg. as SQL statements) 
-- then it is handy for have things like \n and \r made more readable
--   See: http://dev.mysql.com/doc/refman/5.1/en/string-syntax.html

function fixsql (s)

  return (string.gsub (tostring (s), "[%z\n\r\'\"\026\b\t]", 
    function (str)
      return mysql_replacements [str] or str
    end ))

end -- fixsql

html_replacements = { 
   ["<"] = "&lt;",
   [">"] = "&gt;",
   ["&"] = "&amp;",
   }

-- fix text so that < > and & are escaped
function fixhtml (s)

return (string.gsub (tostring (s), "[<>&]", function (str)
  return html_replacements [str] or str
  end ))

end -- fixhtml

-- convert + to space
-- convert %xx where xx is hex characters, to the equivalent byte
function urldecode (s)
  return (string.gsub (string.gsub (s, "+", " "), 
          "%%(%x%x)", 
         function (str)
          return string.char (tonumber (str, 16))
         end ))
end -- function urldecode

-- process a single key=value pair from a GET line (or cookie, etc.)
function assemble_value (s, t)
  assert (type (t) == "table")
  local _, _, key, value = string.find (s, "(.-)=(.+)")

  if key then
    t [trim (urldecode (key))] = trim (urldecode (value))
  end -- if we had key=value

end -- assemble_value

-- output a Lua table as an HTML table
function show_table (t)
  local k, v
  assert (type (t) == "table")
  print "<table border=1 cellpadding=3>"
  for k, v in pairs (t) do
    print "<tr>"
    print ("<th>" .. fixhtml (k) .. "</th>" .. 
           "<td>" .. fixhtml (v) .. "</td>")
    print "</tr>"
  end -- for
  print "</table>"
end -- show_table



In here we have:


  1. split - split a string at a delimiter, into a table

  2. trim - remove leading and trailing spaces from around a string

  3. fixsql - converts a string for use with MySQL (and similar) by escaping out things like quotes by prepending a backspace

  4. fixhtml - converts the symbols < > and & into the HTML equivalents (&lt; &gt;, &amp; )

  5. urldecode - converts special character in a URL (like %20) into their equivalents (space in this case)

  6. assemble_value - takes a key=value string from a URL or cookie, and makes a key/value table entry from it

  7. show_table - converts a simple Lua table into an HTML table


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #3 on Fri 28 Apr 2006 04:28 AM (UTC)

Amended on Fri 28 Apr 2006 04:29 AM (UTC) by Nick Gammon

Message
Extracting out the GET and COOKIE data

This example decodes the query data (the stuff after the "?") into individual table entries, and also the individual cookies.

We are also now using HTML rather than straight text, so we need to use fixhtml to escape out any <, > or & symbols:


#! /usr/local/bin/lua

dofile "cgiutils.lua"

-- HTTP header
print [[
Content-Type: text/html; charset=iso-8859-1
Set-Cookie: foo=bar
Set-Cookie: wonder=always I think

]]

-- body of page

-- XML header, doctype, page header

print [[
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head> 
<title>My title</title>
</head>
<body>
]]

-- show data from the query string

QUERY_STRING = os.getenv ("QUERY_STRING")

get_data = {}

if QUERY_STRING then
  query = split (QUERY_STRING, "&")
  for _, v in ipairs (query) do
    assemble_value (v, get_data)
  end -- for
end -- if

print "<h1>GET data</h1>\n"

show_table (get_data)

-- show cookie data

COOKIE_STRING = os.getenv ("HTTP_COOKIE")

cookie_data = {}

if COOKIE_STRING then
  cookie = split (COOKIE_STRING, ";")
  for _, v in ipairs (cookie) do
    assemble_value (v, cookie_data)
  end -- for
end -- if

print "<h1>COOKIE data</h1>\n"

show_table (cookie_data)

-- end of body

print [[
</body>
</html>
]]



I'll run this with the URL:


http://10.0.0.2/cgi-bin/test2.lua?type=meat&date=Friday%20the%2013th


These are the results (except they actually appear as HTML tables):


GET data

date	Friday the 13th
type	meat

COOKIE data

wonder	always I think
foo	bar


Notice how the spaces in the cookies and the %20 in the URL have been correctly converted into actual spaces.

The file "cgiutils.lua" is described in the post immediately before this one.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #4 on Fri 28 Apr 2006 04:40 AM (UTC)

Amended on Fri 28 Apr 2006 07:10 AM (UTC) by Nick Gammon

Message
Getting form data

If your web page has forms in it (that is, you are using <form> ... </form> ) then we need another method to access that data. Form data can typically be quite long (such as a forum post), so rather than being stored in an environment variable, you can read it by reading from stdin. This example shows reading from stdin to access the form data:


#! /usr/local/bin/lua

dofile "cgiutils.lua"

-- HTTP header
print [[
Content-Type: text/html; charset=iso-8859-1

]]

-- body of page

-- XML header, doctype, page header

print [[
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head> 
<title>My title</title>
</head>
<body>
]]

--- form test

print [[
<form METHOD="post" ACTION=http://10.0.0.2/cgi-bin/test3.lua?humphrey=bogart>
<p>Enter data here: <input type=text Name="Widget" size=20 maxlength=20>
<input Type=hidden Name=date Value="2006-04-28">
<input Type=hidden Name=action Value=fubar>
<input Type=submit Name=Submit Value="Test posting">
</p>
</form>
]]

---- end form test

POST_DATA = io.read ("*a")  -- read all of stdin

post_data = {}

if POST_DATA then
  post = split (POST_DATA, "&")
  for _, v in ipairs (post) do
    assemble_value (v, post_data)
  end -- for
end -- if


print "<h1>POST data</h1>\n"

show_table (post_data)

print ("<p>Request method = ", os.getenv ("REQUEST_METHOD"), "</p>")

-- end of body

print [[
</body>
</html>
]]



The first time I simply enter the URL into my browser:


http://10.0.0.2/cgi-bin/test3.lua


No "POST" data is displayed and I see:


Request method = GET


However if I type "some data" into the text box, and click on the "Test posting" button, I now see:


POST data

action	fubar
date	2006-04-28
Widget	some data
Submit	Test posting

Request method = POST 


Now the request method is POST, which tells me that they have filled in a form, and the various fields, including the hidden ones, are displayed.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #5 on Fri 28 Apr 2006 05:29 AM (UTC)
Message
Putting it all together

We can simplify all the various ways data can be presented to the server from a web page by adding a new routine to the cgiutils.lua file:


-- get query data, cookie data, post data
function get_user_input ()
local _, v

  -- query data (stuff after the ? )
 
  local get_data = {}
  
  for _, v in ipairs (split (os.getenv ("QUERY_STRING"), "&")) do
    assemble_value (v, get_data)
  end -- for
  
  -- cookies
 
  local cookie_data = {}
  
  for _, v in ipairs (split (os.getenv ("HTTP_COOKIE"), ";")) do
    assemble_value (v, cookie_data)
  end -- for
  
  -- posted data (from a form)
  
  local post_data = {}
  
  for _, v in ipairs (split (io.read ("*a"), "&")) do
    assemble_value (v, post_data)
  end -- for
  
 return get_data, cookie_data, post_data, os.getenv ("REQUEST_METHOD")
 
 end -- function get_user_input



This will return four things:


  1. The "get" data from the URL

  2. The cookies data

  3. The "post" data from any form

  4. The request method (GET or POST)


We can use our simplified interface like this:


#! /usr/local/bin/lua

dofile "cgiutils.lua"

-- HTTP header
print [[
Content-Type: text/html; charset=iso-8859-1

]]

-- body of page

-- XML header, doctype, page header

print [[
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head> 
<title>My title</title>
</head>
<body>
]]

--- form test

print [[
<form METHOD="post" ACTION=http://10.0.0.2/cgi-bin/test4.lua?humphrey=bogart>
<p>Enter data here: <input type=text Name="Widget" size=20 maxlength=20>
<input Type=hidden Name=date Value="2006-04-28">
<input Type=hidden Name=action Value=fubar>
<input Type=submit Name=Submit Value="Submit">
</p>
</form>
]]

---- end form test

get_data, cookie_data, post_data, request_method = get_user_input ()

print "<h1>GET data</h1>\n"
show_table (get_data)

print "<h1>COOKIE data</h1>\n"
show_table (cookie_data)

print "<h1>POST data</h1>\n"
show_table (post_data)

print ("<p>Request method = ", request_method, "</p>")

-- end of body

print [[
</body>
</html>
]]



Here, a single call to get_user_input returns 3 tables, which is the data described above. Now we can simply index into the relevant tables if we want to.

For example, to see what cookie "foo" has in it:


print (cookie_data.foo)


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Zeno   USA  (2,871 posts)  [Biography] bio
Date Reply #6 on Sun 30 Apr 2006 05:43 PM (UTC)
Message
Very nice. I need to display a local file (not located in the web dir) on my site, could I use Lua to do that?

Zeno McDohl,
Owner of Bleached InuYasha Galaxy
http://www.biyg.org
[Go to top] top

Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #7 on Sun 30 Apr 2006 08:56 PM (UTC)
Message
If the user Lua is run as has permission to open the file, then yes. If the file and directories containing it are world-readable, then you should be fine.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #8 on Mon 01 May 2006 06:19 AM (UTC)
Message
Indeed. My first example showed reading and displaying a file, which could have any name readable by the Apache CGI scripts:


f = io.open (fname, "r")  -- open it
s = f:read ("*a")  -- read all of it
print (s)
f:close ()  -- close it


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #9 on Mon 01 May 2006 07:14 AM (UTC)

Amended on Wed 03 May 2006 07:22 AM (UTC) by Nick Gammon

Message
How to handle errors in scripts

An interesting problem arose when developing these demonstration pages:


  • A single syntax error (such as leaving off a quote or bracket) would make the page not display at all, as the Lua interpreter would reject it; and

  • A runtime error (like misspelling a function name) would cause the page to terminate prematurely


Both of these were frustrating to locate and correct. There must be an easier way, I thought, especially as Lua is good at letting you call functions in a "protected" mode, and finding out what went wrong.

As it turns out the, answer is simple - to turn the paradigm around. Rather than having:


My script file (eg. test5.lua) ===> calling cgiutils.lua (for useful subroutines)


Do this instead:


The common utilities file (cgiutils.lua) ===> calling my script file (test5.lua) in protected mode


We will assume that a single set of "core" utilites, once debugged, will need infrequent changes, and will be unlikely to have ongoing syntax or runtime problems. However our hundreds of other web pages we develop from time to time are much more likely to be sources of errors, if only because they will be larger.

But, if Apache is really running cgiutils.lua, how will that script know what page the user "really" wants? The key here is to use the "PATH_INFO" environment variable, passed down by Apache. This has the trailing part of the URL in it.

Thus, to run a script "test10.lua" the URL will now be:


http://10.0.0.2/cgi-bin/cgiutils.lua/testing/test10.lua


The two parts in bold above illustrate what is happening. Apache is running the script "cgiutils.lua". This script finds the contents of the "PATH_INFO" variable (which will be "/testing/test10.lua" in this case, prepends a dot to make it a relative pathname, and then does a "dofile" on that filename, thus running the desired file (test10.lua).

The main code to do this (simplified slightly) will be in cgiutils.lua, and look like this:


function load_page ()

  -- get GET, COOKIE, POST data, and request method
  
  get_data, cookie_data, post_data, request_method  =
     get_user_input ()
 
  local scriptname = "." .. os.getenv ("PATH_INFO")
  
  -- do checks here on validity of scriptname

  -- run the file specified in the rest of the URL
  
  dofile (scriptname)
  
end -- load_page


-- main code starts here

ok, error = xpcall (load_page, _TRACEBACK)

if not ok then
  major_problem (error)
end -- error in page



I have added a couple of handy routines to do the main legwork of doing the HTTP and HTML headers and footers, so our file test10.lua can look like this now:


  show_html_header ("my title")
  print ("hello, world<p>")
  show_html_footer ()



Security warning

You would need to be careful in implementing this scheme to make sure that the user of your web site cannot "break out" of your web page hierarchy by doing tricks like this:


http://10.0.0.2/cgi-bin/cgiutils.lua/../../../somedirectory/somefile.lua


You might want to build in a check that disallows the ".." characters in the pathname, for example. Another check might be that the resulting pathname matches a very specific pattern in a regular expression (eg. ends in .lua, starts with the directory where your scripts should be).

The enhanced cgiutils.lua file will appear in the next post.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #10 on Mon 01 May 2006 07:16 AM (UTC)

Amended on Tue 12 Sep 2006 06:41 AM (UTC) by Nick Gammon

Message
Enhanced cgiutils.lua file

Below is the improved cgiutils.lua file. It is similar to the earlier one, with extra functions for automating the production of the http and html headers and footers.


#! /usr/local/bin/lua

-- Lua utilities for CGI web use

-- Written by Nick Gammon - May 2006.

--[[
Copyright © 2006 Nick Gammon.

Permission is hereby granted, free of charge, to any person obtaining a copy 
of this software and associated documentation files (the "Software"), to deal 
in the Software without restriction, including without limitation the rights 
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
copies of the Software, and to permit persons to whom the Software is 
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all 
copies or substantial portions of the Software.    

The software is provided "as is", without warranty of any kind, express or 
implied, including but not limited to the warranties of merchantability, 
fitness for a particular purpose and noninfringement. In no event shall the 
authors or copyright holders be liable for any claim, damages or other 
liability, whether in an action of contract, tort or otherwise, arising from, 
out of or in connection with the software or the use or other dealings in the 
software. 
--]]

-- global flags

done_http_header = false  -- have we sent the HTTP header yet?
done_html_header = false  -- have we sent the HTML header yet?
done_html_footer = false  -- have we sent the HTTP footer yet?

-- split a string with delimiters into a table (reverse of table.concat)
function split (s, delim)

  local t = {}  -- results table

  if s then   
    assert (type (delim) == "string" and string.len (delim) > 0,
            "bad delimiter")
  
    local start = 1
    local pos
  
    -- find each instance of a string followed by the delimiter
  
    repeat
      pos = string.find (s, delim, start, true) -- plain find
  
      if pos then
        table.insert (t, string.sub (s, start, pos - 1))
        start = pos + string.len (delim)
      end
    until not pos
  
    -- insert final one (after last delimiter)
  
    table.insert (t, string.sub (s, start))
  end -- not nil string
  
  return t
 
end -- function split

-- trim leading and trailing spaces from a string
function trim (s)
  return (string.gsub (string.gsub (s, "%s+$", ""), "^%s+", ""))
end -- trim

mysql_replacements = { 
   ["\0"] = "\\0",
   ["\n"] = "\\n",
   ["\r"] = "\\r",
   ["\'"] = "\\\'",
   ["\""] = "\\\"",
   ["\026"] = "\\Z",
   ["\b"] = "\\b",
   ["\t"] = "\\t",
   }

-- Fix SQL text by converting various characters to the format MySQL 
--  will recognise in its string processor
--
-- Note that not all the escapes are necessary for internal SQL use, 
-- however if data is being dumped to disk (eg. as SQL statements) 
-- then it is handy for have things like \n and \r made more readable
--   See: http://dev.mysql.com/doc/refman/5.1/en/string-syntax.html

function fixsql (s)

  return (string.gsub (tostring (s), "[%z\n\r\'\"\026\b\t]", 
    function (str)
      return mysql_replacements [str] or str
    end ))

end -- fixsql

html_replacements = { 
   ["<"] = "&lt;",
   [">"] = "&gt;",
   ["&"] = "&amp;",
   ["\""] = "&quot;",
   }

-- fix text so that < > and & are escaped
function fixhtml (s)

  return (string.gsub (tostring (s), "[<>&\"]", 
    function (str)
      return html_replacements [str] or str
    end ))

end -- fixhtml

-- convert + to space
-- convert %xx where xx is hex characters, to the equivalent byte
function urldecode (s)
  return (string.gsub (string.gsub (s, "+", " "), 
          "%%(%x%x)", 
         function (str)
          return string.char (tonumber (str, 16))
         end ))
end -- function urldecode

-- reverse of urldecode - converts characters that are not alphanumeric
--  into the form %xx (for use in cookies etc.)
function urlencode (s)
 return (string.gsub (s, "%W", 
        function (str)
          return string.format ("%%%02X", string.byte (str))
        end  ))
end -- function urlencode
        
-- process a single key=value pair from a GET line (or cookie, etc.)
function assemble_value (s, t)
  assert (type (t) == "table")
  local _, _, key, value = string.find (s, "(.-)=(.+)")

  if key then
    t [trim (urldecode (key))] = trim (urldecode (value))
  end -- if we had key=value

end -- assemble_value

-- output a Lua table as an HTML table
function show_table (t)
  local k, v
  assert (type (t) == "table")
  print "<table border=1 cellpadding=3>"
  for k, v in pairs (t) do
    print "<tr>"
    print ("<th align=left >" .. fixhtml (k) .. "</th>" .. 
           "<td align=left >" .. fixhtml (v) .. "</td>")
    print "</tr>"
  end -- for
  print "</table>"
end -- show_table


-- get query data, cookie data, post data
function get_user_input ()
local _, v

  -- query data (stuff after the ? )
 
  local get_data = {}
  
  for _, v in ipairs (split (os.getenv ("QUERY_STRING"), "&")) do
    assemble_value (v, get_data)
  end -- for
  
  -- cookies
 
  local cookie_data = {}
  
  for _, v in ipairs (split (os.getenv ("HTTP_COOKIE"), ";")) do
    assemble_value (v, cookie_data)
  end -- for
  
  -- posted data (from a form)
  
  local post_data = {}

  local post_length = tonumber (os.getenv ("CONTENT_LENGTH")) or 0
  if os.getenv ("REQUEST_METHOD") == "POST" and post_length > 0 then
    for _, v in ipairs (split (io.read (post_length), "&")) do
      assemble_value (v, post_data)
    end -- for
  end -- if post
  
 return get_data, cookie_data, post_data, os.getenv ("REQUEST_METHOD")
 
 end -- function get_user_input
 
 -- show a standard HTTP header with optional cookies
function show_http_header (cookies)

  assert (not done_http_header, "Too many HTTP headers")
  
  print ("Content-Type: text/html; charset=iso-8859-1")
  print ("X-Powered-By: cgiutils.lua written by Nick Gammon")
  
  -- output a Set-Cookie line for each cookie in the table
  if cookies then
    assert (type (cookies) == "table")
    for k, v in pairs (cookies) do
      print ("Set-Cookie: " .. urlencode (k) .. "=" .. urlencode (v))
    end -- for each cookie
  end -- we have cookies

  print ("")  -- blank line separates header from body
  done_http_header = true

end -- function show_http_header 

-- show a standard HTML header with optional title, and other free-format things
--  the "other" stuff could be things like stylesheets, links, etc.

function show_html_header (title, other)

  assert (not done_html_header, "Too many HTML headers")

  if not done_http_header then
    show_http_header ()
  end 

  print ('<?xml version="1.0" encoding="iso-8859-1"?>')
  print ('<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">')
  print ('<html lang="en">')
  print ('<head>')
  if title then
    print ('<title>' .. fixhtml (title) .. '</title>')
  end -- having title
  if other then  
    print (other)
  end -- having other stuff
  print ('</head>')
  print ('<body>')

  done_html_header = true
end -- function show_html_header

-- wrap up with an HTML footer
function show_html_footer ()

  assert (not done_html_footer, "Too many HTML footers")

  print ('</body>')
  print ('</html>')
  done_html_footer = true
end -- function show_html_footer

-- helper function to show an error in inverse bold coloured text
function show_error (theerror)
 
  if not done_html_header then
    show_html_header ("Error on web page")
  end 
 
  local colour_error_text = "#FFFFFF"  -- white
  local colour_error_bgnd = "#A52A2A"  -- brown
  local family = "sans-serif"
  local size = "150%"
  local weight = "bolder"
  
  -- style string for the error message
  local style = string.format (
                'color: %s; ' ..
                'background: %s; ' ..
                'font-family: %s; ' ..
                'font-size: %s; ' ..
                'font-weight: %s;',
                
                  colour_error_text,
                  colour_error_bgnd,
                  family,
                  size,
                  weight)
  
  -- show error in desired style
  print (string.format ('<p><span style="%s">&nbsp;%s&nbsp;</span></p>',
         style,
         fixhtml (theerror)))
          
end -- end of show_error
  
-- Call this for things like script errors - it shows the error and exits
function major_problem (why)

  if not done_html_header then
    show_html_header ("Script error on web page")
  end 
  
  print ("<h3>We apologise that there has been a problem with the web server ...</h3>\n")
  print ("<hr>")
  print ("<code><pre>" .. fixhtml (why) .. "</pre></code>")
  print ("<p>Error occurred on " .. os.date ("%#x at %X") .. "</p>\n")
  print ("<hr>")
  
  show_html_footer ()
  
  os.exit (1)  -- let's stop while we are ahead :)
end -- function major_problem
 
-- this function is run in protected mode in case the script page has errors
function load_page ()

  -- get local locale
  
  os.setlocale ("", "time")
  
  -- get GET, COOKIE, POST data, and request method
  
  get_data, cookie_data, post_data, request_method  = get_user_input ()
 
  local scriptname = os.getenv ("PATH_INFO")
  
  -- validity check on requested filename
  if not scriptname or
         string.find (scriptname, "..", 1, true) or
         not string.find (scriptname, "/%a+/[%a%d_%-]+%.lua")
         then
    return show_error ("That URL is not valid")
  end -- if
     
  scriptname = "." .. scriptname  -- make local path
  
  -- check page exists, to avoid annoying user with Lua traceback message
  local f, err1, err2 = io.open (scriptname)
   
  if f then
    f:close ()  -- we found the file, close it now
  else
    return show_error ("The requested page does not exist")
  end -- if
      
  -- run the file specified in the rest of the URL
  dofile (scriptname)
  
end -- load_page


-- main code starts here

ok, error = xpcall (load_page, _TRACEBACK)

if not ok then
  major_problem (error)
end -- error in page

-- ensure footer shown
if done_html_header and not done_html_footer then
  show_html_footer ()
end -- if



[EDIT]

Changed on 4th May 2006, to add various extra functionality:


  • Checking inside show_http_header, show_html_header and show_html_footer to ensure none of these are done twice.

  • Automatic footer generation if not done by the called script.

  • New function urlencode - this encodes a URL (reverse of urldecode) so that things like ';' are converted to '%3B'. This is used internally when setting a cookie.

  • Improved show_error function - colours the error message using <span> rather than a single-item table.

  • Improvements to load_page to do further checks on the requested script file name (using a regular expression). It also checks if the requested script file exists and if not gives a specific error message to that effect.

  • Fixed bug in split function (pos cannot be a local variable)

  • Added license so that you may use this software on your own web pages.


[EDIT]

Changed around 12 September 2006 to fix problems with getting POST data from some web servers using certain browsers. *cough**cough*Safari*cough*

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Sylaer   (20 posts)  [Biography] bio
Date Reply #11 on Mon 01 May 2006 09:11 PM (UTC)
Message
First of all, it's extremely encouraging to share a forum with someone of your calibre. I've heard mention of you before and didn't realize until now that YOU were that guy. For what it's worth, congratulations on all your hard work :)
I have to ask though, with your extensive PHP background, why learn Lua at all? Admittedly I know little about Lua, but PHP seems to be an ideal dynamic web code. Had to wonder what attracted you to it.
[Go to top] top

Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #12 on Mon 01 May 2006 09:36 PM (UTC)
Message
Lua is a much more elegant language than PHP. In fact, PHP as a language has a number of issues with it; issues of inconsistency, hastily implemented features, etc. Of course, PHP has a much more extensive set of helper functions specifically designed for web scripting, but Lua is slowly developing its own set of modules.

In any case, a lot of it really is that Lua is a much nicer language, with much more precise, consistent, and elegant language structures.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #13 on Mon 01 May 2006 11:24 PM (UTC)

Amended on Tue 02 May 2006 05:52 AM (UTC) by Nick Gammon

Message
Quote:

For what it's worth, congratulations on all your hard work :)


Thanks for the complimentary remarks. I am pleased that the forum in general attracts a high standard of posts, there are quite a few knowledgeable and helpful people posting regularly here. :)

Quote:

I have to ask though, with your extensive PHP background, why learn Lua at all?


That is a good question. I should make it clear that so far my Lua web development is at an experimental stage, all of this site here is still in PHP. However, to answer in more detail, here are some considerations:


  • When I first started developing dynamic web pages, I hadn't heard of Lua (it may not have existed then), so I chose the language which was well-known, and fairly close in syntax to C, namely PHP.

  • There is extensive documentation about developing web pages in PHP, virtually none about doing it in Lua. There is indeed a project (now) for doing that, CGILua, however I found the lack of examples on their pages made it very hard to get things going. I am starting to understand more what they are doing, now that I have done a similar thing myself. To give credit to the "official" project, I refer you to this page:


    http://www.keplerproject.org/cgilua/


  • The further I went into adding Lua into MUSHclient, and also using it as a stand-alone tool for other things (like converting old emails from one format to another), the more I liked Lua.

  • Lua has a fundamental simple elegance that appeals to me. The core language is simple (by design) and largely absent of frills. The designers have made a conscious effort to keep Lua minimal, as they have made it simple (and well documented) to add your own extensions by writing your own DLLs (or shared libraries in Unix).

    On my Linux PC, the comparable sizes of Lua to PHP are:


    • Lua: 109 Kb PHP: 1.3 Mb


    On my Windows PC, the comparable sizes of Lua to PHP are:


    • Lua: 104 Kb PHP: 4.07 Mb


  • Lua can be extended easily (for example, you can add MySQL database queries by simply loading a shared library).

  • Various design decisions about PHP that annoy me, but I had learnt to live with, see below.



I don't think PHP is going to be replaced in a long time, if ever. Hundreds of thousands of web pages are developed in it, it has extensive helper functions built in, lots of documentation, and a very good support web site.




Things about PHP that annoy me

The things described below are not bugs, they are design decisions by the PHP developers. As such, PHP is working as designed, and there are always arguments about whether or not a design decision is correct. Much can probably be said in defence of the original design. However, here are my comments about them:



  • The decision (initially) to dump all of the get, post, and cookie variables into the PHP global address space. This was for the ease of programming, so that, if we take this URL as an example:


    http://10.0.0.2/myfile.php?type=meat&date=Friday


    PHP would set global variables:


    $type = "meat";
    $date = "Friday";


    This was certainly convenient, but it could be confusing if you needed to distinguish a variable that came from a cookie, to one that came from the URL. Also, it made scripts a bit more vulnerable to attacks like this:


    http://10.0.0.2/myfile.php?administrator=1


    If the script incautiously assumed that administrator would default to 0, and only set it to 1 if certain checks were passed, then users could assume administrator status (set the global administrator variable) by simply setting it on their URL.

  • The decision to make all variables in a function local, unless specified as global, which is the reverse to what most languages do. For example:

    
    
    Lua
    
    function test ()
      foo = 22 -- sets global foo
      
      local bar
      
      bar = 42  -- sets local bar
    end -- test
    
    PHP
    
    function test ()
      {
      $foo = 22; // sets local foo, global foo is unchanged
      
      global $bar;
      
      $bar = 42; // sets global bar
      }
    


    Personally I think this is messy. Most functions will want to access lots of global variables, but may occasionally use a local variable for looping, or calculations. This decision forces PHP developers to put heaps of "global this-and-that" at the start of each function. If you forget one, or add a new line to a script later on, it will not work correctly. Also, doing something simple like taking a piece of a function and making it into its own function (for readability) won't work by doing a simple copy and paste, unless you carefully inspect it for any references to global variables.

  • The decision (initially) to add slashes automatically to variables from web pages. Thus if you had in a form:


    Nick's cat


    The script actually received:


    Nick\'s cat


    This extra backslash was intended to be helpful if you were going to write the variable to an SQL database, where it would be needed to avoid premature closing of quotes. However if you were not going to do that, you needed to "strip the slashes", and then add them back later.

    With extra slashes being added "behind the scenes" scripts soon take on a nightmare proportion of adding and subtracting slashes, in ways which seem unbalanced (because some are implicit, not explicit).

    Making things worse, this became an option, so that scripts that work on one site stop working on another because the option was reversed.


  • The requirement that all variables must start with a $, which grates with programmers who are used to C. In other words you must do:


    $foo = $bar;

    // not -->

    foo = bar;


    Symbols without the $ prefix are also, rather bizarrely, treated as strings, like this:


    echo hello; // echoes "hello"


    This leads to rather confusing error messages if you forget the $ prefix.


  • The decision to allow variables inside strings, like this:


    $foo = "world";
    $bar = "hello, $foo";


    Again, this is convenient, but what if:


    • You need to put a $ sign inside the string?
    • You need to follow the variable directly by a letter (eg. "hello, worlds"), then this won't work:



    $foo = "world";
    $bar = "hello, $foos";


    In this example we don't get "hello, worlds", because it is trying to find the variable "foos".

    I know you can use single-quoted strings to avoid this behaviour, but it is still a bit messy.


- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Sylaer   (20 posts)  [Biography] bio
Date Reply #14 on Tue 02 May 2006 05:20 AM (UTC)
Message
Nick-
I grinned when you mentioned the same issues that griped me when I was learning PHP (which, incidentally, was before I started learning C). A year or two ago I could be seen at my desk with my nose to the screen in frustration trying to figure out how many escape characters to put into a string to store it in a mySQL database. (I got up to eight once, no kidding. Strangely, it worked.) Dumping cookie and form variables into the url was a pain too, and ultimately, I think, what caused me to give up PHP as a gaming language. Thus I find myself here.
Anyway, I suspect you'll one day be an authority on Lua, and again it's great to have you here. In that spirit, please continue your discussion on Lua :D
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


152,851 views.

This is page 1, subject is 2 pages long: 1 2  [Next page]

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Go to topic:           Search the forum


[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

[Home]


Written by Nick Gammon - 5K   profile for Nick Gammon on Stack Exchange, a network of free, community-driven Q&A sites   Marriage equality

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( https://gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Hosted at HostDash]