Hashing, base64 encoding and decoding

Posted by Nick Gammon on Thu 09 Dec 2004 03:28 AM — 5 posts, 34,132 views.

Nick Gammon Australia Forum Administrator Thu 09 Dec 2004 03:28 AM #0

Further utilities have been added to MUSHclient version 3.57 to work on strings with imbedded null bytes (bytes with 00 in them).

Hashing



utils.hash (s)

Returns a 40-character hex string which is the hash of the string 's'. The string 's' may contain the null byte (ie. hex 00). Otherwise, this is the same behaviour as the world.Hash function.

eg.



print (utils.hash ("Nick Gammon")) --> fe09b07227a4e006213ac005831d55b20508a568

print (Hash ("Nick Gammon")) --> fe09b07227a4e006213ac005831d55b20508a568

Base-64 encoding



utils.base64encode (s, [, linebreaks] )

Encodes the string 's' in base64 encoding (suitable for emails etc.). If 'linebreaks' is true, there will be a carriage return/linefeed every 76 characters. The string 's' may contain the null byte (ie. hex 00). Otherwise, this is the same behaviour as the world.Base64Encode function.

eg.



print (utils.base64encode ("Nick Gammon"))  --> TmljayBHYW1tb24=

The output string will be 4/3 times as large as the input string, plus some possible padding to make up the result to a multiple of 4 (the padding character is "="). Also, if you request linebreaks there will be a further 2 byte for every 76 bytes output (that is, every 57 bytes of input).

Base-64 decoding



utils.base64decode (s)

Decodes the string 's' from base64 encoding to plain text. The decoded string may contain the null byte (ie. hex 00). Otherwise, this is the same behaviour as the world.Base64Decode function. Bytes that are invalid are skipped (eg. spaces, newlines, other junk).

eg.



print (utils.base64decode ("TmljayBHYW1tb24="))  --> Nick Gammon

If the source string is not a multiple of 4 bytes then the last few bytes of the decoded string will be lost (because decoding is done in batches of 4 input bytes to 3 output bytes).

Poromenos Greece Thu 09 Dec 2004 10:57 PM #1

What kind of hashing is that? I know it's not MD5 or DES, but I can't tell what.

Nick Gammon Australia Forum Administrator Fri 10 Dec 2004 12:33 AM #2

SHA - Secure Hash Algorithm.

Poromenos Greece Fri 10 Dec 2004 07:27 AM #3

Ah, thanks.

Nick Gammon Australia Forum Administrator Thu 23 Dec 2004 11:34 PM #4

Further utilities for working with hashes have been added to version 3.61.

256-bit SHA hashing



utils.sha256 (s)

This returns a 256-bit SHA hash (Secure Hash Algorithm) of the string s, which may contain binary zeroes. Unlike the utils.hash function this returns the result as a straight 32-byte (256-bit) field (that is, not converted to printable hex). If you want it in readable form you must then convert it yourself (eg. with utils.tohex).

eg.


print (utils.tohex (utils.sha256 ("nick gammon")))

--> result: B3223193E1C89CB1E42E2BE2DF34874320F43E149DC315A381B08B7BC52849AD

This is a more secure hash than the standard utils.hash algorithm, which returns a 160-bit hash.

MD5 hash



utils.md5 (s)

This returns a 128-bit MD5 hash of the string s, which may contain binary zeroes. Unlike the utils.hash function this returns the result as a straight 16-byte (128-bit) field (that is, not converted to printable hex). If you want it in readable form you must then convert it yourself (eg. with utils.tohex).

eg.


print (utils.tohex (utils.md5 ("nick gammon")))

--> result: 9A380FD967D936AC99ED73B4A038CE8C

You can write a small Lua program to do the same thing that the md5sum program does (in Linux, Cygwin etc.):


f = io.open ("docs/RegularExpressions.txt", "rb")
if f then
  print (utils.tohex (utils.md5 (f:read ("*a"))))
  f:close () 
end -- if

--> result: 3764E22E2AC5BA67997C42C288253101

Compare this to the output from md5sum using Cygwin:


$ md5sum RegularExpressions.txt
3764e22e2ac5ba67997c42c288253101 *RegularExpressions.txt

The hash is the same, apart from not being in lower case, which you can change with the string.lower function if you want.

Convert data to hex



utils.tohex (s)

This converts the string s to hexadecimal (printable) form. The string may contain binary zeroes. Use string.lower to make a lower-case version if that is what you prefer.

eg.


print (utils.tohex ("Nick Gammon")) --> 4E69636B2047616D6D6F6E

Convert data from hex



utils.fromhex (s)

This converts the supplied hexadecimal string s back to a normal string. The converted string may contain binary zeroes.

eg.


print (utils.fromhex ("4E69636B2047616D6D6F6E")) --> Nick Gammon

The supplied string may contain 'space' characters (0x09 – 0x0D or 0x20) which are ignored, otherwise if it contains characters other than A-F, a-f or 0-9 this function raises an error. If the number of characters is odd then the last character is treated as the low-order nibble of the final byte. eg.



a = utils.fromhex ("ABC") --> same as utils.fromhex ("AB0C")

Note that "spaces are ignored" means that a sequence like "A B C D" is treated as the same as "ABCD" not "0A 0B 0C 0D".