How to do AES encryption with Lua and MUSHclient

Posted by Nick Gammon on Thu 09 Dec 2004 02:29 AM — 38 posts, 178,856 views.

Australia Forum Administrator #0
I have been interested in encryption for a long time, however it has been hard to distribute encryption utilities because of the laws passed by various governments to restrict their export.

To work around this, I will describe here a technique for making an encryption library for MUSHclient, to be used in Lua scripting.

The trick is that you need to obtain the actual source code for the AES encryption yourself, providing you are not breaking the laws of the land (whatever that country is) to do so.

You place those files into a directory, along with the "aeslib" Lua library, whose source is given below.

AES is based on the "Rijndael" algorithm by Joan Daemen and Vincent Rijmen.

Once assembled you should have the following files:


aes.c  aeslib.c    lauxlib.h  lua.lib     sha256.c
aes.h  aeslib.dll  lua.h      lualib.lib  sha256.h


Obtain the files from the following places (or others of your choosing):

  • aes.c and aes.h


    http://www.cr0.net:8040/code/crypto/aes/

  • sha256.c and sha256.h


    http://www.cr0.net:8040/code/crypto/sha256/

  • lua.h, lauxlib.h, lua.lib and lualib.lib


    http://www.gammon.com.au/files/mushclient/lua.zip


    You could also simply download Lua from the Lua site www.lua.org and compile it yourself to get those files.

    The files lua.h and luaxlib.h are part of the Lua source, once compiled they become the Lua libraries lua.lib and lualib.lib. That was with Visual C++ 6, if you compiled them under Cygwin you would get liblua.a and liblualib.a.
  • aeslib.c

    This is given below, and will also be available from:


    http://www.gammon.com.au/files/utils/aeslib.c

  • aeslib.dll

    This is the result of the compilation process below.


A simple way of compiling it to install the free Cygwin compiler for Windows. Here are some guidelines for doing that:


http://www.gammon.com.au/smaug/installingcygwin.htm


Once you have got your files assembled, compile them to produce the DLL as follows:


gcc -shared -o aeslib.dll aes.c sha256.c aeslib.c lualib.lib lua.lib


You now have a aeslib.dll file which you can place into the same directory as your other Lua DLLs.

Source for aeslib.c ...



/*

Lua library for AES encryption. 

You need to obtain various files yourself to compile it. See below for details.

Author of this file: Nick Gammon
Date: 10th December 2004

For more information, see: 

  http://www.gammon.com.au/forum/bbshowpost.php?bbsubject_id=4988

  (C) Copyright Nick Gammon 2004. Permission to copy, use, modify, sell and
  distribute this software is granted provided this copyright notice appears
  in all copies. This software is provided "as is" without express or implied
  warranty, and with no claim as to its suitability for any purpose.


The AES implementations used here were written by Christophe Devine.

*/

#define LUA_API __declspec(dllexport)

#include "lua.h"
#include "lauxlib.h"

#include "aes.h"
#include "sha256.h"
#include <malloc.h>
#include <memory.h>
#include <stdlib.h>

#pragma comment( lib, "lua.lib" )
#pragma comment( lib, "lualib.lib" )

/* Uncomment this to provide a "test" function */

/* #define TEST */

/* 

  For more information about the AES encryption, see:

    http://www.cr0.net:8040/code/crypto/

  Source needed in addition to this file is:

    aes.c
    aes.h

      For those two files, see: http://www.cr0.net:8040/code/crypto/aes/


    sha256.c
    sha256.h

      For those two files, see: http://www.cr0.net:8040/code/crypto/sha256/

    lua.h
    lauxlib.h

      For those two files, see: http://www.lua.org/download.html

  
  To link you also need:

    lua.lib
    lualib.lib

  To compile under Cygwin:
  
    gcc -shared -o aeslib.dll aes.c sha256.c aeslib.c lualib.lib lua.lib
    
*/

/*

  Usage: encrypted_text = encrypt (plaintext, key)
  
*/

static int encrypt (lua_State *L)
  {
  aes_context aes_ctx;
  sha256_context sha_ctx;
  unsigned char digest [32];
  unsigned char IV [16];  /* for cipher block chaining */
  unsigned char * buf;
  size_t offset;
  int i,
      lastn;

  /* get text to encrypt */
  size_t textLength;
  const unsigned char * text = luaL_checklstring (L, 1, &textLength);
    
  /* get key */
  size_t keyLength;
  const unsigned char * key = luaL_checklstring (L, 2, &keyLength);

  /* allocate memory to work in, rounded up to next 16 byte boundary
     plus 16 bytes for the IV */
  buf = (unsigned char *) malloc (textLength + 15 + 16);

  if (!buf)
    luaL_error (L, "not enough memory for encryption");

  /* generate random IV */
  for (i = 0; i < 16; i++)
    IV [i] = rand () & 0xFF;

  /* calculate how many bytes of final block are real data */
  lastn = textLength & 0x0F;

  /* use last 4 bits of IV to store length of final block */
  IV [15] &= 0xF0;
  IV [15] |= lastn;  /* number of bytes in final block */
  
  /* hash supplied key,and IV, to give a key for encryption */
  sha256_starts (&sha_ctx);
  sha256_update (&sha_ctx, IV, 16);
  sha256_update (&sha_ctx, (uint8 *) key, keyLength);
  sha256_finish (&sha_ctx, digest);

  /* use hashed supplied key (digest) for encryption */
  aes_set_key( &aes_ctx, digest, 256);
  
  /* copy initialization vector into output buffer */
  memcpy (buf, IV, 16);
  /* make sure all zero, in case not mod 16 bytes */
  memset (&buf [16], 0, textLength + 15);
  /* copy supplied text into it */
  memcpy (&buf [16], text, textLength);
  
  /* encrypt in blocks of 16 bytes (128 bits) */

  for (offset = 16; offset < (textLength + 16); offset += 16)
    {
    /* xor in the IV for this block */
    for (i = 0; i < 16; i++)
      buf [i + offset] ^= IV [i];
    aes_encrypt (&aes_ctx, &buf [offset], &buf [offset]);
    memcpy (IV, &buf [offset], 16);
    }

  /* push results */
  lua_pushlstring (L, buf, offset);

  free (buf);  /* don't need buffer any more */
  return 1;  /* number of result fields */
  } /* end of encrypt */

/*

  Usage: plaintext = decrypt (encrypted_text, key)
  
*/

static int decrypt (lua_State *L)
  {
  aes_context aes_ctx;
  sha256_context sha_ctx;
  unsigned char digest [32];
  unsigned char IV [16];  /* for cipher block chaining */
  unsigned char tmp [16];
  unsigned char * buf;
  size_t offset;
  int i,
      lastn;

  const unsigned char * text ;
  size_t textLength;

  const unsigned char * key;
  size_t keyLength;

  /* get text to decrypt */
  text = luaL_checklstring (L, 1, &textLength);
  
  if (textLength < 16)
    luaL_error (L, "encrypted data too short, must be at least 16 bytes");
  
  /* encrypted block starts with 16-byte initialization vector */
  memcpy (IV, text, 16);
  textLength -= 16;
  
  /* find how many bytes in final block */
  lastn = IV [15] & 0x0F;
  
  /* get key */
  key = luaL_checklstring (L, 2, &keyLength);

  /* hash supplied key ,and IV, to give a key for decryption */
  sha256_starts (&sha_ctx);
  sha256_update (&sha_ctx, IV, 16 );
  sha256_update (&sha_ctx, (uint8 *) key, keyLength);
  sha256_finish (&sha_ctx, digest);

  /* use hashed supplied key (digest) for decryption */
  aes_set_key( &aes_ctx, digest, 256);

  buf = (unsigned char *) malloc (textLength + 15);

  if (!buf)
    luaL_error (L, "not enough memory for decryption");

  /* make sure all zero, in case not mod 16 bytes */
  memset (buf, 0, textLength + 15);
  /* copy supplied text into it, skipping IV */
  memcpy (buf, &text [16], textLength);

  /* decrypt in blocks of 16 bytes (128 bits) */

  for (offset = 0; offset < textLength; offset += 16)
    {
    memcpy (tmp, &buf [offset], 16); /* this will be the IV next time */
    aes_decrypt (&aes_ctx, &buf [offset], &buf [offset]);
    for ( i = 0; i < 16; i++ )  /* xor in current IV */
      buf [offset + i] ^= IV [i];
    memcpy (IV, tmp, 16);    /* new IV */
    }

  /* trim final length */
  if (lastn)
    offset -= 16 - lastn;
  
  /* push results */
  lua_pushlstring (L, buf, offset);

  free (buf);  /* don't need buffer any more */
  return 1;  /* number of result fields */
  } /* end of decrypt */

#ifdef TEST
static int test (lua_State *L)
  {
  aes_context aes_ctx;
  sha256_context sha_ctx;
  unsigned char digest[32];
  unsigned char buf [16];
  unsigned char result1 [16];
  unsigned char result2 [16];
  
  memset (digest, 0, sizeof (digest));
  memset (buf, 0, sizeof (buf));

  aes_set_key( &aes_ctx, (uint8 *) digest, 256);
  aes_encrypt (&aes_ctx, buf, result1); 
  aes_encrypt (&aes_ctx, result1, result2); 
  
  lua_pushlstring (L, result1, sizeof (result1));
  lua_pushlstring (L, result2, sizeof (result2));
  
  /*
  Results should be: 
    DC95C078A2408989AD48A21492842087
    08C374848C228233C2B34F332BD2E9D3
    
  See "The Design of Rijndael" by Joan Daemen and Vincent Rijmen.
  Test vector results for block length 128, key length 256.
  
  */
  
  return 2;  /* number of result fields */
  } /* end of test */
#endif

/* table of operations */
static const struct luaL_reg aeslib [] = 
  {

  {"encrypt", encrypt},
  {"decrypt", decrypt},

#ifdef TEST
  {"test", test},
#endif

  {NULL, NULL}
  };

/* register library */

LUALIB_API int luaopen_aes(lua_State *L)
  {
  luaL_openlib(L, "aes", aeslib, 0);
  return 1;
  }



We are now ready to start encrypting.

In your script (or command window with scripting prefix) we need to load the DLL (once per session) like this:


assert (loadlib ("aeslib.dll", "luaopen_aes")) ()


The loadlib function returns a function (if possible) from the DLL (the luaopen_aes function) and the final parentheses on that line run that function. This installs two functions:


aes.encrypt
aes.decrypt


In Lua 5.1 (MUSHclient 3.80 onwards) you need to use package.loadlib, instead of loadlib on its own. Also, you need to make sure that the loadlib function has not been removed in the Lua sandbox.

The encrypt and decrypt functions need to use 256-bit keys, so they take the supplied key (the second argument) and use the sha256 function (Secure Hash Algorithm, 256 bit version) to turn whatever string you supply into a 256-bit key.

They then take the text you supply to be encrypted or decrypted, and make a temporary buffer, rounded up in size to the next 16-byte boundary, as the encryption works in blocks of 128 bits (16 times 8).

The functions then iterate over the text, encrypting each 16-byte block, and then returning the result.

An example of use:


x = aes.encrypt ("Nick Gammon", "swordfish")
y = aes.decrypt (x, "swordfish")
print (y)  --> Nick Gammon


Note that since the encryption works in 16-byte blocks, the result will always be a multiple of 16, even if the text isn't. In addition, the encrypted text will be 16 bytes larger, because the Initialization Vector (IV) is stored at the start of it.

The last 4 bits of the initialization vector are used to remember how many bytes in the last block are "real", so that decryption will return the correct length string.

Encryption mode

For added security the encryption is done in cipher-block chaining mode. What this means is that a random 16-byte string is generated at the commencement of encryption. This string is stored at the start of the encrypted block, so it can be used for decryption.

The important part about the random string (the so-called Initialization Vector or IV) is that it is XOR'ed with each block prior to encrypting it. This guarantees that each block will encrypt differently, even if you happen to have a sequence of letters (like "aaaaaaaaaa") in your plain text. Without the cipher-block chaining, it is also possible for someone to reassemble a message by cutting and pasting (in 16-byte blocks) parts from different messages encrypted with the same key.




Disclaimer

Security and encryption are big fields. The methods described above are intended to get you started, if you want to use encryption with MUSHclient (or Lua). You should make enquiries about other aspects of security before relying upon a single encryption algorithm to protect your data or privacy.

Also, the source file given above has not been subject to peer review, there may be errors in the implementation that are not obvious. If you find any please let me know.
Amended on Mon 02 Oct 2006 08:50 AM by Nick Gammon
Australia Forum Administrator #1

The file aeslib.c can be downloaded from http://www.gammon.com.au/files/utils/aeslib.c

The MD5 sum of the file is:


$ md5sum aeslib.c
 db2e8bdf2cf6987c5f72c812d9ca8d50  *aeslib.c
Amended on Wed 15 Dec 2004 04:35 AM by Nick Gammon
Australia Forum Administrator #2
Hopefully it goes without saying that the security of any encrypted data is dependent on (other things being equal) the length of the key used to encrypt it.

Although the algorithm above uses a 256-bit key, you will only get the benefit of that if the "passphrase" or key you supply when calling it, has that much "entropy".

There is an easy way of calculating the entropy (ie. the randomness) of a given human-readable key. (This assumes that the key is chosen at random).

Use a scientific calculator, and work out the entropy per character, and multiply by the number of characters.

For example, if you choose a passphrase or key made of lowercase alphabetic characters (a-z) then there are 26 possible combinations.

Use the calculator to work out;


log (combinations) / log (2)


You can use log to the base 10, or natural logs, you will get the same answer.

eg.


log (26) / log (2) = 4.7004


Now multiply by the length of the word you are planning to use. For example, "swordfish" = 9 characters.


entropy = 4.7004 * 9 = 42.30 bits


Thus, using "swordfish" as a password gives you only 42 bits of key, which could probably be broken in under a second.

The other problem with real words, like "swordfish", is that they can be subjected to a dictionary attack. Say you choose any word from a possible 100,000 words that you know.


entropy = log (100000) / log (2) = 16.61


So in fact, "swordfish" is an appallingly bad choice of password. It is equivalent to using a 17-bit key.

Of course, things improve if you use more than one word, but to get a full 256 bits of entropy from using words from a 100,000-word dictionary you need over 15 words!


words needed = 256 / 16.61 = 15.41


Using 256 bits of entropy is probably overkill today, probably around 100 is enough for domestic purposes.

There is an interesting discussion about this on the Diceware page:


http://world.std.com/~reinhold/diceware.html


He describes method of generating keys by rolling dice.

They supply a 7776 word dictionary, designed to fit in with rolling a 6-sided die 5 times per word. (In other words, 6 to the power 5 is 7776). The entropy of that would be:


entropy = log (7776) / log (2) = 12.9248


Thus you would need only about 8 such words to get 100 bits of entropy.
Amended on Thu 27 May 2021 02:44 AM by Nick Gammon
Australia Forum Administrator #3
If you are using Lua you can make an "entropy" function to save a bit of work:


function entropy (x) return (math.log (x) / math.log (2)) end


Now you can test things out with a simple call. For example, using a 15-digit number (where each number can have 10 possibilites) ...


print (entropy (10) * 15) --> 49.82892142331


Similarly for the earlier example of 100,000 words:


print (entropy (100000)) --> 16.609640474437


Or, looking at it another way ... How many digits do I need to have 256 bits of entropy, if I use a simple numeric passphrase?


print (256 / entropy (10)) --> 77.063678889979


OK, you need a 77-digit number. Hope your memory is better than mine at remembering it. ;)
Amended on Thu 09 Dec 2004 03:14 AM by Nick Gammon
Australia Forum Administrator #4
A few more points:

  • Although this library was written for MUSHclient, there is no particular reason it couldn't be used for general encryption by anyone using Lua.
  • You may want to consider adding a message sequence, or a date/time stamp, inside any encrypted message, to guard against a "replay attack". That is, an attack where someone stores an old message from/to you and resends it, thus fooling you into thinking it is a recent one.
  • You may also want to add a "message digest" such as a hash of the complete unencrypted message, into the body of the message. This hash is a guarantee that the message appears in its entirety. Without it, it is possible for someone to truncate your message, possibly altering the sense of it.
  • Suggested key length (entropy). The Diceware page mentioned above quotes:

    Quote:

    In their February 1996 report, "Minimal Key Lengths for Symmetric Ciphers to Provide Adequate Commercial Security" a group of cryptography and computer security experts -- Matt Blaze, Whitfield Diffie, Ronald Rivest, Bruce Schneier, Tsutomo Shimomura, Eric Thompson, and Michael Weiner -- stated:

    "To provide adequate protection against the most serious threats... keys used to protect data today should be at least 75 bits long. To protect information adequately for the next 20 years ... keys in newly-deployed systems should be at least 90 bits long."

  • This page describes symmetrical encryption, where both the sender and receiver of a message have the same key. Securely exchanging keys is a big field in itself, and one that is not directly addressed here. One possible method of exchanging a key is to physically meet and exchange keys. If this is not practical, you could generate a random key (for example, using Diceware) and then exchange that key using public/private key encryption, such as PGP (Pretty Good Privacy) or GPG (Gnu Privacy Guard).

    Also see the new post further down about exchanging keys using the Diffie-Hellman protocol.
  • I mentioned earlier that for security you need a long key. Obviously it also needs to be randomly-generated. A long key consisting of 100 times the letter "a" will not be secure. Also pseudo-random numbers (such as computer-generated random numbers) will not be secure. Throwing dice is one approach. Picking words out of a book at random is another. You can buy inexpensive Bingo games from toy shops. These can be used to randomly generate a number in the range 1 to 90 (or whatever number the game supplies). If it generates 90 numbers (ie. each with an entropy of 6.49) then you would only need around 15 such numbers to get 100 bits of entropy.

    This compares favourably to rolling dice. Each die roll would have 2.58 bits of entropy compared to 6.49 per bingo ball. Thus to get, say, 100 bits of entropy you need 39 dice rolls, compared to 16 Bingo balls. That's somewhat less tedium, if you are going to do it often.

Amended on Mon 02 Oct 2006 08:54 AM by Nick Gammon
Australia Forum Administrator #5

Below is a picture of an example of a Bingo rolling game. This lets you generate numbers randomly in the range 1 to 90.

Greece #6
There are random number generators you can download that generate purely random numbers (you can check them with ent), or you can go to www.random.org and get a live random stream.
Australia Forum Administrator #7
What do you mean "purely" random. By definition almost every generator must follow some mathematical sequence. The only things that are going to be purely random are randomly-occuring things in nature like atomic decay, or thermal noise in a resistor. You can buy gadgets that measure those, but they aren't cheap. How many people do you know with a geiger counter, for example? And there is the question of interfacing it with the PC, in a way that doesn't corrupt the data stream

As for downloading, yes there may be sites that connect to geiger counters that measure atomic decay but downloading your random numbers from the Net is hardly secure, even if may be random. What if someone is logging every byte that enters your PC for example?

Flipping a coin, rolling a die, or the bingo machine (I suggest) are about the only ways of easily generating true randomness that doesn't rely upon something else.
Greece #8
I think those programs work by measuring the time it takes for it to access random locations in the memory. I don't remember exactly, but it really is random. See http://comscire.com/FAQ/ for details.
Australia Forum Administrator #9
I would have thought that the time to access memory was fairly predictable. This sentence on that page makes me worry a bit:

Quote:

Q. Can I use the [product] as a random event generator (REG) for micro-PK, DMMI or other parapsychological experiments?

A. Yes - The [product], which is a true random number generator, will respond to operator intention like other REG's.


So, it's a random number generator that is truly random, except it is influenced by what you are thinking?
Greece #10
Well, yes, as much as any other random generator (or at least they think so). Don't take my word for it, download it and use ent to check it. The chi squared distribution of the bitstream at least was 50 and all the other figures checked out too...
Australia Forum Administrator #11
I am sure that these generators pass the various statistical tests. However with (say) the Marsenne Twister (recently added to MUSHclient) you can probably get very good statistical results. However it is still deterministic. If I guess your seed, I can reproduce your sequence exactly. Say you do what a lot of texts recommend, and seed with the system time. Well, the time of day is hardly a secret. Some systems have been hacked into by people knowing the random-number generator was seeded by the time, thus greatly reducing the set of possible sequences they might use (knowing the time the message originated).

The pseudo-sequences may "look random" but they're not. They are the output of a computer program.
Greece #12
This program doesn't seed an algorithm and then retrieve the sequence, it's not a PRNG. And the chi square distribution test is very hard to pass, the Mersenne Twister would probably fall on 10 or so. I'm trying to find a program to test it.
USA #13
Quote:
So, it's a random number generator that is truly random, except it is influenced by what you are thinking?
Funny thing, but I believe that's one of the basic premises of quantum physics anyhow... :) If you look through their site they do make a lot of mumbling about quantum physics. I'm not sure I buy it, but hey...

By the way, I'm not sure that accessing memory really is in deterministic time. Probably nearly deterministic, but not quite - especially according to quantum mechanics and all that bla bla. Like I said I'm not sure I buy it but there might be some truth to it...
Greece #14
The Mersenne Twister output analysis:

Entropy = 7.997857 bits per byte.

Optimum compression would reduce the size
of this 20078025 byte file by 0 percent.

Chi square distribution for 20078025 samples is 76829.54, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 127.0523 (127.5 = random).
Monte Carlo value for Pi is 3.150064085 (error 0.27 percent).
Serial correlation coefficient is 0.009176 (totally uncorrelated = 0.0).


This basically means that the data is quite deterministic. I had done the same with the other program some time ago and I got a 50% chi square distribution.
Amended on Mon 20 Dec 2004 10:50 PM by Poromenos
Australia Forum Administrator #15
Are you saying the Mersenne Twister is bad on those figures? On this page:

http://www.pierssen.com/arcview/upload/esoterica/randomizer.html

The Mersenne Twister fails 0 tests (as do some other algorithms).

According to the ent page:

http://www.fourmilab.ch/random/

Quote:

Applying this test to the output of various pseudorandom sequence generators is interesting. The low-order 8 bits returned by the standard Unix rand() function, for example, yields:

Chi square distribution for 500000 samples is 0.01, and randomly would exceed this value 99.99 percent of the times.


That was a bad result, a true random number generator does better 99.99% of the time. However on your figures true random numbers do better than the Mersenne Twister only 0.01% of the time.
Australia Forum Administrator #16
My results were ...

Using math.random in Lua, which uses the operating system random-number generator:


Entropy = 7.999992 bits per byte.

Optimum compression would reduce the size
of this 20078025 byte file by 0 percent.

Chi square distribution for 20078025 samples is 222.66, and randomly
would exceed this value 90.00 percent of the times.

Arithmetic mean value of data bytes is 127.5098 (127.5 = random).
Monte Carlo value for Pi is 3.141246085 (error 0.01 percent).
Serial correlation coefficient is -0.000200 (totally uncorrelated = 0.0).


Using the Mersenne Twister:


Entropy = 7.999991 bits per byte.

Optimum compression would reduce the size
of this 20078025 byte file by 0 percent.

Chi square distribution for 20078025 samples is 237.94, and randomly
would exceed this value 75.00 percent of the times.

Arithmetic mean value of data bytes is 127.5020 (127.5 = random).
Monte Carlo value for Pi is 3.141733782 (error 0.00 percent).
Serial correlation coefficient is -0.000307 (totally uncorrelated = 0.0).


I chose a 20078025 byte sample to agree with your example.

My results on the chi-square distribution are somewhat different to yours.

However it is interesting to note that a chi square percentage of 75% is regarded as good. From the ent documentation:

Quote:

Thus, the standard Unix generator (or at least the low-order bytes it returns) is unacceptably non-random, while the improved generator is much better but still sufficiently non-random to cause concern for demanding applications. Contrast both of these software generators with the chi-square result of a genuine random sequence created by timing radioactive decay events.

Chi square distribution for 32768 samples is 237.05, and randomly would exceed this value 75.00 percent of the times.


Thus the Mersenne Twister, and a radioactive decay generator, both have a chi square percentage of 75%. Sounds good to me.

Also the Mersenne Twister value for pi had an error of 0% in my test, which differs markedly from your result of 0.27%.
Amended on Tue 21 Dec 2004 05:07 AM by Nick Gammon
Australia Forum Administrator #17
My big question is, how did you generate your data? For my test I didn't actually use MUSHclient, as I didn't want to write a 20 Mb file to disk, so I used another version of the Mersenne Twister library.

Maybe there is a bug in the MUSHclient implementation. If you give me the exact method you did your test, I'll try to reproduce it.
Greece #18
Also according to the ent page,
Quote:

We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is "almost suspect".

I generated my numbers from a Mersenne C++ class I downloaded from somewhere and generated random bytes.
Amended on Tue 21 Dec 2004 12:11 PM by Poromenos
Greece #19
I don't even know why we're arguing. People can use any (P)RNG they want.
Australia Forum Administrator #20
Well, it has been useful to clarify what all this randomness stuff means. Also, I don't get to chat with my friends much about the Mersenne Twister and chi squared probability. :)

I did a bit more research about what the chi squared test is, and basically it is a comparison of the actual outcome for a given test compared to the expected outcome.


   actual expected diff  squared  sq / count
0  78766  78430    336   113040   1.44129
1  78304  78430   -126    15822   0.20173
2  77966  78430   -464   215097   2.74254
3  78700  78430    270    73016   0.93097
4  78512  78430     82     6759   0.08618
5  78420  78430    -10       96   0.00122
...
and so on up to 255


In this case of a 20,078,025 byte sample, we expect 78,430 occurrences of each byte. The 'diff' column is the amount by which we differ from the exact expectation.

However, if the difference was always zero that would be suspicious too. It would be like going into a room where everyone was exactly average height.

So what we are looking for is a discrepancy from the expected amount, but not too large a discrepancy. I gather that what we want is a chi square total that is around 50% probable. Too high is too large a variation from the average, too low is too small a variation.

In the course of researching this I found that MUSHclient's Mersenne Twister had a slightly out-of-date seeding algorithm, so the next version will have an improved one.

I then wrote a small script inside MUSHclient to output the file for testing with ent:


f = io.open ("mt.txt", "wb")
MtSrand (5679)
for i = 1, 20078025 do
  f:write (string.char (MtRand () * 256))
end
f:close ()	
print "Done"


Running the results of this through ent gave:


Entropy = 7.999991 bits per byte.

Optimum compression would reduce the size
of this 20078025 byte file by 0 percent.

Chi square distribution for 20078025 samples is 244.23, and randomly
would exceed this value 50.00 percent of the times.

Arithmetic mean value of data bytes is 127.4666 (127.5 = random).
Monte Carlo value for Pi is 3.141021362 (error 0.02 percent).
Serial correlation coefficient is 0.000026 (totally uncorrelated = 0.0).


I think this is a good result - someone correct me if I am wrong. The chi square distribution of 50% is about the best you can get.
Greece #21
It is a good result, maybe too good for a PRNG... Fun fact (well, for geeks): someone can guess the seed from 624 32-bit numbers from the output.
Australia Forum Administrator #22
The Mersenne Twister docs suggets you hash the output if you want to use it for cryptographic purposes. Then you can't deduce where in the generator sequence you are.
Greece #23
Yes, or hash multiple outputs for better security. I don't think that 99.9999% of the people have any use for that strong encryption though, Rijndael is way more secure than anything I need (hell, XOR is more secure than I need).
USA #24
Quote:
I don't think that 99.9999% of the people have any use for that strong encryption though
Speak for yourself - I can see many reasons why electronic commerce and voting should be as cryptographically secure as possible. :) Who exactly are you talking about?
Australia Forum Administrator #25
He probably means in MUSHclient?

I agree that most of us expect that commerce sites are secure, however my personal experience has been that practically no-one is interested in swapping (say) encrypted emails with me.

However, having easy access to encryption (say, as a scripting option) may encourage its use. I think the idea of encrypted chat sessions has some merit, especially if you are on a network that might have people eavesdropping on packets.

I think there is a lot to be said for trying to increase one's privacy. It is not that you intend to do anything wrong, however you would be outraged if you knew the government opened (say) every piece of mail you sent and read it "just in case". However that is probably what is happening to a lot of messages that fly around on the Internet.
Greece #26
I agree 100% with Nick. I DO mean MC, obviously sensitive data should be encrypted, but it's not like I exchange corporate secrets chatting in the MUD. Still, everyone has (and should excercise) their right to privacy. I'm all for PGP, GPG, etc, but I don't NEED them for 99% of my emails. Anyway, I was just pointing out that AES is much more secure than anyone will ever need while MUDding.
Australia Forum Administrator #27
Interesting point. The problem with XOR is that it isn't at all secure. I used to think that XORing against a password was pretty smart, but I now see from reading various books about encryption that it is easily broken.

Most people would know that a simple substitution cypher (A=M, B=N etc.) can be broken by simply counting letter frequencies.

However if you have an XOR cypher especially with a short password (eg. BORIS), all you have done is make it into 5 substitution cyphers. (eg. every 5th letter is transformed by B, the next 5 by O, the next 5 by R and so on). So you break up the message into batches of 5, and do a frequency count on each batch. I think there are even automated tools to do that.

My point is, if it is secret enough (or private enough) to encrypt in the first place, you may as well choose a system that can't easily be broken, otherwise there isn't a huge amount of point.

It is possible to get overconfident, and exchange data (eg. credit card numbers) that you normally wouldn't because you think "it's safe, I have encryption".
Greece #28
True, but I might want to say "The admin is an asshole." I don't want it to appear in plaintext, and it's not like the admin is going to try and crack XOR (as simple as it may be) to see that I called him names...
Australia Forum Administrator #29
Just out of curiosity, I tried to see what would happen if I generated the random numbers, and then hashed them. If the hashing has problems you would expect a worse result on the ent test than the non-hashed version. However it went quite well.


function randtest ()

f = io.open ("mt.txt", "wb")
MtSrand (5679)
for i = 1, 627000 do
  s = ""

  -- generate 32 random numbers
  for j = 1, 32 do
    s = s .. string.char (MtRand () * 256)
  end -- j loop

  s = utils.sha256 (s)	-- hash it
  f:write (s)
end -- i loop
f:close ()	
print "Done"

end -- randtest



The code above generates a 20,064,000 byte file, which is roughly the same size as the earlier samples. This uses the sha256 hash (256 bit SHA hash) which is in the latest version of MUSHclient.

Running that file through ent gives:



Entropy = 7.999991 bits per byte.

Optimum compression would reduce the size of this 20064000 byte file by 0 percent.

Chi square distribution for 20064000 samples is 257.00, and randomly would exceed this value 50.00 percent of the times.

Arithmetic mean value of data bytes is 127.4930 (127.5 = random).
Monte Carlo value for Pi is 3.142032297 (error 0.01 percent).
Serial correlation coefficient is -0.000304 (totally uncorrelated = 0.0).



The value for pi seems slightly better, and the other figures are certainly acceptable.

Australia Forum Administrator #30
Version 3.61 adds the ability to seed the Mersenne Twister with a table of seeds, rather than a single number, like this:


t = { 1234, 5678, 9876, 5432 }
MtSrand (t) -- seed with table of seeds


Conceivably you could get a reasonably long sequence of pseudo-random numbers by:

  • Generate a genuinely-random sequence of seeds manually (eg. with dice, using the Bingo game).
  • Use that to seed the Mersenne Twister algorithm, as in the example above.
  • Take the output from the Mersenne Twister and hash it as described earlier.


The resulting sequence should be pretty hard for someone to predict (as they wouldn't have any way of guessing the starting seeds), and by choosing enough seed values (say, 20 of them) the PRNG generator should be in a state that would be hard to establish.

Then by hashing the output you make it virtually impossible to deduce what the state of the PRNG generator is.

However, I am not a cryptographic expert, there may be a flaw in that plan. :)
Amended on Thu 23 Dec 2004 11:42 PM by Nick Gammon
Australia Forum Administrator #31
How to share a secret key

I mentioned earlier the problem of sharing a secret key for use with symmetrical encryption, where you might be using an insecure channel.

There is a solution to this, called the Diffie-Hellman Key Exchange Protocol.

Basically this works by having each party generate a secret number (which they keep secret).

They then generate a "public key" by doing this calculation:


public = g ^ private (mod p)


That is, a generator (2 is a possible good choice) is raised to the power of the private key, modulus a large prime number p.

The two parties exchange their public keys.

Now they can each find a shared number by doing this calculation:


shared = other_public_key ^ my_private_key (mod p)


For example, given a generator of 2, and a prime number:

Prime number = 1012885251643 ("EBD4AA663B" in hex)
Generator = 2

Alice generates a secret key = 79C68E818 (eg. by hashing a randomly chosen phrase)

Alice works out her public key by doing this:


print (aes.dh ("2", "79C68E818", "EBD4AA663B" )) --> 7D458DBEF3


Alice sends this number (7D458DBEF3) to Bob.

Meanwhile, Bob generates a secret key = 49560AEC2

Bob works out his public key by doing this:


print (aes.dh ("2", "49560AEC2", "EBD4AA663B" )) --> 7D8A6551EE


Bob sends this number (7D8A6551EE) to Alice.

Alice can now work out the shared key by using Bob's public key (7D8A6551EE) and her secret key (79C68E818):


print (aes.dh ("7D8A6551EE", "79C68E818", "EBD4AA663B" )) --> 317261C804


Bob can also work out the shared key using Alice's public key (7D458DBEF3) and his secret key (49560AEC2):


print (aes.dh ("7D458DBEF3", "49560AEC2", "EBD4AA663B" )) --> 317261C804


Notice how at the end of this session, both Alice and Bob have arrived at the same shared secret (317261C804).

This can now be used by both of them for encrypting messages to each other.

The strength of this system is based on the problem of working out what the exponent must have been in the original calculation.

In practice, larger prime numbers are needed, otherwise an adversary to simply do an exhaustive search to find the key.

One method I have found of generating large prime numbers is to obtain GPG (Gnu Privacy Guard) and type this:


gpg --gen-prime 1 1024



Output: BA0BB972EED88B2B2D8FB3BAB587CFA9FC5811DAE457C88BADB8690DB5D494EC6D91D6
B29F59EEECB42E4494A48F072AB61A490562F7E631D768E986EF56DE17BBE19BB8FC94F6EF07C1
5409427CA726C885E68502ABED8C63B9BBBF4E1568E4C8032E43C257CA4413E98B701CA4F8BD1C
9B61BE26014073244EF552FD386B8B


That was a 1024-bit prime number. However see caveat below - the prime number is supposed to have the property that (p-1)/2 is also prime. That particular prime does not have that property. I am now using the prime number recommended for the DNS system in RFC 2539.

To make this work in practice, I found a C implementation of the Diffie-Hellman at this page:

http://www.cypherspace.org/adam/rsa/dh-in-C.html

That version was very compact, and designed as a stand-alone program. The version below has been pretty-printed up a bit, and designed to interface with Lua.

I made a separate file in the aeslib project (dh.c) however you could simply add this code to the existing file given earlier in this post.


#include "lua.h"
#include "lauxlib.h"

#include <string.h>

#define MAX_BYTES 1024

unsigned char prime[MAX_BYTES], 
              generator[MAX_BYTES], 
              exponent[MAX_BYTES], 
              result[MAX_BYTES];

/* bytes = number of bytes in prime + 1 */
static int n, v, d, z, bytes;

static void a (unsigned char * x, unsigned char * y, int o)
{
  d = 0;
  for (v = bytes; v--;)
    {
      d += x[v] + y[v] * o;
      x[v] = d;
      d = d >> 8;
    }
}

static void s (unsigned char * x)
{
  for (v = 0; (v < bytes - 1) && (x[v] == prime[v]);)
    v++;
  if (x[v] >= prime[v])
    a (x, prime, -1);
}

static void r (unsigned char * x)
{
  d = 0;
  for (v = 0; v < bytes;)
    {
      d |= x[v];
      x[v++] = d / 2;
      d = (d & 1) << 8;
    }
}

static void M (unsigned char * x, unsigned char * y)
{
  unsigned char X[MAX_BYTES], Y[MAX_BYTES];
  memcpy (X, x, bytes);
  memcpy (Y, y, bytes);
  memset (x, 0, bytes);
  for (z = bytes * 8; z--;)
    {
    if (X[bytes - 1] & 1)
      {
        a (x, Y, 1);
        s (x);
      }
    r (X);
    a (Y, Y, 1);
    s (Y);
    }
}

static void fromhex (char *x, unsigned char * y)
{
  memset (y, 0, bytes);
  for (n = 0; x[n] > 0; n++)
    {
    for (z = 4; z--;)
      a (y, y, 1);
    x[n] |= 32;
    y[bytes - 1] |= x[n] - 48 - (x[n] > 96) * 39;
    }
}

static void output (lua_State * L, unsigned char * x)
{
char buff [MAX_BYTES * 2 + 1];
char * p = buff;

  for (n = 0; !x[n];)
    n++;
  for (; n < bytes; n++)
    p += sprintf (p, "%c%c", 48 + x[n] / 16 + (x[n] > 159) * 7,
      48 + (x[n] & 15) + 7 * ((x[n] & 15) > 9));
lua_pushstring (L, buff);
}

/* dh generator exponent prime */

int dh (lua_State * L)
{
unsigned char p [MAX_BYTES * 2], 
              g [MAX_BYTES * 2], 
              e [MAX_BYTES * 2];

const unsigned char * generatorText;
const unsigned char * exponentText;
const unsigned char * primeText;

  /* get generator */
  generatorText = luaL_checkstring (L, 1);
  if ((strlen (generatorText) / 2) > (MAX_BYTES - 1))
    luaL_error (L, "generator too long");
  strcpy (g, generatorText);

  /* get exponent */
  exponentText = luaL_checkstring (L, 2);
  if ((strlen (exponentText) / 2) > (MAX_BYTES - 1))
    luaL_error (L, "exponent too long");
  strcpy (e, exponentText);

  /* get prime */
  primeText = luaL_checkstring (L, 3);
  if ((strlen (primeText) / 2) > (MAX_BYTES - 1))
    luaL_error (L, "prime too long");
  strcpy (p, primeText);

  if (strlen (exponentText) > strlen (primeText))
    luaL_error (L, "exponent length > prime length");

  if (strlen (generatorText) > strlen (primeText))
    luaL_error (L, "generator length > prime length");
  
  // bytes in prime number
  bytes = ((strlen (primeText) + 1) / 2) + 1;

  fromhex (g, generator);
  fromhex (e, exponent);
  fromhex (p, prime);
  memset (result, 0, bytes);
  result[bytes - 1] = 1;
  for (n = bytes * 8; n--;)
    {
    if (exponent[bytes - 1] & 1)
      M (result, generator);
    M (generator, generator);
    r (exponent);
    }
  output (L, result);
  return 1;
}



You also need to add the dh function to the list of the functions exported to Lua:


/* table of operations */
static const struct luaL_reg aeslib [] = 
  {
  // CBC encrypting
  {"encrypt", encrypt},
  {"decrypt", decrypt},

  // Diffie-Hellman
  {"dh", dh},

#ifdef TEST
  {"test", test},
#endif

  {NULL, NULL}
  };


I do not think this code would be subject to any export restrictions, it is simply C code to raise a number to a power and take the modulus of the result. The code on its own cannot be used to encrypt messages, it simply lets you establish a shared secret.

Once this is successfully compiled, you can make a test for it. I did this test in the Immediate window of MUSHclient:




-- Diffie-Hellman Key Exchange

-- See: http://www.cypherspace.org/adam/rsa/dh-in-C.html

-- this took 29 seconds to run on my PC

assert (package.loadlib ("aeslib.dll", "luaopen_aes")) ()

generator = "2" -- see RFC 4419 - "It is recommended to use 2 as generator"

-- Prime number, got from RFC 2539 (also recommends 2 as the generator)

prime = "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E08" ..
"8A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B302B0A6D" ..
"F25F14374FE1356D6D51C245E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6" ..
"F406B7EDEE386BFB5A899FA5AE9F24117C4B1FE649286651ECE65381FFFFFFFF" ..
"FFFFFFFF" -- 1024 bits

Alice_secret = utils.hash ("Alice chooses some secret phrase that is her private key")

Alice_secret = string.sub (Alice_secret, 1, #prime - 1)

Alice_public = aes.dh (generator, Alice_secret, prime)

print ("Alice secret key = ", Alice_secret)
print ("Alice public key = ", Alice_public)

-- work out the public key Bob sends to Alice
Bob_secret = utils.hash ("Bob has a secret phrase we worked out from a book")
Bob_secret = string.sub (Bob_secret, 1, #prime - 1)

Bob_public = aes.dh (generator, Bob_secret, prime)

print ("Bob secret key = ", Bob_secret)
print ("Bob public key = ", Bob_public)

-- work out the shared secret
Shared_secret_A = aes.dh (Alice_public, Bob_secret, prime)

print ("Shared secret - method A = ", Shared_secret_A)

Shared_secret_B = aes.dh (Bob_public, Alice_secret, prime)

print ("Shared secret - method B = ", Shared_secret_B)

-- check the same
assert (Shared_secret_A == Shared_secret_B)

x = aes.encrypt ("Nick Gammon", Shared_secret_A)
y = aes.decrypt (x, Shared_secret_A)
print (y) --> Nick Gammon




As noted in the code, this isn't the fastest algorithm in the world - it takes 29 seconds to run on my PC. However exchanging secret keys is something you don't do every minute. Also, the test above does the calculations for both parties (Alice and Bob), when in practice they would each do half of it.

The output of the above test was:


Alice secret key = 79c68e818a0c4d84f8c68d7904897200dfa488f2
Alice public key = 855F0FE034691F3B3D3B0FAE03A498D3820E6F50
CFAE9C1DD24D7351CFF64D21168A6A6D1CFE4C3D0DF681C29E01475E136F
423A5AEF091DA5BB8B9F41059C93A41F3B97091582C41C7F4D68A73E69D3
B3A5DCEF9C82806424A149549BDCB858EB7CAC55733F17F4B2D4B8FE96C3
5279BDAC1073AC94F82370EDC4A4A0AB5A9C
Bob secret key = 49560aec25e73db6e2154f53c7c7b9f666b0d533
Bob public key = 74C1BA20B83E794F3243CCA924C48088EA7AA598E6
8A42B3218048C70DF6340C84EB97D15766CA539E2D9F18BA1F8DB406E3A0
1BCCA394B91200C8C16402FCEC5037B46B6FAD73048F9A8B168A420ECE03
429C21C52B19EEF5257BADF9637A0853F75371A5126D11A31B38EE6599A5
5B6CC0A236C3337885A110FB0DDFD6057F
Shared secret - method A = A1E21E54817DDA58306CE8CA32CADB20
25DE23417DAE8B684F7CA4E0EF040C89730AE58096B3F83C4D7E819B5E72
1FC16AE5FEA96317D41507C5C4CE9C68BDABC33518DB367ECDCD5CC86376
6AE9812A9EABECAACB6995493FF0AE1F6AB74FCAD631440FA0DA158791A1
71EF49FC669DE64EF307FB9DA4730CE8880EE623CF06
Shared secret - method B = A1E21E54817DDA58306CE8CA32CADB20
25DE23417DAE8B684F7CA4E0EF040C89730AE58096B3F83C4D7E819B5E72
1FC16AE5FEA96317D41507C5C4CE9C68BDABC33518DB367ECDCD5CC86376
6AE9812A9EABECAACB6995493FF0AE1F6AB74FCAD631440FA0DA158791A1
71EF49FC669DE64EF307FB9DA4730CE8880EE623CF06
Nick Gammon


In practice, Alice would send a message (eg. an email) to Bob, saying:


generator = "2"

prime = "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E08
8A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B302B0A6D
F25F14374FE1356D6D51C245E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6
F406B7EDEE386BFB5A899FA5AE9F24117C4B1FE649286651ECE65381FFFFFFFF
FFFFFFFF"

my_public_key = "855F0FE034691F3B3D3B0FAE03A498D3820E6F50
CFAE9C1DD24D7351CFF64D21168A6A6D1CFE4C3D0DF681C29E01475E136F
423A5AEF091DA5BB8B9F41059C93A41F3B97091582C41C7F4D68A73E69D3
B3A5DCEF9C82806424A149549BDCB858EB7CAC55733F17F4B2D4B8FE96C3
5279BDAC1073AC94F82370EDC4A4A0AB5A9C"


Bob would use the same generator and prime, but make his own secret key, and then reply with:


my_public_key = "74C1BA20B83E794F3243CCA924C48088EA7AA598E6
8A42B3218048C70DF6340C84EB97D15766CA539E2D9F18BA1F8DB406E3A0
1BCCA394B91200C8C16402FCEC5037B46B6FAD73048F9A8B168A420ECE03
429C21C52B19EEF5257BADF9637A0853F75371A5126D11A31B38EE6599A5
5B6CC0A236C3337885A110FB0DDFD6057F"


Now they are both in a position to calculate the shared key, and use that to encrypt messages to each other. You can see from the test that both Alice and Bob end up with the same shared key.

Of course, in practice, you would change the secret keys (Alice_secret and Bob_secret) to be something that only you know. One way would be to roll dice, look up words randomly in a book, get a bingo game, that sort of thing.

For more information, search the web for "Diffie-Hellman" - you will find thousands of pages describing it, and various comments about it.

According to the web page cited earlier: The time to calculate a modular exponent is roughly proportional to the cube of the number of bits, so if a 1024-bit number takes 5 minutes, a 2048-bit number will take 40 minutes, and a 4096-bit number would take about 5 hours.

I found in my case that whilst using a 1024-bit prime number the above test took 29 seconds to run, with at 2048-bit prime number it took about 4 minutes.

A larger prime number is certainly more secure, but the time taken to generate the public and shared keys is somewhat longer. However, as I said before, you only need to do this once per shared key you need to establish, it might not bother you if it takes an hour to work out the shared key.

For more security you probably need a larger secret key (the hash algorithm only generates a 160-bit hash), you might consider using the sha256 hash to generate a longer hash.
Amended on Mon 02 Oct 2006 08:38 AM by Nick Gammon
Australia Forum Administrator #32
I should point out that the Diffie-Hellman protocol is vulnerable to the "man in the middle" attack. This is where a 3rd party intercepts both sides of the communication, pretending to be Bob (to Alice) and Alice (to Bob). If he can manage to do that, then the Alice and Bob are really sharing a secret with the middleman, who then reads all their messages, re-encrypts with his own key, and forwards it on.

However it is much harder to do the man-in-the-middle attack than to simply eavesdrop on network traffic. Monitoring network traffic is pretty trivial, but setting up the man-in-the-middle attack involves spoofing IP addresses and a lot more mucking around.

A potential way of avoiding this attack is to "sign" the messages that are forwarded (the public keys). You can sign messages with PGP or GPG, if you want to install those.

Or, you could confirm the public keys by hashing them and sending the hash (or simply the keys themselves) by another means (eg. snail mail, post on a forum page, that sort of thing).

You obviously also do not "confirm we got the same secret key" by sending a copy to each other. ;)

The way to confirm you arrived at the same key is to encrypt a small message (eg. "test") with the derived secret key, and send a copy to each other, to make sure it decrypts to that word at the other end.
Amended on Mon 02 Oct 2006 06:27 AM by Nick Gammon
Australia Forum Administrator #33
I am having trouble confirming whether the prime number generated by GPG is suitable for use with this algorithm.

For security it might be wise to use the 1024-bit prime number mentioned in RFC 2539 (just remove the spaces to use it):


FFFFFFFF FFFFFFFF C90FDAA2 2168C234 C4C6628B 80DC1CD1 29024E08 8A67CC74 020BBEA6 3B139B22 514A0879 8E3404DD EF9519B3 CD3A431B 302B0A6D F25F1437 4FE1356D 6D51C245 E485B576 625E7EC6 F44C42E9 A637ED6B 0BFF5CB6 F406B7ED EE386BFB 5A899FA5 AE9F2411 7C4B1FE6 49286651 ECE65381 FFFFFFFF FFFFFFFF


According to various sources the prime number should not only be prime, but (p-1)/2 should also be prime. I am not sure of an easy way of calculating that for an arbitrary prime number
Australia Forum Administrator #34
I eventually found a program to test for primality on this page:

http://primes.utm.edu/links/programs/large_arithmetic/

I downloaded the "factor.exe" program, and ran it against the number mentioned in RFC 2539 above. Converted to decimal that number is:


17976931348623159077083915679378745319786029604875601170644442368419
71802161585193689478337958649255415021805654859805036464405481992391
00050792877003355816639229553136239076508735759914822574862575007425
30207744771258955095793777842444242661733472762929938766870920560605
0270810842907692932019128194467627007


Testing that gave this result:


factor -d310 17976931348623159077083915679378745319786029
60487560117064444236841971802161585193689478337958649255415021805654859805036464
40548199239100050792877003355816639229553136239076508735759914822574862575007425
30207744771258955095793777842444242661733472762929938766870920560605027081084290
7692932019128194467627007
this number is prime!


I then calculated (p-1)/2, namely:


89884656743115795385419578396893726598930148024378005853222211842098
59010807925968447391689793246277075109028274299025182322027409961955
00253964385016779083196147765681195382543678799574112874312875037126
51038723856294775478968889212221213308667363814649693834354602803025
135405421453846466009564097233813503


According to the factor program, that is also prime, which is what we want.
Amended on Mon 02 Oct 2006 08:32 AM by Nick Gammon
Australia Forum Administrator #35
Some compile notes for Lua 5.1.2.

I copied a few .h files from the Lua distribution, plus the Lua 5.1 DLL, as follows:


aes.h     lua.h       luaconf.h   sha256.h
aes.c     aeslib.c    lauxlib.h   lua5.1.dll  sha256.c


(Copied files in bold)

In the file aeslib.c I changed the line:


#define LUA_API __declspec(dllexport)


to read:


#define LUA_BUILD_AS_DLL
#define LUA_LIB


I then compiled like this (under Cygwin):


gcc -mno-cygwin -shared -o aeslib.dll aes.c sha256.c aeslib.c lua5.1.dll


This gave a 44 Kb file: aeslib.dll
Australia Forum Administrator #36
There is a way of doing modular exponentiation with the bc library alone. In particular:


-- converts a hex string into a big number
function hex_to_bignum (h)
  local result = bc.number (0)
  local c, n
  assert (not string.match (h, "%X")) -- must be in hex
  for i = 1, #h do
    c = string.sub (h, i, i)  -- pull out digit
    n = tonumber (c, 16)      -- convert base-16 (hex) to decimal
    result = result * 16 + n  -- previous result 16 larger, then add in this digit
  end -- for
  
  return result
end -- function hex_to_bignum 

-- See: http://en.wikipedia.org/wiki/Modular_exponentiation
-- (Right-to-left binary method)

function modular_pow (base, exponent, modulus)
  local old_digits = bc.digits (0)  -- working with integers
  local result = bc.number (1)

  base = bc.number (base)
  exponent = bc.number (exponent)
  modulus = bc.number (modulus)
  
  while not bc.iszero (exponent) do
    if not bc.iszero (bc.mod (exponent, 2)) then
      result = bc.mod (result * base, modulus)
    end -- if
    exponent = exponent / 2
    base = bc.mod (base * base, modulus)
  end -- while
  
  bc.digits (old_digits)  -- back to old number of digits
  return result
end -- function modular_pow 



Applied to the earlier example, it ran quite quickly:



generator = 2

prime = "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024E08" ..
"8A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD3A431B302B0A6D" ..
"F25F14374FE1356D6D51C245E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6" ..
"F406B7EDEE386BFB5A899FA5AE9F24117C4B1FE649286651ECE65381FFFFFFFF" ..
"FFFFFFFF" -- 1024 bits


print ("prime (dec) = ", hex_to_bignum (prime))

Alice_secret = utils.hash ("Alice chooses some secret phrase that is her private key")
Alice_secret = string.sub (Alice_secret, 1, #prime - 1)

print ("Alice_secret (hex) =", Alice_secret)
print ("Alice_secret (dec) =", hex_to_bignum (Alice_secret))

Alice_public = modular_pow (generator, hex_to_bignum (Alice_secret), hex_to_bignum (prime))

print ("result = ", Alice_public)

expected = "855F0FE034691F3B3D3B0FAE03A498D3820E6F50" ..
"CFAE9C1DD24D7351CFF64D21168A6A6D1CFE4C3D0DF681C29E01475E136F" .. 
"423A5AEF091DA5BB8B9F41059C93A41F3B97091582C41C7F4D68A73E69D3" ..
"B3A5DCEF9C82806424A149549BDCB858EB7CAC55733F17F4B2D4B8FE96C3" ..
"5279BDAC1073AC94F82370EDC4A4A0AB5A9C"

print ("expected = ", hex_to_bignum (expected))

print ("expected == actual?", Alice_public == hex_to_bignum (expected))


-- work out the public key Bob sends to Alice
Bob_secret = utils.hash ("Bob has a secret phrase we worked out from a book")
Bob_secret = string.sub (Bob_secret, 1, #prime - 1)

print ("Bob_secret (hex) =", Bob_secret)
print ("Bob_secret (dec) =", hex_to_bignum (Bob_secret ))

Bob_public = modular_pow (generator, hex_to_bignum (Bob_secret), hex_to_bignum (prime))

print ("result = ", Bob_public )

expected = "74C1BA20B83E794F3243CCA924C48088EA7AA598E6" .. 
"8A42B3218048C70DF6340C84EB97D15766CA539E2D9F18BA1F8DB406E3A0" ..
"1BCCA394B91200C8C16402FCEC5037B46B6FAD73048F9A8B168A420ECE03" ..
"429C21C52B19EEF5257BADF9637A0853F75371A5126D11A31B38EE6599A5" ..
"5B6CC0A236C3337885A110FB0DDFD6057F"

print ("expected = ", hex_to_bignum (expected))

print ("expected == actual?", Bob_public == hex_to_bignum (expected))


-- work out the shared secret
Shared_secret_A = modular_pow (Alice_public, hex_to_bignum (Bob_secret), hex_to_bignum (prime))

print ("Shared secret - method A = ", Shared_secret_A)

Shared_secret_B = modular_pow (Bob_public, hex_to_bignum (Alice_secret), hex_to_bignum (prime))

print ("Shared secret - method B = ", Shared_secret_B)

expected = "A1E21E54817DDA58306CE8CA32CADB20" ..
"25DE23417DAE8B684F7CA4E0EF040C89730AE58096B3F83C4D7E819B5E72" ..
"1FC16AE5FEA96317D41507C5C4CE9C68BDABC33518DB367ECDCD5CC86376" ..
"6AE9812A9EABECAACB6995493FF0AE1F6AB74FCAD631440FA0DA158791A1" ..
"71EF49FC669DE64EF307FB9DA4730CE8880EE623CF06"

print ("expected = ", hex_to_bignum (expected))

print ("expected == actual?", Shared_secret_B == hex_to_bignum (expected))


Results (omitting the long numbers):


expected == actual? true
expected == actual? true
expected == actual? true

Amended on Mon 16 Aug 2010 05:55 AM by Nick Gammon
Australia Forum Administrator #37

See Diffie Hellman Secret Key Exchange using OpenSSL for how to exchange a secret with someone without using MUSHclient, but just using OpenSSL.