Lama file transfer protocol for sounds, graphics and even pluggins!

Ok iv been working on this for awhile and was waylaid by moving back into town from the middle of the woods but I think I have finaly got this working to a point where Im hoping people can make use of it.

It transfers files without establishing any extra connections. I made it first for midi or wav files so a mud could stream and play music and sounds to players without the player having to spend extra time downloading files.
I relised pretty quick it can be used for many differnt files including graphics and pluggins. A mud can now update its plugins and distribute them to its players without the players having to do much of anything.

https://github.com/Vessarr/Lama-protocols

It is currently only built for use with a lama mud server
https://github.com/milkmanjack/lama
but it could easily be reworked for any server that can use lua. It could even be rewritten in c for use with non lua servers.

Anywho here is a basic overview of the protocol. I tryed to go over it step by step as much as I could.
https://github.com/Vessarr/Lama-protocols/blob/master/LFTP%20overview

Just the LFTP without my color code stuff
https://github.com/Vessarr/Lama-protocols/blob/master/ProtocolsNC.lua

And the all important mushclient pluggin
https://github.com/Vessarr/Lama-protocols/blob/master/lama_file_protocol.xml

There are restrictions on what files can be sent
out of 20 differnt midi files I have tested only 1 failed to send correctly all others sent and played 100% of the time.

Out of 20 wav files I had a higher fail rate with 3 files not sending correctly

I tested 10 gif and jpeg pictures of different sizes and they all sent correctly every time

mp3 files dont seem to like being sent but I only tryed 2 as those are not my target files

basicly test files before adding them to your mud as some files seems are iffy. Size of the file is the biggest factor so choose midi and wave files that arnt to complicated and are smaller in size. Take care not to send to large of a file. Anything bigger then 100 kb will take a while to send and 200 kb seems to take alot serverside to send and may cause lag.

Pluggins seem to send perfectly every time. I tested this by sending almost every offical Aardwolf pluggin(my favorite mud) and each one sent and worked once recived.

Is that on purpose?

Not so much atm though .exe files and the like send just fine so that might be a sucurity issue. I plan on updateing the plugin to only except certent file types because of it.
Its one of the reasons Im curius what other peoples thoughts are, im not sure exactly what caused the few failures I had but they were consistently the same files each time and they sent they just wouldent play on being recived so I think its got somthing to do with how the files were formated to start with.

Iv got it so it sends and plays midi and wav files reliably so long as they arnt overly large files which along with updating pluggins was my main goal. A 100 kb midi file can easily be 5 min of background music and sends quickly.

pluggins send every time.

Take a file that you know fails to play on the other end and do a binary diff (try http://www.cjmweb.net/vbindiff/ ) between the original version and the received version to see exactly which bytes are changed. That will help determine what if anything can be done to improve success.

If you don't have experience digging into binary formats, post the two files somewhere and I can help you investigate.

Ok well that was def easier then trying to write my own program to do the same thing. Looks like its just the size of the file and a block of data was being dropped cus the client couldnt keep up with the ammount of data. So I increased the delay between each data block so it is based on file size and can be 0.01 for small files to 0.03 for larger. Seems to have taken care of the issue and small files send just a tad faster then just using a standered 0.02 delay for everything. I havent tested this on anything but the midi file I had problems with and a couple of the wav files as of yet but all has worked so far.

It's really weird to me that you're doing all of this stuff to encode away telnet option codes when generic file encodings have already been done for decades. And then to prevent overrunning a buffer you insert arbitrary delays which are destined to fail on other systems.

So here's what I would do instead, because what you're trying to do has already been perfected...

I would do one of three things, in order from likely least difficult to likely most difficult to get right:

1) Use asynchronous sockets to fetch the files via HTTP(S) as described in http://www.mushclient.com/forum/bbshowpost.php?id=12520&page=1


2) Send data over the Telnet stream, but out of band using IAC SB <Unassigned>. The file just needs to have byte 255 doubled before sending to not interfere with IAC (see rfc856 below for insight).
When requesting a file from the server, I would include how much of the file the client has already downloaded in the request to support resuming unfinished downloads.
All SB <Unassigned> transmissions from the server would start with the file name or unique hash and the next byte index in that file so that multiple files can be transferred simultaneously.
To prevent sending too quickly, the server should wait for receipt confirmation from the client after every few packets. Arbitrary delays are not allowed. They break slow buffers.
A file transmission should either begin or end with its SHA1 checksum for verification.

2.5) Send data over the Telnet stream, but out of band using IAC SB TRANSMIT-BINARY ( https://tools.ietf.org/html/rfc856 ). This is like (2) but uses an existing option code instead of making up your own from the unassigned blocks.

3) Send data over the Telnet stream, but out of band using IAC SB KERMIT if a proper Kermit service isn't too hard to set up, which it might be. ( https://tools.ietf.org/html/rfc2840 http://en.wikipedia.org/wiki/Kermit_%28protocol%29 )


I very very very strongly recommend (1). Use a dedicated file server to serve files, not your MUD server.

Alot of it has to do with me having fun figuring it out. It started as my attempt to make a music code protocol and I kinda just went with it till it could do more. I had the thought of doing it with a seperete file server but that was after I already did most of the work on this. I liked the idea of being able to send music to a player as needed.