Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to verify your details, confirm your email, resolve issues, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.

Due to spam on this forum, all posts now need moderator approval.

 Entire forum ➜ MUDs ➜ General ➜ Anachronism, a brand-new Telnet library

Anachronism, a brand-new Telnet library

It is now over 60 days since the last post. This thread is closed.     Refresh page


Posted by Twisol   USA  (2,257 posts)  Bio
Date Sat 14 May 2011 12:16 PM (UTC)
Message
I'm extremely proud to announce that I've finished my Telnet library (written in C), Anachronism! It's fresh out of the oven, so it probably contains a number of bugs, but I'm stamping those out as I find them. While I designed it with MUDs in mind, Anachronism should be suitable for any Telnet-based application.

Github: https://github.com/Twisol/anachronism
Ruby binding: https://github.com/Twisol/anachronism-rb
(Node.js binding in the works.)

Anachronism is comprised of three primary components: the parser, the NVT, and the channels. The closest to the metal is the parser, which translates a Telnet stream into a stream of events.

The NVT provides a convenient interface for you to produce an outgoing Telnet stream. This includes functions such as telnet_send_option() and telnet_send_subnegotiation_start(). The NVT also exposes a parser (via telnet_recv()), so you only need to deal with a single object.

The most unique part of Anachronism are the channels. At this level, Telnet is treated as a data multiplexer, with 256 channels (one for each option) plus the main channel. Channels may be created and registered with an option on an NVT, and events will be routed to the channel. Through this mechanism, each Telnet subprotocol (such as MCCP, GMCP, et cetera) can be implemented in isolation, resulting in cleaner and more reusable code.

If anyone wants to use Anachronism and/or has any questions/concerns/criticism, please let me know! I'm using this for my in-progress MUD client, but it's meant to be used by anyone, so I'm open to comments.

Here's an example program showing Anachronism at work. (Note that while this uses the GMCP option code 201, it's not really GMCP. It's just an example.)
#include <string.h>
#include <stdio.h>
#include <anachronism/nvt.h>


typedef struct gmcp_data
{
  telnet_channel* channel;
} gmcp_data;

// This is called when the option is negotiated on (DO/WILL) or off (DONT/WONT).
void on_gmcp_toggle(telnet_channel* channel,
                    telnet_channel_mode on,
                    telnet_channel_provider who)
{
  // Get the gmcp_data object back from the channel.
  if (on)
  {
    if (who == TELNET_CHANNEL_LOCAL)
    {
      const telnet_byte* data = "Foo.Bar [4, 5, 6]";
      telnet_channel_send(channel, data, strlen(data));
    }
  }
}

// This is called for IAC SB <option>, IAC SE, and any data between the two.
void on_gmcp_data(telnet_channel* channel,
                  telnet_channel_event type,
                  const telnet_byte* data,
                  size_t length)
{
  // Get the gmcp_data object back from the channel.
  gmcp_data* gmcp = NULL;
  telnet_channel_get_userdata(channel, (void**)&gmcp);
  
  switch (type)
  {
    case TELNET_CHANNEL_EV_BEGIN:
      printf("<GMCP>\n");
      break;
    case TELNET_CHANNEL_EV_END:
      printf("</GMCP>\n");
      break;
    case TELNET_CHANNEL_EV_DATA:
      printf("[GMCP] %.*s\n", length, data);
      break;
  }
}

gmcp_data* gmcp_new(telnet_channel** out)
{
  gmcp_data* gmcp = malloc(sizeof(gmcp_data));
  if (gmcp)
  {
    // Create a new channel, hooking it up to our two GMCP event handlers.
    // We store the gmcp_data object with it so we have a context
    // in the event handlers.
    telnet_channel* channel = telnet_channel_new(&on_gmcp_toggle,
                                                 &on_gmcp_data,
                                                 (void*)gmcp);
    if (channel)
    {
      gmcp->channel = channel;
      *out = channel;
    }
    else
    {
      free(gmcp);
      gmcp = NULL;
    }
  }
  
  return gmcp;
}


void on_event(telnet_nvt* nvt, telnet_event* event)
{
  switch (event->type)
  {
    // Command, warning, and send events aren't routed to channels.
    // The others can be processed here as well, but it's not very useful
    // except for logging.
    case TELNET_EV_COMMAND:
      break;
    case TELNET_EV_WARNING:
      break;
    
    // A send event is emitted when the NVT has data to send to the remote host.
    case TELNET_EV_SEND:
    {
      telnet_send_event* ev = (telnet_send_event*)event;
      printf("[OUT] %.*s\n", ev->length, ev->data);
      break;
    }
    
    default:
      // everything else is handled by channels, so we ignore it
      break;
  }
}

int main()
{
  // Create an NVT, routing events to the on_event method.
  // We're not storing any extra data with this NVT.
  telnet_nvt* nvt = telnet_nvt_new(&on_event, NULL);
  
  // Create a Telnet channel for GMCP.
  telnet_channel* channel = NULL;
  gmcp_data* gmcp = gmcp_new(&channel);
  
  // Bind the GMCP channel to option 201.
  // The two final parameters are comparable to WILL/WONT and DO/DONT.
  // In this case, we WILL offer GMCP, but we DONT want the client to offer it.
  //
  // TELNET_CHANNEL_LAZY is also available. In this mode, we only accept
  // if the remote host asks first. This is useful for clients, as
  // some MUDs don't understand Telnet very well. It's best to wait until
  // the server offers a channel before accepting it.
  telnet_channel_register(channel, nvt, 201, TELNET_CHANNEL_ON, TELNET_CHANNEL_OFF);
  
  // Process some "incoming" data.
  const char* data = "\xFF\xFD\xC9"      // IAC DO GMCP
                     "\xFF\xFA\xC9"      // IAC SB GMCP
                     "Foo.Bar [1, 2, 3]" // data
                     "\xFF\xF0";         // IAC SE
  size_t bytes_used;
  telnet_recv(nvt, (const telnet_byte*)data, strlen(data), &bytes_used);
  
  return 0;
}

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #1 on Sat 14 May 2011 12:22 PM (UTC)

Amended on Sat 14 May 2011 12:27 PM (UTC) by Twisol

Message
A slightly more interesting example, this time using the Ruby binding. I have a working implementation of MCCP using Anachronism's channels in this one. Notice how the Telnet side of MCCP is completely contained; you simply call .inflate as you receive data, and the details are hidden away.

There's another problem solved here which is somewhat unique to MCCP. Once you receive IAC SB COMPRESSv2 IAC SE, the data immediately after is going to be compressed. However, it's extremely likely that you received some of that data in the same packet as the IAC SB ... sequence. You need some way to stop the parser and decompress the rest of the data before continuing, and that's something Anachronism provides in its telnet_interrupt() function (seen here as nvt.interrupt).

require "anachronism"
require "eventmachine"
require "zlib"

class MccpChannel < Anachronism::Channel
  # Called when the channel is negotiated open.
  def on_open (where)
    case where
    when :local
      @out = Zlib::Deflate.new
      send ''
    end
  end
  
  # Called when the channel is closed.
  def on_close (where)
    case where
    when :remote
      @in = nil
    when :local
      @out = nil
    end
  end
  
  # Called on IAC SB <option>.
  def on_focus
  end
  
  # Called on IAC SE.
  def on_blur
    @in = Zlib::Inflate.new
    nvt.interrupt(option, 0)
  end
  
  # Called with data from the subnegotiation as it's received.
  def on_data (data)
  end
  
  def inflate (data)
    return data unless @in
    data = @in.inflate(data)
    
    if @in.finished?
      # There may be some uncompressed data after the compressed stream ends,
      # so get that out too.
      data << @in.flush_next_out
      @in = nil
    end
    
    data
  end
  
  def deflate (data)
    return data unless @out
    @out.deflate(data, Zlib::SYNC_FLUSH)
  end
end

class MainChannel < Anachronism::Channel
  def on_data (data)
    print(data)
  end
end

class MudNVT < Anachronism::NVT
  def initialize (sock)
    super()
    @sock = sock
    @mccp = MccpChannel.new
    
    main = MainChannel.new
    main.register(self, :main)
    @mccp.register(self, 86, :local => :false, :remote => :lazy)
  end
  
  def receive (data)
    data = @mccp.inflate(data)
    while true
      bytes_used = super(data)
      break unless bytes_used < data.length
      data = data[bytes_used..-1]
      
      if last_interrupt_code == [86, 0]
        data = @mccp.inflate(data)
      end
    end
  end
  
  def on_send (data)
    @sock.send_data(data)
  end
end

module MyAdapter
  def post_init
    @nvt = MudNVT.new(self)
  end
  
  def receive_data (data)
    @nvt.receive(data)
  end
  
  def << (data)
    @nvt.send_data (data)
  end
end

module InputAdapter
  attr_accessor :out
  
  def receive_data (data)
    data.gsub! "\n", "\r\n"
    out << data
    puts data
  end
end

EventMachine.run do
  sock = EM.connect "localhost", 5445, MyAdapter
  
  adapter = EM.attach $stdin, InputAdapter
  adapter.out = sock
end


You should have seen MCCP before I implemented channels. >_<

[EDIT]: Oh cool, this was my 1992'nd post. My birthyear!

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Erendir   Germany  (47 posts)  Bio
Date Reply #2 on Sat 14 May 2011 04:13 PM (UTC)
Message
is there any chance You will make a Lua binding for this library?
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #3 on Sat 14 May 2011 09:02 PM (UTC)
Message
It's certainly possible. Javascript and Lua are actually fairly similar to begin with, so once I've finished my Node extension (which I need for my MUD client), it shouldn't be hard to refit it for Lua.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #4 on Mon 16 May 2011 06:50 AM (UTC)
Message
To give some more explanation on what Anachronism is, it's basically a full layer between the socket and your application code, and it saves you from having to write your own Telnet parser and option handling code. It hides it all behind the abstraction of channels (1 per option, plus 1 for data outside subnegotiations).

For example, in Aspect (my MUD client) I would create an NVT and listen for events on the main channel. That's where visible text, ANSI sequences, MXP, etc. is received. If I support MCCP, I can add another channel bound to option 86 (almost as though it were a port), letting me know when I need to start decompressing. If I later added GMCP support, I'd just build another set of handlers to process the data from, and send my own GMCP messages through the channel as needed.

In effect, Anachronism just makes it easier to use Telnet. I've done the work so you don't have to, and you can keep the details of Telnet away from what your code actually does with the data you get. And since channels are very modular, you can write support for a protocol once and release it as a snippet for anyone else to use.

As a bonus, you can drop to a lower level if you don't need all the power Anachronism provides. Just handle everything in the NVT's event callback, or create a parser directly.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Nick Gammon   Australia  (23,140 posts)  Bio   Forum Administrator
Date Reply #5 on Mon 16 May 2011 07:57 AM (UTC)

Amended on Mon 16 May 2011 07:58 AM (UTC) by Nick Gammon

Message
What's all this in the source?


#line 74 "src/parser.c"
{
parser->cs = telnet_parser_start;
}

#line 142 "src/parser.rl"
    parser->callback = callback;
    parser->userdata = userdata;
  }
  return parser;
}


Why the #line directives?

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #6 on Mon 16 May 2011 08:00 AM (UTC)

Amended on Mon 16 May 2011 08:02 AM (UTC) by Twisol

Message
You're looking at parser.c, which is actually generated by Ragel from parser.rl. The #line directives are inserted by Ragel so that error messages still make sense.

You can look at src/README.md for brief explanations of what each file does.

[EDIT]: For clarity, Ragel [1] is the parser generator that I use to build the Telnet parser. The grammar itself is in parser_common.rl.

[1]: http://www.complang.org/ragel/

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Nick Gammon   Australia  (23,140 posts)  Bio   Forum Administrator
Date Reply #7 on Mon 16 May 2011 08:01 AM (UTC)

Amended on Mon 16 May 2011 08:02 AM (UTC) by Nick Gammon

Message
And the gotos?


    if (parser->callback && parser->buf)
    {
      telnet_option_event ev;
      EV_OPTION(ev, parser->option_mark, (*( parser->p)));
      parser->callback(parser, (telnet_event*)&ev);
    }
  }
goto st8;
st8:
if ( ++( parser->p) == ( parser->pe) )
goto _test_eof8;


I can feel a language war about to break out. ;)




When I got my first programming job (quite some time ago now) I was asked to write some code for my new employers.

After a few days' work I submitted my program. The senior programmer glanced through it, said "rewrite it without the gotos" and threw it back at me.

So I rewrote it. And I haven't used a "goto" since. And haven't regretted it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #8 on Mon 16 May 2011 08:03 AM (UTC)
Message
That's Ragel again. I'd like to emphatically emphasize that I did not write most of parser.c. You can see what I did write in parser.rl. :)

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Nick Gammon   Australia  (23,140 posts)  Bio   Forum Administrator
Date Reply #9 on Mon 16 May 2011 08:25 AM (UTC)
Message
Well in that case I am not qualified to comment. I am no expert in Ragel coding.

If I look at the generated code I say "don't code like that". But code generators may, perhaps, because of their nature do that.

A while ago I looked at YACC and Bison. Both interesting tools for automating compiler-generation. However I'm not sure how many "real life" compilers use either of those.

I should add here that this is not a criticism of your code - it may well do exactly what it is intended to do. But whether it works perfectly would need a Ragel expert to say.

Honestly, it's hard to evaluate. A finite state machine is what is required here, and Ragel claims to do that. So far so good.

I hadn't heard of Ragel before so maybe that shows something, I'm not sure what. You are probably at the cutting edge of technology, perhaps that is it.




In case anyone is wondering what this is all about, Twisol asked me to comment on the Anachronism code.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #10 on Mon 16 May 2011 08:37 AM (UTC)

Amended on Mon 16 May 2011 08:40 AM (UTC) by Twisol

Message
Nick Gammon said:
Well in that case I am not qualified to comment. I am no expert in Ragel coding.

Sure. Anachronism is more than just the parser though. In fact, the parser is the lowest-level part of the library, which everything else just stacks on top of. nvt.c has much more interesting code. You can also look at the headers in the include/ directory (which are commented) to get a broader view of the library itself.

Most of parser.rl is really C, anyways. It's only the part bounded by %%{ and }%%, and the two lines that start with %%, that Ragel does anything with. The rest is just C.

If you want the tl;dr version of the file, it just takes data into the parser and emits event objects to a callback function. telnet_parser_interrupt() lets you stop the parser where it's at, which is useful for MCCP (as I noted earlier in the thread).

Nick Gammon said:
If I look at the generated code I say "don't code like that". But code generators may, perhaps, because of their nature do that.

Hah, absolutely. Ragel actually has three optimization levels for C-based code (it can target a few other languages too), and I'm using the third, which uses gotos and inlined actions.

Nick Gammon said:
A while ago I looked at YACC and Bison. Both interesting tools for automating compiler-generation. However I'm not sure how many "real life" compilers use either of those.

Typically you wouldn't include the generators themselves in your application, just the generated parser itself. In my experience (which of course is very little!) it's hard to beat the result of a good generator anyways. Certainly you'd introduce more bugs than otherwise - I mean really, what normal human can adequately reason with gotos?

Of course, parsers live in more places than just compilers.

Nick Gammon said:
I should add here that this is not a criticism of your code - it may well do exactly what it is intended to do. But whether it works perfectly would need a Ragel expert to say.

There's actually a state graph you can generate, and perhaps I should put it up somewhere. If you look in the Rakefile (like a Makefile but in Ruby) you can see a shell command wrapped in backticks. It runs ragel to generate a DOT file, then converts the DOT file to a PNG displaying the graph.

Nick Gammon said:
In case anyone is wondering what this is all about, Twisol asked me to comment on the Anachronism code.

I appreciate it!

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #11 on Mon 16 May 2011 09:07 AM (UTC)

Amended on Mon 16 May 2011 09:08 AM (UTC) by Twisol

Message
Okay, I've uploaded the graph visualization. Probably was a good idea anyways, since it's great documentation for the parser itself. I don't expect everyone to understand Ragel.

https://github.com/Twisol/anachronism/raw/master/doc/parser.png

The lines/arrows depict state transitions (you can ignore the number in the center of each circle). For each transition it shows the bytes that cause the transition to the left of a slash (/), and the actions that occur on that transition to the right.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


38,914 views.

It is now over 60 days since the last post. This thread is closed.     Refresh page

Go to topic:           Search the forum


[Go to top] top

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.