Adapting GMCP Handler to ATCP2 on Geas

Posted by Chaotzin on Thu 19 Jul 2018 09:43 AM — 31 posts, 115,890 views.

#0
So just this week the admins over at Geas MUD (geas.de) implemented an upgrade that will provide players with ATCP2 data that can enable some really sweet mapping to happen.

It's exciting stuff and I'm keen to try my hand at adapting the MUSHclient mapping capability for Geas.
There's just one issue; I'm struggling to get the handler plugin to function correctly (or to understand whether it is working correctly) and, while I'm comfortable with basic lua scripts now, I feel a bit out of my depth with this one. I've had a crack at it before coming to the forums here but haven't gotten very far.

Firstly, Geas describes this in terms of ATCP2 rather than GMCP. To be honest, I don't understand the implications of that difference except that I've been told they're basically the same thing.

Assuming that they are the same (and I'm very unsure of that assumption), I've been trying to understand how a GMCP handler plugin would work on the MUD.

Here are the steps I've taken so far:
I'm currently watching Trace and Debug Packets while I connect to the MUD and move to a new room.

And I'm using two of Nick's plugins listed on this GMCP thread https://www.gammon.com.au/gmcp
1. GMCP_handler_NJG
2. GMCP_message_receiver_test

How I'm interpretting things is that there's some initial TelnetSubnegotiation going on at connection time but then it ceases when the client doesn't return a sufficient response to the server. Here's the connection sequence I've monitored...

On connecting to the MUD:
Quote:

TRACE: Executing Plugin GMCP_handler_NJG script "OnPluginTelnetRequest"
TRACE: Executing Plugin GMCP_handler_NJG script "OnPluginTelnetRequest"
Enabling GMCP.



Client then sends from GMCP_handler_NJC:

Send_GMCP_Packet (string.format ('Core.Hello { "client": "MUSHclient", "version": "%s" }', Version()))
Send_GMCP_Packet ('Core.Supports.Set [ "Char 1", "Comm 1", "Room 1" ]')


Server then sends (Debug Packets):
Quote:

APS ["AP","LI","CV"]



Client receives and handler broadcasts to all plugins:
Quote:

TRACE: Executing Plugin GMCP_handler_NJG script "OnPluginTelnetSubnegotiation"
APS ["AP","LI","CV"]


GMCP_message_receiver_test plugin receives broadcast:
Quote:

TRACE: Executing Plugin GMCP_message_receiver_test script "OnPluginBroadcast"


Debugging from GMCP_message_receiver_test plugin echoes the message in the world window:
Quote:

GMCP: APS


But there's no handler table set up for that message in the test plugin:
Quote:

Warning: No handler for: APS


Now, by comparison, here is what guidance Geas provides in their ATCP2 helpfile:

Quote:

Telnet Protocol Codes:
Name Code
IAC 255
DO 253 DONT 254
WILL 251 WONT 252
SB 250 SE 240
ATCP2 201

ATCP2 packages:

CORE 065 080 # 'AP'
LOCATION 076 073 # 'LI' info about the characters surrounding
VITALS 067 086 # 'CV' info about the characters vitals

Server-tags:
SUPPORTS 083 032 # 'S ' Server send a list of supported features
DATA 068 032 # 'D ' Server sends data

Client-tags:
BEGIN 066 # 'B' Server should send data then changes detected
END 069 # 'E' Server should stop send data
REQUEST 082 # 'R' Server should send actual data

LOCATION-package DATA

"exits" -> mapping of exits with exit type
"hash" -> hash-id of the room (if avaibale)
"links" -> hash-id of linked rooms (if avaibale)
"inside" -> send by inside rooms
"short" -> short desc of the room (if not normal)
"temperature" -> temperature in the room
"type" -> type of the room

exit-types:
undefined -1
room-type 1-13
road 100
door 101 (closed -101)
TRACE: Matched trigger "\b(manhole|door|gate)\b"
hall 102
yard 103
path 104

room types:
water 1
under water 2
air 3
desert 4
arctic 5
mountain 6
meadow 7
forest 8
beach 9
swamp 10
town 11
jungle 12
cave 13

temperature-level
very cold -2
cold -1
hot 1
very hot 2

VITALS-package DATA:

"fatigue" -> 1-20 fatigue level see 'help fatigue'
"health" -> 1-11 health level, see ' help hitpoints'
"intox" -> 1-10 intoxicated level
"mana" -> 1-11 mana level, see 'help mana'
"soaked" -> 1-6 soaked level
"stamina" -> 1-9 stamina level, seel help 'stamina'
"stuffed" -> 1-10 stuffed level


EXAMPLE
Server IAC WILL ATCP
Client IAC DO ATCP2
Server IAC SB ATCP2 CORE SUPPORTS [CORE,LOCATION,VITALS] IAC SE
Client IAC SB ATCP2 LOCATION REQUEST IAC SE
Server IAC SB ATCP2 LOCATION DATA"{"exits":{"north":8,
"northeast":8,"northwest":100,"south":100,"west":8},
"hash":"1A8549F81F6C","links":{"north":"F48B49A87E40",
"northeast":"6AEF4FA8EBE3","northwest":"838C44A84ED6",
"south":"060949403E41","west":"710E44F00ED7"},
"short":"The entrance to a light forest","temperature":-1,
"type":8} IAC SE


If I'm reading this right, the message currently being received:

APS ["AP","LI","CV"]

is actually this:

Server IAC SB ATCP2 CORE SUPPORTS [CORE,LOCATION,VITALS] IAC SE

However, it's not decoded.

So, where is it all going wrong? Are there some changes need to GMCP_handler_NJG to be compatible with the ATCP2 info set out by Geas?

Very appreciative of any insight.
Amended on Thu 19 Jul 2018 09:48 AM by Chaotzin
Australia Forum Administrator #1
Their documentation is rather unclear. I can answer your question about APS. If you add this to the commands table in GMCP_message_receiver_test.xml:


  ["aps"] = gotAPS,


And above it put a function to handle that:


function gotAPS (what)
  tprint (what)
end -- gotAPS


Then you see in the output window:


GMCP: APS
1="AP"
2="LI"
3="CV"


So that has handled the APS message from the server.

However it isn't clear from the help how to get something like location data.

The help seems to suggest doing this:


Server IAC SB ATCP2 CORE SUPPORTS [CORE,LOCATION,VITALS] IAC SE
Client IAC SB ATCP2 LOCATION REQUEST IAC SE
Server IAC SB ATCP2 LOCATION DATA"{"exits":{"north":8,


But is "LOCATION REQUEST" the characters "LI"?

I tried this but nothing happened:


CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , "LI")


Maybe the request has to be in JSON mode, but the example above doesn't really clarify what that would be.

Maybe you can contact the developer and ask what the format is to trigger room information.
Amended on Sat 21 Jul 2018 02:10 AM by Nick Gammon
#2
Thanks Nick.

I've just tried a few combinations of Send_GMCP_Packet and from how I'm understanding it:


AP = CORE
LI = LOCATION
CV = VITALS
S = SUPPORTS
D = DATA
B = BEGIN
E = END
R = REQUEST


So APS = CORE SUPPORTS

Following that logic:


CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , "LIR")


Results in LOCATION REQUEST and the server returns LOCATION DATA:
Quote:

TRACE: Executing Plugin GMCP_handler_NJG script "OnPluginTelnetSubnegotiation"
LID {"exits":{"east":101},"hash":"F52A6420E4E2","inside":1,"links":{"east":"62F863203D6A"},"short":"In a keep"}
TRACE: Executing Plugin GMCP_message_receiver_test script "OnPluginBroadcast"
GMCP: LID
Warning: No handler for: LID


EDIT:
I've gone ahead and created handlers in table for LID and CVD. LID contains everything I want to use in the mapper. Just trying to understand now how to tell the server to send LID when that data changes. At the moment I can only get it to supply the data when I manually requested it via Send_GMCP_Packet.
Amended on Sat 21 Jul 2018 07:47 AM by Chaotzin
Australia Forum Administrator #3
Oh, wow! I think you've cracked it. Therefore LIB is "begin sending data when it changes".

So, add a handler for LID to the table in the example plugin:


 -- for geas

  ["aps"] = gotAPS,
  ["lid"] = gotLID,



And a handler:


function gotLID (what)
  tprint (what)
end -- gotLID


Now when I send:


CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , 'LIR')


I get:


LID {"exits":{"south":101,"up":201},"hash":"68E44AA06B9E","inside":1,"links":{"south":"06164A0B1248","up":"F1ED4AA03A24"},"short":"The 'Golden Dragon Inn'"}
GMCP: LID
"hash"="68E44AA06B9E"
"exits":
  "up"=201
  "south"=101
"short"="The 'Golden Dragon Inn'"
"links":
  "up"="F1ED4AA03A24"
  "south"="06164A0B1248"
"inside"=1


The bottom part is a table returned by the LID call, which means you can now find the room hash.

For example, change the handler to:


function gotLID (LID)
  print ("Room is now", LID.hash)
end -- gotLID


Ask for the room information and you see printed:


Room is now B3464F0B4BAA


This means you can now catch the message broadcasts in other plugins, and using a similar technique find out the room hash.

It doesn't seem to automatically tell you when the room changes, but detecting a room change should be simple enough, then you just ask for the room information.
Amended on Sat 21 Jul 2018 07:45 AM by Nick Gammon
#4
Ah, things are becoming slightly clearer now.

The following seems to enable the automatic updating on info (both just done manually by me):

Just sending DO first:

CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , 'DO')


Followed by LOCATION BEGIN:

CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , 'LIB')


And VITALS BEGIN:

CallPlugin ( "74f8c420df7d59ad5aa66246", "Send_GMCP_Packet" , 'CVB')


Now, where to place these in the plugin to have it complete this sequence upon connection...
Australia Forum Administrator #5
I think the developer at Geas has misunderstood the GMCP spec a bit. If you advertise that you support "LI" then the "LI" message should work. Thus "LI" "data" should really be "LI.D".

Alternatively (since the "supports" message is only sent once) make it explicit:


Core.Supports ["Core", "Core.Supports", "LI.D", "LI.R", 
               "CV.D", "CV.R"]


Making up a new message "APS" seems a bit strange, because the "core" message should be "Core.Supports" so that a client can find out which messages actually work.

Unfortunately the project to standardize all this disintegrated before all this could be finalized properly.
Australia Forum Administrator #6
Really, now you should move the "test" plugin stuff into your mapper plugin. Maybe at startup time it tells the server to send the messages, and then processes the room information when you change rooms.

"Broadcast" messages go to all plugins. You don't have to try to squeeze the mapper functionality into the "GMCP test" plugin.
#7
Thanks for your help Nick. I'm looking ahead at what the mapper plugin will need and I may have a few more kinks to work out there... such as, how to draw the 26 possible directions Geas' rooms support (all laterals like n, ne, e plus their up/down variants like u, nu, neu)

I've noticed something odd about this port now though.

GMCP is only enabled for Geas.de on port 3334. When connecting on that port I am finding that stacked commands (whether from world aliases, plugins or typed into the immediate command line) are not being properly 'interpreted' beyond the second command in a stack.

For example, an alias to send:

say one
say two
say three
say four
say five


Results in:
Quote:

You say in Common: one
You say in Common: two
say three
say four
say five


Three to five are not command echoes. They're mud output, complete with colour bleeding, of the commands I just sent.

Very odd! Looks specific to port 3334 and occurs with or without any GMCP plugins firing behind the scenes. Is this likely to be a MUD dev/server issue or something to do with playing on a port with GMCP?
Australia Forum Administrator #8
Looking at the packet debug, there doesn't seem to be any misbehaviour on the client side:



Sent  packet: 8 (9 bytes) at Sunday, July 22, 2018, 2:10:03 PM

say one..          73 61 79 20 6f 6e 65 0d 0a

Sent  packet: 9 (9 bytes) at Sunday, July 22, 2018, 2:10:03 PM

say two..          73 61 79 20 74 77 6f 0d 0a

Sent  packet: 10 (11 bytes) at Sunday, July 22, 2018, 2:10:03 PM

say three..        73 61 79 20 74 68 72 65 65 0d 0a

Sent  packet: 11 (10 bytes) at Sunday, July 22, 2018, 2:10:03 PM

say four..         73 61 79 20 66 6f 75 72 0d 0a

Sent  packet: 12 (10 bytes) at Sunday, July 22, 2018, 2:10:03 PM

say five..         73 61 79 20 66 69 76 65 0d 0a

Incoming packet: 8 (28 bytes) at Sunday, July 22, 2018, 2:10:03 PM

You say in Commo   59 6f 75 20 73 61 79 20 69 6e 20 43 6f 6d 6d 6f
n: one.[0m.>       6e 3a 20 6f 6e 65 1b 5b 30 6d 0a 3e

Incoming packet: 9 (59 bytes) at Sunday, July 22, 2018, 2:10:04 PM

You say in Commo   59 6f 75 20 73 61 79 20 69 6e 20 43 6f 6d 6d 6f
n: two..say thre   6e 3a 20 74 77 6f 0d 0a 73 61 79 20 74 68 72 65
e..say four..say   65 0d 0a 73 61 79 20 66 6f 75 72 0d 0a 73 61 79
 five.[0m.>        20 66 69 76 65 1b 5b 30 6d 0a 3e


It looks like the server has taken the second packet (say two) and then treated the other three as part of it. This looks like a server bug to me.

To be honest, if they have just started implementing this, they are probably ironing bugs out of it. I suggest making a bug report to the admins.

You might also suggest that they implement some message type that notifies you when you have changed rooms. Then you can query what the new room number is. When I was testing it didn't seem to do that, unless I had never visited the room before, which was only partly helpful.
#9
I've just sent them a report on that one. Currently getting around it by throttling all commands down to 0.4seconds. Automatic supply of LOCATION DATA is now working on movement now too.

But in other matters, I'm having a harder time than I expected adapting a mapper plugin to this.

The two examples I'm working with are the GMCP_Mapper configured for IRE muds: https://github.com/nickgammon/plugins/blob/master/GMCP_Mapper.xml

...and the Aardwolf GMCP mapper.

I'm finding the Aardwolf plugin in particular very hard to follow and probably far more complicated than I'm cut out for.

But I'm also having a hard to finding which content in the IRE GMCP_Mapper I can use (eg drawing rooms, saving room info to database) and what I need to configure myself for the Geas GMCP info (eg handlers).

The functionality of these two plugins in action is fantastic though so I'd love to be able to preserve as much of that as possible.

I was hoping it'd be as simple as renaming the handlers around the OnPluginBroadcast function, but I'm not getting very far with that approach.

So, should I be attempting to work from an example like these two plugins or would it actually be easier to start with something almost blank?
Australia Forum Administrator #10
The names of the handlers doesn't really matter. Each handler just handles a high-level ATCP message (eg. "LID") and you can call a function of whatever name you want, which takes a table argument.

The table will be the JSON message sent, converted into a Lua table.

Quote:

So, should I be attempting to work from an example like these two plugins or would it actually be easier to start with something almost blank?


It depends a bit on your objectives. I think starting minimalist and understanding everything is better in the long run:

http://gammon.com.au/forum/?id=12635


However the existing mappers already handle a lot of stuff for you like storing rooms in a database.

The trouble is, the existing mappers (especially the Aardwolf one) are quite sophisticated, and if something goes wrong (because of your changes) you might take a while to work out how to fix it.

Really, though, something like the GMCP mapper should be possible to get working with minimal effort, because the only major difference is the interface to the ATCP stuff, which might be quite a small change.
#11
Ah that's the article I had seen before but couldn't find. Thanks for that link and for your patience, Nick. I know the forums are flooded with questions about the mapper so I'll try to ask only what I can't find through searching.

I've persisted with working through the GMCP_Mapper plugin and there are now two sections that I'm not certain I'm understanding.

Part of what I'm doing is substituting parts of the GotRoomInfo function to match what LID provides. But I'm confused by the inclusion of the following:

function gotRoomInfo (info)
  
  if info.num then
    got_room_number (info.num)
  end -- if

  if info.name then
    got_room_name (info.name)
  end -- if
  
  if info.exits then
    local t = {}
    for k, v in pairs (info.exits) do
      table.insert (t, k .. "(" .. v .. ")")
    end -- for
    
    got_room_full_exits (table.concat (t, ","))
  end -- if
  
  if info.environment then
    got_environment (info.environment)
  end -- if
  
  if info.coords then
    got_coordinates (info.coords)
  end -- if
  
  if info.details then
    got_info (table.concat (info.details, ","))
  end -- if
  
  if info.area then
    areas [current_area] = info.area:sub (1, 1):upper () .. info.area:sub (2)
  end -- if 
  
end  -- gotRoomInfo


For the bolded if statement I've substituted info.exits with LID.links, and have done similar for the other relevant info such as LID.hash for info.num. But what is the relevance of having a got_room_full_exits function in addition to a got_room_exit function elsewhere in the code? I can't really see what the latter does.

The second issue is I think what is preventing the mapper from actually drawing a room, and that is the fact that Geas room hashes come as a string of numbers and letters (eg, A1AB6FF0E88E), and the plugin currently requires the hash as a number:

-- here when location changes, eg. : Room.Num 7476
function got_room_number (s)
  
  local room_number = tonumber (s)
    
  if not room_number then
    return
  end -- no room number

  current_room = room_number
  mapper.draw (tostring (room_number))
  
  if expected_exit == "0" and from_room then
    fix_up_exit ()
  end -- exit was wrong

end -- got_room_number


At the moment it's not getting a number so the action stops there, no rooms drawn. Removing 'tonumber' and leaving as it comes results in several errors. Is there something about it being in number format that is critical and I'm just missing it?
Amended on Mon 23 Jul 2018 10:56 AM by Chaotzin
Australia Forum Administrator #12
I think in that case the vnums (room numbers) really were numbers and there were times when it was comparing a string (eg. "12345") to a number (eg. 12345) which would not compare equal, thus doing a tonumber makes sure that they are. In your case as hashes are strings, just don't do a "tonumber" on them).

As for the exits, that MUD sent two messages, "room info" which included exits, and just the exits, so to save duplicating the code, the room info message turned the exits into a message that looked like it had got the exits message.
#13
Progress update, I've successfully gotten the mapper up and going and it's super nice. :) :) :)

Had to go through with a very keen eye to change a few key little things that were causing errors - and learned a ton in the process.

Here's a little error I'm coming across now though and I believe it relates to how exit "touid"s are being saved to or loaded the database.

Occasionally, along a path I'll see that a room has mapped a 'break' in the path surround a single room. All rooms surrounding it will identify a one-way exit to that room but no path back from it.

When I enter the mystery room it maps one-way exits to all its surrounding rooms.

The cause is the room uid hash:

"060949403E41"

I've told the plugin to treat these hashes as strings rather than numbers, and it's working fine for most, except in this case where the hash begins with a "0".

Interestingly, it's correctly recording the room uid when I stand in it, but not when reading the exit data. Exit data instead reads it as:

"6.0949403e+48"

For a temp fix I can change the exits like so:

Quote:
Modified exit Southeast --> 6.0949403e+48 from room 710E44F00ED7 to be to room 060949403E41 in database.


And it stays corrected while that room is cached. But as soon as I reload things from the database it's back to being "6.0949403e+48"

Now, I'm not sure what the underlying issue is here as I've got it reading it as a string. Is there any obvious reason why it'd be doing this?

I can paste some relevant chunks of the code if needed.

Cheers
C
Australia Forum Administrator #14
It looks like the database loading is the issue. Please post the code which loads the exits from the database and puts them into a table.

Or, possibly, the way it is saving to the database.
Amended on Sun 29 Jul 2018 05:22 AM by Nick Gammon
#15
Ok here are I think the two relevant functions - saving exits and loading rooms. I've bolded with notes the specific changes I've made.

The exits are saving via this function:


function save_full_exits_to_database (uid, exits)
--  Note ("Mapper doing: save_full_exits_to_database.")
  
  local room = rooms [uid]
  
  db:exec ("BEGIN TRANSACTION;") 

  for exit in string.gmatch (exits, "[^,]+") do

    dir, touid = string.match (exit, "^(%a+)%((%w+)%)$")	-- Modified note: changed %d to %w to match alphanumeric room uids
    
    touid = tostring (touid) -- instead of tonumber
    
    if dir then
      -- fix up in and out
      dir = ({ ['i'] = "in", o = "out", }) [dir] or dir
      
      dbcheck (db:execute (string.format ([[
        INSERT INTO exits (dir, fromuid, touid, date_added) 
            VALUES (%s, %s, %s, DATETIME('NOW'));
      ]], fixsql  (dir),  -- direction (eg. "n")
          fixsql  (uid),  -- from current room
          fixsql  (touid) -- destination room 
          )))
      if show_database_mods then
        mapper.mapprint ("Added full exit", dir, "from room", uid, "to room", touid, "to database.")
      end -- if
      
      room.exits [dir] = touid
    else
      mapper.maperror ("Cannot make sense of:", exit)
    end -- if can decode
    
  end -- for each exit
  
  db:exec ("COMMIT;") 
  
end -- function save_full_exits_to_database



And this is how it's loading from database:


function load_room_from_database (uid)
--	Note ("Mapper doing: load_room_from_database.")
	
  local room
  
  uid = tostring (uid)	-- Geas: from tonumber to tostring
 
  assert (uid, "No UID supplied to load_room_from_database")
  
  -- if not in database, don't look again
  if room_not_in_database [uid] then
    return nil
  end -- no point looking
  
  for row in db:nrows(string.format ("SELECT * FROM rooms WHERE uid = %s", fixsql (uid))) do
     room = {
       name = row.name,
       area = row.area,
       building = row.building,
       terrain = row.terrain,
       info = row.info,
       notes = row.notes,
       x = row.x,
       y = row.y,
       z = row.z,
  
       exits = {} }
      
    for exitrow in db:nrows(string.format ("SELECT * FROM exits WHERE fromuid = %s", fixsql (uid))) do
       room.exits [exitrow.dir] = tostring (exitrow.touid)	-- Modified note: changed from tonumber to tostring touid
    end -- for each exit
    
  end   -- finding room

  if room then
    rooms [uid] = room
    for row in db_bm:nrows(string.format ("SELECT * FROM bookmarks WHERE uid = %s", fixsql (uid))) do
      rooms [uid].notes = row.notes
    end   -- finding room
        
    return room
  end -- if found
  
  room_not_in_database [uid] = true
  return nil
    
end -- load_room_from_database


In that function I have modified:

room.exits [exitrow.dir] = tonumber (exitrow.touid)


to

room.exits [exitrow.dir] = tostring (exitrow.touid)
Australia Forum Administrator #16
I can't see anything obviously wrong with either the loading or saving. Can you check in the database and see if the uids are wrongly on disk? You can use the sqlite3.exe program that comes with SQLite.

Do a SELECT to find what is actually stored for the exits in question. Also can you find the way that the exits table was set up? I think that is:


.schema exits
#17
Ok so the exits table looks to be set up fine.

It follows the structure described in the plugin where:

exitid INTEGER
dir TEXT (not NULL)
fromuid STRING (not NULL)
touid STRING (not NULL)
data_added DATE


I can see plenty of fromuid and touid entries in there where the string begins with "0" too so that's not it. But there are a bunch of weird data entries in here, just in touid and fromuid exits (ie not in room uid data).

There entries look like this:
Quote:
6094940299999999944756163936158435001692602761216


and

Quote:
inf.0


Really strange. Kind of looks like it's occasionally interpreting the exit uid as a mind-bogglingly huge number.

I've only run this testing the plugin so I could remove that db file and run it fresh to see if the error occurs in the same rooms.
Australia Forum Administrator #18
Quote:

INSERT INTO exits ...


There would also be a place where exits are modified ("UPDATE exits SET ..."). Can you post that too please?
Amended on Sun 29 Jul 2018 10:46 PM by Nick Gammon
Australia Forum Administrator #19
Quote:

060949403E41

...

Kind of looks like it's occasionally interpreting the exit uid as a mind-bogglingly huge number.


Well, taken as a number, that would be interpreted as:


60949403 * 10^41

= 

6094940300000000000000000000000000000000000000000

But you got:

6094940299999999944756163936158435001692602761216
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         41 zeroes, however not completely accurate


So clearly it is interpreting that as a large number, storing as a floating point number (double, probably) which has an accuracy of 15 to 17 decimal digits.

See Wikipedia article about floating point numbers


6094940299999999944756163936158435001692602761216
^^^^^^^^^^^^^^^^^
17 digits are accurate (apart from rounding)


SQLite3 documentation says that data types are suggestions rather than rigidly enforced, so you can, for example, store a string in a field designated as a number.
#20
Aha, how interesting! Should've noticed it's the same number over and over just in different forms.

Actual UID is:
Quote:
060949403E41


Stored in database as:
Quote:
6094940299999999944756163936158435001692602761216.


Represented in text as:
Quote:
6.0949403e+48


And that text representation in Hex as:
Quote:
36 2e 30 39 34 39 34 30 33 65 2b 34 38


I don't really know what to do about it but it is interesting. I've played around with fresh db files and cleared cache and it will repeat these results with a few specific room like that number above.

Another example is:
Quote:
7E3744474144


re your last post about finding 'MODIFY exits SET', I think I'm a bit out of my depth finding that. Haven't actually had to try opening a SQLite db file before now :D
Australia Forum Administrator #21
I made a mistake. It's "UPDATE exits SET ...". It will be in the plugin where it changes an existing exit to lead somewhere else.
#22
Oh ok. I can see that in two particular functions of the plugin:
1. fix_up_exit
2. room_change_exit

'Fix_up_exit' is called from the following 'got_room_number' function when we get a room number via the GMCP handler. Here, (s) is (LID.hash):

-- here when location changes, eg. : Room.Num 7476
function got_room_number (s)
--  Note ("Mapper doing: got_room_number.")
  local room_number = tostring (s)
    
  if not room_number then
    return
  end -- no room number

  current_room = room_number
--  Note ("Mapper doing: current room is " .. current_room)
  mapper.draw (tostring (room_number))
  
  if expected_exit == "0" and from_room then
    fix_up_exit ()
  end -- exit was wrong

end -- got_room_number


And this is the full fix_up_exit function:

function fix_up_exit ()

  local room = rooms [from_room]
  
  dbcheck (db:execute (string.format ([[
      UPDATE exits SET touid = %s WHERE fromuid = %s AND dir = %s;
    ]], 
        fixsql  (current_room),     -- destination room
        fixsql  (from_room),       -- from previous room
        fixsql  (last_direction_moved)  -- direction (eg. "n")
        )))
        
  if show_database_mods then
    mapper.mapprint ("Fixed exit", last_direction_moved, "from room", from_room, "to be to", current_room)
  end -- if
  
  room.exits [last_direction_moved] = current_room
    
  last_direction_moved = nil
  from_room = nil
  
end -- fix_up_exit


The only other function that includes that text is room_change_exit:

function room_change_exit (room, uid)

local available =  {
  n = "North",
  s = "South",
  e = "East",
  w = "West",
  u = "Up",
  d = "Down",
  ne = "Northeast",
  sw = "Southwest",
  nw = "Northwest",
  se = "Southeast",
  nu = "nu",
  su = "su",
  eu = "eu",
  wu = "wu",
  neu = "neu",
  swu = "swu",
  nwu = "nwu",
  seu = "seu",
  nd = "nd",
  sd = "sd",
  ed = "ed",
  wd = "wd",
  ned = "ned",
  swd = "swd",
  nwd = "nwd",
  sed = "sed",
  ['in'] = "In",
  out = "Out",
  }  -- end of available

  -- remove non-existent exits
  for k in pairs (available) do
    if room.exits [k] then
      available [k] = available [k] .. " --> " .. room.exits [k] 
    else
      available [k] = nil
    end -- if not a room exit
  end -- for
  
  if next (available) == nil then
    utils.msgbox ("There are no exits from this room.", "No exits!", "ok", "!", 1)
    return
  end -- not known
  
  local chosen_exit = utils.listbox ("Choose exit to change destination of:", "Exits ...", available )
  if not chosen_exit then
    return
  end

  exit_destination = utils.inputbox ("Enter destination room identifier (number) for " .. available [chosen_exit], room.name, "")

  if not exit_destination then
    return
  end -- cancelled
  
    -- look it up
  local dest_room = rooms [exit_destination]
  
  -- not cached - see if in database
  if not dest_room then
    dest_room = load_room_from_database (exit_destination)
    rooms [exit_destination] = dest_room -- cache for later
  end -- not in cache
  
  if not dest_room then
    utils.msgbox ("Room " .. exit_destination .. " does not exist.", "Room does not exist!", "ok", "!", 1)
    return
  end -- if still not there
    
  dbcheck (db:execute (string.format ([[
    UPDATE exits SET touid = %s WHERE dir = %s AND fromuid = %s;
  ]], fixsql  (exit_destination),
      fixsql  (chosen_exit),  -- direction (eg. "n")
      fixsql  (uid)  -- from current room
      )))
      
  if show_database_mods then
    mapper.mapprint ("Modified exit", available [chosen_exit], "from room", uid, "to be to room", exit_destination, "in database.")
  end -- if
  
  -- update in-memory table
  rooms [uid].exits [chosen_exit] = exit_destination
  mapper.draw (tostring (current_room))
   
end -- room_change_exit


There's a fair bit going on between these functions that I'm not entirely sure I need.

For instance, I don't quite understand the purpose behind fix_up_exit but I would guess it's used for mapping your movement back to the room you just came from. If that's true it's not really needed as "LID.links" provides all the exit data I need.
Australia Forum Administrator #23
I don't understand why this is happening. The function fixsql is supposed to quote its argument, so that a string that "looks like a number" should still be a string. For example, this test:


function dbcheck (code)
 if code ~= sqlite3.OK and    -- no error
    code ~= sqlite3.ROW and   -- completed OK with another row of data
    code ~= sqlite3.DONE then -- completed OK, no more rows
    local err = db:errmsg ()  -- the rollback will change the error message
    db:exec ("ROLLBACK")      -- rollback any transaction to unlock the database
    error (err, 2)            -- show error in caller's context
  end -- if
end -- dbcheck 

function fixsql (s)
  if s then
    return "'" .. (string.gsub (s, "'", "''")) .. "'" -- replace single quotes with two lots of single quotes
  else
    return "NULL"
  end -- if
end -- fixsql

-- start here

db    = assert (sqlite3.open(GetInfo (66) .. "test.db"))

dbcheck (db:execute[[
  CREATE TABLE IF NOT EXISTS test ( t text );
  DELETE FROM test;
  INSERT INTO test (t) VALUES ('060949403E41');
  ]])

for row in db:nrows(string.format ("SELECT * FROM test")) do
  print (row.t, type (row.t))
end   -- finding room

db:close()


Running that I get:


060949403E41 string


In other words, I get out what I put in.

Try running that in your MUSHclient "Immediate" window (Ctrl+I).

Also, check that your "fixsql" function looks like the above.

Also please show your MUSHclient version.
#24
Running that I also got:
060949403E41 string


...and fixsql is the same in the plugin:
function fixsql (s)
  
  if s then
    return "'" .. (string.gsub (s, "'", "''")) .. "'" -- replace single quotes with two lots of single quotes
  else
    return "NULL"
  end -- if
end -- fixsql


I'm using MUSHclient 5.06-pre having come from using the Aardwolf installer version. But I also just ran the same map and GMCP plugins on a fresh install of version 5.05 now and the issue is consistent across both.

I can replicate the result around room 060949403E41 consistently now too - with fresh database files.

First encounter of the room as an exit destination looks fine:
Added full exit s from room 1A8549F81F6C to room 060949403E41 to database.


The mapper draws this correctly and displays the info correctly immediately after this as it's reading it from a lua table called 'room.exits':
"s"="060949403E41"


But if I clear those tables and ask it to reload the room from the database, the error shows up in 'touid':
"s"="6.0949403e+048"


If I map that room by walking into it, the room uid will save and load that particularly room correctly as:
"s"="060949403E41"


But again, like above, the issue shows up in 'fromuid':
"s"="6.0949403e+048"


The fact that the database has no issues saving that string to the rooms table uid column I'm thinking must narrow the issue down to saving/modifying touid and fromuid with certain strings like this one.

I don't suppose there's a simple way to force the string to be treated as such, like encasing it in additional "" that are saved with it?
Australia Forum Administrator #25
There's more going on here than meets the eye. We need to do some debugging. In the plugin, after opening the database, add the lines shown (in my case I tried it on the test script above):


db    = assert (sqlite3.open(GetInfo (66) .. "test.db"))

--> ADD THIS:

if old_db_execute == nil then
  old_db_execute = getmetatable (db).execute
end -- if

getmetatable (db).execute = function (this, what)
  ColourNote ("orange", "", "** SQL debugging: About to run: " .. what)
  return old_db_execute  (this, what)
end -- function


That will make db:execute (which is used for inserting and modifying the database) output a debugging line, and then do what it originally did.

Once you have done that, reproduce the problem, and see if there is in fact a SQL statement that does not put the exit into quotes.

With the test script I got this output:


** SQL debugging: About to run:   CREATE TABLE IF NOT EXISTS test ( t text );
  DELETE FROM test;
  INSERT INTO test (t) VALUES ('060949403E41');


That proves that the string was quoted.
#26
Alright, I've put it here in the plugin file following opening the db on disk:

function OnPluginInstall ()
  
  config = {}  -- in case not found

  -- get saved configuration
  assert (loadstring (GetVariable ("config") or "")) ()

  -- allow for additions to config
  for k, v in pairs (default_config) do
    config [k] = config [k] or v
  end -- for
  
  -- initialize mapper engine
  mapper.init { config = config,            -- colours, timing etc.
                get_room = get_room,        -- get_room (uid) called to get room info
                show_help = OnHelp,         -- to show help
                room_click = room_click,    -- called on RH click on room square
                timing = show_timing,       -- want to see timing
                show_completed = show_completed,  -- want to see "Speedwalk completed." message
                show_other_areas = show_other_areas,  -- want to see areas other than the current one?
                show_up_down = show_up_down,          -- want to follow up/down exits?
                show_area_exits = show_area_exits,    -- want to see area exits?
                speedwalk_prefix = speedwalk_prefix,  -- how to speedwalk
                } 
  
  mapper.mapprint (string.format ("MUSHclient mapper installed, version %0.1f", mapper.VERSION))
  
  -- open databases on disk 
  db    = assert (sqlite3.open(GetInfo (66) .. Trim (WorldAddress ()) .. "_" .. WorldPort () .. ".db"))
  db_bm = assert (sqlite3.open(GetInfo (66) .. Trim (WorldAddress ()) .. "_" .. WorldPort () .. "_bookmarks.db"))
  
  
  --> ADD THIS:

if old_db_execute == nil then
  old_db_execute = getmetatable (db).execute
end -- if

getmetatable (db).execute = function (this, what)
  ColourNote ("orange", "", "** SQL debugging: About to run: " .. what)
  return old_db_execute  (this, what)
end -- function


  create_tables ()    -- create database structure if necessary
  
  -- grab all area names
  for row in db:nrows("SELECT * FROM areas") do
    areas [row.uid] = row.name
  end   -- finding areas
  
  -- grab all user terrain info
  for row in db_bm:nrows("SELECT * FROM terrain") do
    user_terrain_colour [row.name] = row.color
  end   -- finding terrains
 
  -- grab all environment names
  for row in db:nrows("SELECT * FROM environments") do
    environments [tostring (row.uid)] = row.name		-- Geas: changed from tonumber
    terrain_colours [row.name] = tonumber (row.color)
  end   -- finding environments
  
end -- OnPluginInstall


Now when mapping the room as a southern exit:

** SQL debugging: About to run: INSERT INTO rooms (uid, name, area, date_added) VALUES ('1A8549F81F6C', 'The entrance to a light forest', '0', DATETIME('NOW'));
** SQL debugging: About to run:         INSERT INTO rooms_lookup (uid, name) VALUES ('1A8549F81F6C', 'The entrance to a light forest');
      
Added room 1A8549F81F6C to database. Name: The entrance to a light forest
** SQL debugging: About to run:         INSERT INTO exits (dir, fromuid, touid, date_added) 
            VALUES ('s', '1A8549F81F6C', '060949403E41', DATETIME('NOW'));
      
Added full exit s from room 1A8549F81F6C to room 060949403E41 to database.


As well as for mapping the room itself:
** SQL debugging: About to run: INSERT INTO rooms (uid, name, area, date_added) VALUES ('060949403E41', 'A road junction surrounded by forest', '0', DATETIME('NOW'));
** SQL debugging: About to run:         INSERT INTO rooms_lookup (uid, name) VALUES ('060949403E41', 'A road junction surrounded by forest');
      
Added room 060949403E41 to database. Name: A road junction surrounded by forest

** SQL debugging: About to run:         INSERT INTO exits (dir, fromuid, touid, date_added) 
            VALUES ('n', '060949403E41', '1A8549F81F6C', DATETIME('NOW'));
      
Added full exit n from room 060949403E41 to room 1A8549F81F6C to database.


Should I put that debug function elsewhere in the plugin for it to read the loading part of the sequence?
Australia Forum Administrator #27
When you read, you don't quote stuff (eg. SELECT foo FROM bar) because they are just variable names.

Are you saying that after running the stuff you posted above, the data on the database is actually wrong (if you use the sqlite3 program to read it)?

For example:


SELECT * FROM exits WHERE touid = '1A8549F81F6C';


I also suggest scanning the entire plugin to see if there is a stray "tonumber" there.
#28
That's right. Despite the debug messages, the data showing up in the db file when reading it with either the sqlite3 or with the MUSH mapper is incorrect.

I've stripped back the plugin in a new version, just keeping the basic functions that map via:
1. uid as string
2. room name
3. full exits with destination
4. area name

I could post that in full if it will help and some instructions on how to produce these results to see if it can be replicated.

Even stripped back quite a lot the plugin is 840 lines though.
#29
Solved! :D :D :D

And it looks so obvious now I'm wondering how it never occurred to me.

The issue was this:

CREATE TABLE IF NOT EXISTS exits (
      exitid      INTEGER PRIMARY KEY AUTOINCREMENT,
      dir         TEXT    NOT NULL, -- direction, eg. "n", "s"
      fromuid     STRING  NOT NULL, -- exit from which room (in rooms table)
      touid       STRING  NOT NULL, -- exit to which room (in rooms table)
      date_added  DATE              -- date added to database
    );


I just had to change the table for touid and fromuid to TEXT rather than STRING.

It still seems odd now though that the STRING type in the database would evaluate only some very particular strings as numbers.
Australia Forum Administrator #30

Hmm, I see. And the quick test I did used TEXT not STRING.

From this SQLITE3 page:

3.1. Determination Of Column Affinity

The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown:

  1. If the declared type contains the string “INT” then it is assigned INTEGER affinity.

  2. If the declared type of the column contains any of the strings “CHAR”, “CLOB”, or “TEXT” then that column has TEXT affinity. Notice that the type VARCHAR contains the string “CHAR” and is thus assigned TEXT affinity.

  3. If the declared type for a column contains the string “BLOB” or if no type is specified then the column has affinity BLOB.

  4. If the declared type for a column contains any of the strings “REAL”, “FLOA”, or “DOUB” then the column has REAL affinity.

  5. Otherwise, the affinity is NUMERIC.

Note that the order of the rules for determining column affinity is important. A column whose declared type is “CHARINT” will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.

Since STRING is not in the list, the affinity is NUMERIC.

Well spotted!