Gammon Forum : Programming : General : Expression parser

Log on

Gammon Forum

Entire forum

Programming

General

Expression parser

Expression parser

It is now over 60 days since the last post. This thread is closed. Refresh page

Pages: 1 2 3 4 5

Posted by

Djones USA (3 posts)

bio

Date

Reply #30 on Sat 31 Jan 2009 01:15 AM (UTC)

Message

Nick,

Yes, it *seems* to work fine. I just wasn't sure if I was missing anything obvious or asking for trouble by making the change.

- Dennis

top

Posted by

Bill Bach (1 post)

bio

Date

Reply #31 on Wed 11 Mar 2009 07:44 PM (UTC)

Message

Nick:

Stumbled upon this recently while searching for an expression parser. Sweet piece of code!

I do have one question for you, though. What I am writing is a data extraction tool that offers generic filtering capability based on the expression. I can now easily create and parse filters like
CustID == 76555
by adding all SQL field names as their own variables. That's just awesome!

However, one thing is missing -- the ability to evaluate string expressions, as in
CountryCode == "AUS"
or
Last_Name < "C"
(The latter makes it easy to generate test data with a smaller subset of data.)

I understand that string expression support would bloat the code in terms of storage space, but I can see this as a very useful feature to add.

Anyway -- my question is this: Has anyone ever asked about adding string comparison support, or have you ever looked into extending the code for this? I think it won't be too bad to extend the existing code by adding a data type field and a bunch of IF checks for the string fields, and even some simple string handling functions (LEFT, RIGHT< etc.), but I'd hate to re-invent the wheel if I can avoid it, and Lua seems a bit overkill.

top

Posted by Nick Gammon Australia (22,985 posts) bio Forum Administrator

Date

Reply #32 on Wed 11 Mar 2009 08:38 PM (UTC)

Amended on Wed 11 Mar 2009 08:39 PM (UTC) by Nick Gammon

Message

No-one has asked, and I haven't tried it. The considerable complexity would arise from having to have data types (ie. numbers, strings) where it doesn't currently have that. Then when you do things (like testing for less-than) you would need to switch between numeric or string compare, or indeed throw an error if not possible.

If you want to go that far, I would use Lua. It is extremely simple to imbed in an application (particularly C or C++), and you can always not load unwanted libraries (eg. the io or package libraries).

Then you only have to get your variables into/out of Lua address space, which it is specifically designed to do with a minimum of hassle. See my examples here:

http://www.gammon.com.au/forum/bbshowpost.php?id=6400

Something like your example could be:


if string.sub (Last_Name, 1, 1) == "C" then   -- whatever

This also gives you the advantage that your application (the data extraction tool) can then lever off things Lua offers, for example regular expression matching.

So for example, you could test if the last name started with one of a set of characters, eg.


if string.match (Last_Name, "^[CbvX]") then   -- whatever

Also, for documentation, refer your users to the Lua documentation.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

RayGun (2 posts)

bio

Date

Reply #33 on Fri 27 Mar 2009 01:26 AM (UTC)

Message

I get an error if I try to parse something like:

(2+3)-1
exception: Unexpected text at end of expression: -1

(2+3) +4-5
exception: Unexpected text at end of expression: +4-5

It seems to be something to do with how you handle RHPAREN.

top

Posted by Nick Gammon Australia (22,985 posts) bio Forum Administrator

Date Reply #34 on Fri 27 Mar 2009 02:57 AM (UTC)

Message

Around line 515, change:


    case LHPAREN:
      {
      double v = CommaList (true);    // inside parens, you could have commas
      CheckToken (RHPAREN);
      GetToken ();                    // eat the )
      return v;
      }

to:


    case LHPAREN:
      {
      double v = CommaList (true);    // inside parens, you could have commas
      CheckToken (RHPAREN);
      GetToken (true);                    // eat the )
      return v;
      }

That will fix it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by Nick Gammon Australia (22,985 posts) bio Forum Administrator

Date Reply #35 on Fri 27 Mar 2009 05:37 AM (UTC)

Message

By way of explanation, without the change, it was treating the -1 after the bracket as a single token < -1 > not two tokens namely < - > and < 1 >. Thus it was complaining that there was no operation after the brackets.

The distinction in the call to GetToken is to allow for something like this:

 
5+-6

The plus in this example is one token, and the -6 is another token, so we can validly build an arithmetic expression out of it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

RayGun (2 posts)

bio

Date

Reply #36 on Fri 27 Mar 2009 03:49 PM (UTC)

Message

Thanks for the quick response! :-)

top

Posted by

AdamTheEngineer (2 posts)

bio

Date

Reply #37 on Wed 15 Apr 2009 04:06 PM (UTC)

Message

Hi Nick,

I too found this recently and started to play with it. I discovered however that if I wanted to use the same expression but change a parameter value calling p.Evaluate again caused an error.

It seemed to work if I reset the expression using the Evaluate(exp) form.

So looking at the code I added 2 lines:

to the top of the definition of
const double Parser::Evaluate()
{
pWord = program_.c_str();
type = None;

....

This works. Is it necessary to do both?
Is there an uptodate version of the tarball with the all latest fixes in?

Thanks, Adam

top

Posted by Nick Gammon Australia (22,985 posts) bio Forum Administrator

Date Reply #38 on Thu 16 Apr 2009 07:38 AM (UTC)

Message

pWord_ is const, so I doubt you need to re-establish that.

You are probably correct that putting:


type_ = NONE;

into Evaluate is a good idea. I notice that the constructor for Parser has:


 type_ (NONE)

... so it is initially NONE (notice the caps), and putting it back as NONE for a re-evaluate is probably an excellent idea.

Quote:

Is there an uptodate version of the tarball with the all latest fixes in?

I haven't done a new .tar file, but will soon.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

AdamTheEngineer (2 posts)

bio

Date

Reply #39 on Thu 16 Apr 2009 08:26 AM (UTC)

Message

Thanks.

The "none" was a typo, I had NONE in the code.

Thanks for thisd, it's a handy bit of code.

FYI I added the the following lines to test.cpp to check the "re-evaluate" option works:

after the line:
double abc = p ["abc"];

// example of resetting symbol
p["abc"] = 52;

// parse expression again
value = p.Evaluate();

// display result
std::cout << "New Result = " << value << std:endl;

// example of retrieving symbol again
abc = p ["abc"];
std::cout << "New abc = " << abc << std:endl;

top

Posted by

Nick Gammon Australia (22,985 posts)

bio Forum Administrator

Date

Reply #40 on Wed 13 Jan 2010 01:15 AM (UTC)

Amended on Mon 15 Feb 2010 08:42 PM (UTC) by Nick Gammon

Message

Uploaded an improved version today:

~~http://www.gammon.com.au/files/utils/parser.tgz~~

(Now out of date, see below about GitHub).

File size: 8 Kb (source code only).

Changed getrandom to work more reliably (see page 2 of discussion thread)
Changed recognition of numbers to allow for .5 (eg. "a + .5" where there is no leading 0)
Also recognises -.5 (so you don't have to write -0.5)
Fixed problem where (2+3)-1 would not parse correctly (- sign directly after parentheses)
Fixed problem where changing a parameter and calling p.Evaluate again would fail because the initial token type was not reset to NONE.

The latest version is now available on GitHub:

http://github.com/nickgammon/parser

If you have git installed, you can do this to get a copy:

git clone git://github.com/nickgammon/parser.git

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Nick Gammon Australia (22,985 posts)

bio Forum Administrator

Date

Reply #41 on Fri 22 Jan 2010 11:01 PM (UTC)

Amended on Mon 15 Feb 2010 08:44 PM (UTC) by Nick Gammon

Message

This parser is now available on GitHub:

http://github.com/nickgammon/parser

You can download the complete source from:

http://github.com/nickgammon/parser/downloads

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

PacManFan (1 post)

bio

Date

Reply #42 on Wed 27 Jan 2010 02:59 PM (UTC)

Message

Nick,
I wanted to thank you for this parser, I've created several other parsers before using tools like COCO, and I've also handcrafted small expression parsers as well. Yours is extremely easy to use and fits my current programming project nicely.

I've added a few things into your code, and I wanted to give them back to you in case you want to add them into the code base.

I'm using this expression parser in a project to help evaluate binary bitstreams.
Specifically, I added in support for hex numbers:
0x01, 0xf, 0x1234, etc...

I also added in support for bitwise AND and OR expressions using the | and & operators
you can express things such as:
somevar = (0x3f & 0x23) | 0x1234

now, since you internally use doubles to store variables, I cast the intermediate results to unsigned longs before storing the results back into doubles

I additionally added in shift operators through the << and >> ops.

somevar = 0x2345 << 5
somevar2 = 123 >> 2

How should I give you the code? should I post it on here?

-PacManFan

top

Posted by Nick Gammon Australia (22,985 posts) bio Forum Administrator

Date

Reply #43 on Wed 27 Jan 2010 07:38 PM (UTC)

Amended on Wed 27 Jan 2010 07:39 PM (UTC) by Nick Gammon

Message

You can post it here, if it doesn't exceed 6000 characters, which I hope it won't. ;)

If you pulled the source from GitHub, you should be able to produce a patch file:


git format-patch master --stdout > my_patch.diff

(Or, a straight git "diff" between versions should work, I think).

Then paste that file here, with forum codes, and fixing up certain characters, if possible. See:

To make your code more readable please use [code] tags as described here.

If you can't easily fix the forum tags (basically you need to put \ before [ ] and \) just post it "as is" and I'll fix it up.

I'll take a look and release an updated version.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by Bruno Jobard (2 posts) bio

Date Reply #44 on Mon 15 Feb 2010 03:44 PM (UTC)

Message

Hello

and thanks a lot for your parser. It is exactly what I need.

However, I am experiencing a very strange limitation: if I invoke p.Evaluate() twice, it raises an exception. Here is the minimalist program that reproduces the "bug" (or feature?):


#include "parser.h"
#include <iostream>

int main()
{
  Parser p("x+y");
	
  p["x"] = 1;
  p["y"] = 2;
  std::cout << "Value1: " << p.Evaluate() << std::endl;

  p["x"] = 3;
  p["y"] = 4;
  std::cout << "Value2: " << p.Evaluate() << std::endl;

  return 1;
}

Here is the execution trace:


./parser 
Value1: 3
terminate called after throwing an instance of 'std::runtime_error'
  what():  Unexpected end of expression.
Abandon

If I change the second call to


  std::cout << "Value2: " << p.Evaluate("x+y") << std::endl;

it works as expected:


./parser 
Value1: 3
Value2: 7

But I guess the expression is parsed again, no? Why the parser doesn't keep the expression he received in the constructor for multiple calls to Evaluate()?

Thanks in advance,

Bruno.

top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

195,148 views.

This is page 3, subject is 5 pages long: 1 2 3 4 5

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.