Project

General

Profile

Miscellaneous Notes

Check list(s) of work hopefully to be completed by the end of the project

The git repository

git clone ssh://p-lqcdmonitoringdb@cdcvs.fnal.gov/cvs/projects/lqcdmonitoringdb

yappur

To install: retrieve from the git repository and run from the retrieved directory.

This the web interface into the parameter set database.

:32767 works with the production copy of the database and :32766 works with the development copy of
the database.

The current version is in yappurV3.

This is a Ruby on Rails application. The application administrator's password is in the db/seeds.rb file.

It is running currently from terminal windows on my work station.

cluster_monitoring database

To install: install the cluster monitoring Rails appication, modify if necessary config/database.yml and run db:migrate.

The data base is resident on ds2.

The TCP/IP port, 5432, is available to 131.225.0.0/16 (basically FNAL on-site machines).
It would be avaialble to machines using the FNAL VPN facility.

The database construction code is in the cluster_monitoring (immediately below).

The data base currently is backed up using pg_dump at 19:00 each day.
The crontab entry belongs to the user postgres.

The script /home/postgres/makepgdumps:

#!/bin/bash -x
PLACE=/home/postgres
FORMAT="ccyy hyphen mm hyphen dd" 
if cd "$PLACE" 2>/dev/null; then
  /bin/pwd
  connection="dbname=jml_prod user=herber host=ds2 password=MagicMaster" 
  /usr/bin/pg_dump "$connection" -Z9 -f temporary.dmp.gz
  rc=$?
  if [ "$rc" -eq 0 ]; then
    today=$(/usr/local/bin/getdate in $FORMAT)
    /bin/mv temporary.dmp.gz ${today}.dmp.gz
    twodaysago=$(/usr/local/bin/getdate in $FORMAT, minus 2 days).dmp.gz
    /bin/ls -1 20??-??-??.dmp.gz | \
    /bin/gawk '{if($0<"'"$twodaysago"'") print}' | \
    xargs rm
  else
    echo /usr/bin/pg_dump exit code is $rc.  File is still temporary.dmp.gz.
    exit $rc
  fi
else
  echo Unable to cd "$PLACE" 
  exit 1
fi
The database can be and has been reconstructed by this method:
  1. Stop the appropriate scanner on the job manager machines.
    • The scanners are ran from /etc/inittab.
    • Comment out the appropriate line
    • /sbin/telinit q tell init to stop the process(es).
    • ps -flea | grep scan verify the process(es) have stopped.
  2. Log into the postgres account by some means on ds2.
  3. Log into the database server using psql.
  4. Drop the desired database: drop database jml_prod, e.g.
  5. Create a new database: create database jml_prod, e.g.
  6. Assign some account all privileges on the database: grant all on jml_prod to herber, e.g.
  7. Quit psql: \q
  8. Log into the selected account, e.g., herber
  9. Using the selected backup, e.g., do: gunzip /home/postgres/2011-09-14.dmp.gz | psql -h ds2 -d jml_prod
    It will be necessary to supply the selected account's Postgres password.
  10. Wait.
  11. Start the appropriate scanner on the job manager machines.
    • The scanners are ran from /etc/inittab.
    • Uncomment the appropriate line
    • /sbin/telinit q tell init to start the process(es).
    • ps -flea | grep scan verify the process(es) have started.

cluster_monitoring

The application is in the cluster_monitoring directory of the git repository.

In particular, this application contains the log monitoring database construction code

complex_work_flow_engine

A Ruby on Rails workflow engine. Neither completed nor tested.

simple_work_flow_engine

Only available as source; retrieve from the git repository.

A Bash script based workflow engine. Completed and tested.

While cd'ed into the directory

./workflow -t Template -m metadata -w participants -a execute -x 1

The program works correctly for both 1 and 2 concurrent processes.

There is no reason to believe that it would not work correctly
with more than 2; but, that is the largest amount actually tested.

The program correctly detects that it has executed to completion.

I believe that it would handle also a "clean" restart. A "clean"
restart is one where any remaining processes from previous
executions have terminated.

The script uses wrapped participants and executes them in separate
processes. In this example, the wrapped participants are shell
scripts.

I am not checking the manifests as of yet.

I have not checked wrapped participants which submit and wait
for PBS job completions.

The workflow description, "Template":

Start: { output: "initiation" },
make_constant_A: { input: "initiation",
                   output: ["constantA","value"],
                   participant: "selectRandom",
                   time: 3 },
make_constant_B: { input: "initiation",
                   output: ["constantB","value"],
                   participant: "selectRandom",
                   time: 2 },
combine_AB:      { input : [["constantA","left"],["constantB","right"]],
                   output : ["result_AB","output"],
                   participant : "combiner",
                   time : 4 },
combine_BA:      { input : [["constantB","left"],["constantA","right"]],
                   output : ["result_BA","output"],
                   participant : "combiner",
                   time : 4 },
compare:         { input: [ ["result_AB","first"] , ["result_BA","second"] ],
                   output : ["report","report"],
                   participant: "comparer",
                   time: 1 },
Done: { input: "report" }

The workflow description is written in the parameter set language.
For each named step, its inputs, outputs, participant (program) and
expected execution time is given. An input or output may be a single
string naming an input to output or a tuple of such a string and the
name of the parameter to which the actual files are named. A unnested
tuple is assumed to embedded in another array. Arrays of inputs or
outputs are permitted.

The simple workflow engine can produce a graphwiz dot language dependency
graph of the workflow as well as run the workflow.

mangle and pandora

To install: retrieve from the git repository. install pandora into your Ruby environment.
'mangle' requires that the yapser gem be installed into your Ruby environment.
You may also want to install pandora, an example embedded templating helper class.

'mangle' is Ruby script accepting a template file and a sequence of parameter set files
or URLs to such files. The Yappur web service currently running on oink.fnal.gov can
provide such files. The parameter set values are merged from left to right inside the
program; therefor, the last parameter set to specify a value "wins."

The purpose for permitting the use of parameter set files is so that mangle and
parameter_value can be used on worker nodes which do not have access to the outside,

parameter_value uses a specialized variant of mangle to extract
parameter values from a set of parameter sets.https://secure.wikimedia.org/wikipedia/en/wiki/Here_document

The template file is, in effect, a here document
using $< variable_expression > to inject variable values into the output,
$<: code fragment > to inject code fragments into the generated program without
visible effect and $&<_optional space text> to provide for the removal of leading
white space.

Python implementation of the parameter set parser

To install: retrieve from the git repository and move to wherever desired. It is a standalone program.

The application is in the PyParser directory of the git repository.

The source also is in the files area as parser.py, with a test program
test.py, The test program also provides an usage example.

This was developed using Python 2.7.1.
Earlier versions of Python might not work.

This is a hand-written recursive descent parser.

This parser and tokenizer passes all of the tests of the Ruby implementation, yapser.

Its tokenizer uses regular expressions to recognize tokens and accepts the first longest match.

Ruby implementation of the parameter set parser

To install: avail the yapser-1.0.9.gem and install into your Ruby environment.
The gem is in the Redmine files area for this project and call be built by retrieving
yapserV3 and doing gem build.

The source is contained within a Ruby gem in the files area as yapser-1.0.9.gem.

The application is in the yapser directory of the git repository.

This was developed using Ruby 1.9.2
Earlier versions of Ruby might not work.

This is a hand-written recursive descent parser.

Its tokenizer uses regular expressions to recognize tokens and accepts the first longest match.

The use of a hand-written parser and tokenizer was to maximize the program portability.
An example of such porting is the Python implementation, which took about 3 FTE days, including testing.

Scripts

All scripts are single files and can be installed as desired.
Many of the scripts require a recent Postgres installation.

get_last_queue.c

This is an example C / Postgres program used to denormalize the last queue value from the
job details to the jobs record itself.

jobstat

This Bash shell script is used in a number of ad hoc workflow systems. 'qstat' only returns the
status of a job when PBS is still managing a job; when the job is complete, 'qstat' reports nothing.
'jobstat' returns the qstat answer, if any, and if not, then returns the answer from the Postgres
database. It is based on a previous version written by Ron Rechenmacher which used the PBS logs
as the database.

mass

This Bash script uses the database to return week by week account and total core-hour usages.
This was used to debug the log scanning and database contents by comparing these results with
those of Ken Schumacher's accounting. Errors were found and corrected. The two systems now
report the same answers.

selectjob

Usage: ${name} [-A] [-c cluster] [-s startTime] [-e endTime]
         [-j jobNumber] [-i element] [-n node] [-u user] [-a account]
         [-q queue] [-x exitCode ] [-h] [-S]

start and end times: whatever getdate accepts without error
  at least the following are accepted: mm/dd/yyyy hh24:mi:ss,
  yesterday [hh24:mi:ss] and [hh24:mi:ss].

-A flags an array job (only actually needed for the master job)

jobNumber: single job number

element: element number in job array (non empty implies array job)

node: single node name

user: single user name

account: single account name

queue: single name name

exitCode: decimal number or !0

-h or an error produces this text

-S: show the generated SQL

whichjob

Usage: node 'YYYY-MM-DD HH24:MI:SS'

This Bash scripts tries to determine which job was running on a node at a specific time.
Often a node crashes in a manner that does not permit a job to report the difficulty.
The cluster administrators would like to notify the affected user.

simone

This Postgres query was written to find the wall seconds and exit statuses of
Jim Simone's Zvhh jobs that executed on JPSI over the Labor Day weekend. Those
data are hard-coded into the query.

simone-graphic

This Postgres query was written to display a histogram of the wall seconds of
Jim Simone's Zvhh jobs that executed on JPSI over the Labor Day weekend. The
query produces the histogram directly.

LogCheck

This Bash shell script avails a list of all log record positions recorded in
the jml_prod database, splits that list into a list for each cluster job
manager and ssh remote shell commands using gawk avails a list
of all log record positions in the logs of each of the cluster job managers
and diffing the lists. Because of the separation in time
between when the database was examined and when the job managers' logs are
each examined, the job managers' log may have a few recent entries that the
database does not have and this is normal.

high_state_counts

This script accepts an interval specification and defaults to "1 day".
It reports all jobs that were queued
during that interval before now that had more than one instance of a given
valid job state so far.

currency

This script accepts an interval specification and defaults to "1 day".
It reports the log time of the
last log entry during that interval back from now. If a cluster is not
reported, then that cluster had not log entries during the interval.

histogram

This scripts a variety of job selection criteria and computes a text histogram of the wall seconds used by those selected jobs.

frequent

This SQL query reports those nodes which were involved in short (less than 3 minutes) and a specified cluster (JPSI) and in
a specified interval (first two weeks of September 2011) more than 1.5 standard deviations above the mean number of times
for such nodes.

h3 reran

conditions sets cluster and interval
with_a_rerun selects jobs which had at least one rerun
earlier_starts selects the starts of the with_a_rerun jobs before the
last start (presumed to be the run that either was deleted, aborted
or successful.
node_counts counts the number of times that a node was associated
with one of those earlier starts
the main query reports the node counts with the node names researched

with
conditions as (
  select to_timestamp('2011-09-01 00:00:00','YYYY-MM-DD HH24:MI:SS') beginning,
         to_timestamp('2011-09-14 00:00:00','YYYY-MM-DD HH24:MI:SS') ending,
         (select id from clusters where name = 'JPSI') cluster_id
),
with_a_rerun as (
  select distinct job_id
  from jobs j
  join job_logs l on(j.id = l.job_id)
  join conditions c on(    j.cluster_id = c.cluster_id
                       and l.log_time >= c.beginning
                       and l.log_time <  c.ending)
    and valid_job_state_id = (
          select id from valid_job_states where code = 'R')
),
earlier_starts as (
  select l.id job_log_id
  from job_logs l
  join with_a_rerun r on(l.job_id = r.job_id)
  join jobs j on(j.id = l.job_id)
  where l.log_time < j.last_start
    and l.valid_job_state_id = (
          select id from valid_job_states where code = 'S')
),
node_counts as (
  select jn.node_id, count(*) "#" 
  from job_nodes jn
  join earlier_starts es on(jn.job_log_id = es.job_log_id)
  group by jn.node_id
)
select n.name, "#" 
from nodes n
join node_counts nc on(nc.node_id = n.id)
order by "#" desc, n.name
   name   | # 
----------+---
 jpsi1327 | 6
 jpsi0112 | 5
 jpsi0139 | 5
 jpsi0143 | 5
 jpsi0144 | 5
 jpsi0201 | 5
 jpsi0211 | 5
 jpsi0221 | 5
 jpsi0306 | 5
 jpsi0420 | 5
 jpsi0421 | 5
 jpsi0427 | 5
 jpsi0430 | 5
 jpsi0431 | 5
 jpsi0432 | 5
 jpsi0433 | 5
 jpsi0438 | 5
 jpsi0439 | 5
 jpsi0440 | 5
 jpsi0924 | 5
 jpsi1017 | 5
 jpsi1111 | 5
 jpsi1315 | 5
 jpsi1316 | 5
 jpsi1320 | 5
 jpsi1321 | 5
 jpsi1322 | 5
 jpsi1323 | 5
 jpsi1324 | 5
 jpsi1325 | 5
 jpsi1326 | 5
 jpsi1328 | 5
 jpsi1001 | 1
 jpsi1004 | 1
 jpsi1006 | 1
 jpsi1008 | 1
 jpsi1012 | 1
 jpsi1013 | 1
 jpsi1018 | 1
 jpsi1020 | 1
 jpsi1021 | 1
 jpsi1022 | 1
 jpsi1023 | 1
 jpsi1024 | 1
 jpsi1025 | 1
 jpsi1026 | 1
 jpsi1027 | 1
 jpsi1028 | 1
 jpsi1029 | 1
 jpsi1030 | 1
 jpsi1031 | 1
 jpsi1033 | 1
 jpsi1034 | 1
 jpsi1343 | 1
 jpsi1344 | 1
 jpsi1401 | 1
 jpsi1402 | 1
 jpsi1403 | 1
 jpsi1404 | 1
 jpsi1405 | 1
 jpsi1413 | 1
 jpsi1414 | 1
 jpsi1415 | 1
(63 rows)

A discussion of recursive descent parsing

Some useful links

An edited quotation from Wikipedia

A recursive descent parser is built from a set of mutually-recursive procedures where each such procedure
usually implements one of the production rules of the grammar. Thus the structure of the resulting program
closely mirrors that of the grammar it recognizes.

Implementing the Kleene operators

a := b
boolean a() {
  b();
}

a: b?
boolean a() {
  b();
  return true;
}

a := b*
boolean a() {
  while(b()) {}
  return false;
}

a := b{n,m}
boolean a(int n,int m) {
  current_tokenizer_position = tokenizer.position();
  int count = 0;
  while(b()) ++count;
  if(count>= n && count <= 
  if(b()) return true;
  tokenizer.position(current_tokenizer_position);
  return false;
}

a := 0 | '(' a ( ',' a )* ')'

boolean a() {
  current_tokenizer_position = tokenizer.position();
  boolean successful = false;
  if(zero()) {
    successful = true;
  } else {
    if(token('(')) {
      if(a()) {
        while(token(',') && a()) {}
        if(token(')')) {
          successful = true;
        }
      }
    }
  }
  if(!successful) tokenizer.position(current_tokenizer_position);
  return successful;
}

Simplified parsers in Ruby with pre-done tokenization

This version only determines whether the token stream is valid.

@tokens = [ 5, '+', 7, '*', '(', 2, '+', 3, ')', 'EOT' ]

@next_token = 0;

def is_number
  begin
    Kernel.Complex(@tokens[@next_token])
    @next_token += 1
    return true
  rescue
    return false
  end
  return nil
end

def is_a(symbol)
  begin
    if @tokens[@next_token] == symbol
      @next_token += 1
      return true
    end
  rescue
    return fals3
  end
  return nil
end

# factor := NUMBER | '(' expression ')

def factor
  current_token = @next_token
  return true if is_number
  @next_token = current_token
  return true if is_a('(') && expression() && is_a(')')
  @next_token = current_token
  return false
end

# term := factor ('*' | '/') term | factor

def term
  current_token = @next_token
  return true if factor() && (is_a('*') || is_a('/')) && term()
  @next_token = current_token
  return true if factor()
  @next_token = current_token
  return false
end

# expression := term ('+' | '-') expression | term

def expression
  current_token = @next_token
  return true if term() && (is_a('+') || is_a('-')) && expression()
  @next_token = current_token
  return true if term()
  @next_token = current_token
  return false
end

# value := expression 'EOT'

def value
  current_token = @next_token
  return true if expression && is_a('EOT')
  @next_token = current_token
  return false
end

puts "parse result is #{value()}." 
The execution result:$ ruby sp.rb
parse result is true.

This version has embedded evaluation and tracing code

The structure of the two versions is the same.
The addition of the tracing code and of embedded evaluation of the expression make it appear more complex;
in fact, it is not!

@tokens = [ 5, '+', 7, '*', '(', 2, '+', 3, ')', 'EOT' ]

@next_token = 0;

def remainder
  "#{@tokens[@next_token..-1].join(',').inspect}" 
end

def indent
  "  "*((Kernel.caller.size)-2)
end

def is_number
  print "#{indent}is_number(#{@tokens[@next_token].inspect})" 
  begin
    v = Kernel.Complex(@tokens[@next_token])
    puts "#{v.inspect}" 
    @next_token += 1
    return v
  rescue
    puts ' false'
    return nil
  end
  puts ' false'
  return nil
end

def is_a(symbol)
  print "#{indent}is_a(#{@tokens[@next_token].inspect},#{symbol.inspect})" 
  begin
    if @tokens[@next_token] == symbol
      @next_token += 1
      puts ' true'
      return symbol
    end
  rescue
    puts ' false'
    return nil
  end
  puts ' false'
  return nil
end

# factor := NUMBER | '(' expression ')

def factor
  puts "#{indent}factor(#{remainder})" 
  current_token = @next_token
  v = is_number
  if v
    puts "#{indent}factor(#{remainder}) #{v.inspect}" 
    return v
  end
  @next_token = current_token
  if is_a '('
    v = expression
    if v
      if is_a ')'
        puts "#{indent}factor(#{remainder}) #{v.inspect}" 
        return v
      end
    end
  end
  @next_token = current_token
  puts "#{indent}factor(#{remainder}) returns false" 
  return nil
end

# term := factor ('*' | '/') term | factor

def term
  puts "#{indent}term(#{remainder})" 
  current_token = @next_token
  v1 = factor
  if v1
    op = is_a '*'
    op = is_a '/' unless op
    if op
      v2 = term
      if v2
        if op == '*'
          v = v1 * v2
        else
          v = v1 / v2
        end
        puts "#{indent}term(#{remainder}) #{v.inspect} is #{v1}#{op}#{v2}" 
        return v
      end
    end
  end
  @next_token = current_token
  v = factor
  if v1
    puts "#{indent}term(#{remainder}) #{v.inspect}" 
    return v
  end
  puts "#{indent}term(#{remainder}) false" 
  @next_token = current_token
  return false
end

# expression := term ('+' | '-') expression | term

def expression
  puts "#{indent}expression(#{remainder})" 
  current_token = @next_token
  v1 = term
  if v1
    op = is_a '+'
    op = is_a '-' unless op
    if op
      v2 = expression
      if v2
        if op == '+'
          v = v1 + v2
        else
          v = v1 - v2
        end
        puts "#{indent}expression(#{remainder}) #{v.inspect} is #{v1}#{op}#{v2}" 
        return v
      end
    end
  end
  @next_token = current_token
  v = term
  if v
    puts "#{indent}expression(#{remainder}) #{v.inspect}" 
    return v
  end
  puts "#{indent}expression(#{remainder}) false" 
  @next_token = current_token
  return false
end

# value := expression 'EOT'

def value
  puts "#{indent}value(#{remainder})" 
  current_token = @next_token
  v = expression
  if v
    if is_a 'EOT'
      puts "#{indent}value(#{remainder}) #{v.inspect}" 
      return v
    end
  end
  @next_token = current_token
  puts "#{indent}value(#{remainder}) false" 
  return false
end

puts "parse result is #{value()}." 

The execution results, including tracing and evaluation:

herber@xyzzy herber$ ruby p.rb
value("5,+,7,*,(,2,+,3,),EOT")
  expression("5,+,7,*,(,2,+,3,),EOT")
    term("5,+,7,*,(,2,+,3,),EOT")
      factor("5,+,7,*,(,2,+,3,),EOT")
        is_number(5)(5+0i)
      factor("+,7,*,(,2,+,3,),EOT") (5+0i)
      is_a("+","*") false
      is_a("+","/") false
      factor("5,+,7,*,(,2,+,3,),EOT")
        is_number(5)(5+0i)
      factor("+,7,*,(,2,+,3,),EOT") (5+0i)
    term("+,7,*,(,2,+,3,),EOT") (5+0i)
    is_a("+","+") true
    expression("7,*,(,2,+,3,),EOT")
      term("7,*,(,2,+,3,),EOT")
        factor("7,*,(,2,+,3,),EOT")
          is_number(7)(7+0i)
        factor("*,(,2,+,3,),EOT") (7+0i)
        is_a("*","*") true
        term("(,2,+,3,),EOT")
          factor("(,2,+,3,),EOT")
            is_number("(") false
            is_a("(","(") true
            expression("2,+,3,),EOT")
              term("2,+,3,),EOT")
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
                is_a("+","*") false
                is_a("+","/") false
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
              term("+,3,),EOT") (2+0i)
              is_a("+","+") true
              expression("3,),EOT")
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
                is_a(")","+") false
                is_a(")","-") false
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
              expression("),EOT") (3+0i)
            expression("),EOT") (5+0i) is 2+0i+3+0i
            is_a(")",")") true
          factor("EOT") (5+0i)
          is_a("EOT","*") false
          is_a("EOT","/") false
          factor("(,2,+,3,),EOT")
            is_number("(") false
            is_a("(","(") true
            expression("2,+,3,),EOT")
              term("2,+,3,),EOT")
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
                is_a("+","*") false
                is_a("+","/") false
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
              term("+,3,),EOT") (2+0i)
              is_a("+","+") true
              expression("3,),EOT")
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
                is_a(")","+") false
                is_a(")","-") false
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
              expression("),EOT") (3+0i)
            expression("),EOT") (5+0i) is 2+0i+3+0i
            is_a(")",")") true
          factor("EOT") (5+0i)
        term("EOT") (5+0i)
      term("EOT") (35+0i) is 7+0i*5+0i
      is_a("EOT","+") false
      is_a("EOT","-") false
      term("7,*,(,2,+,3,),EOT")
        factor("7,*,(,2,+,3,),EOT")
          is_number(7)(7+0i)
        factor("*,(,2,+,3,),EOT") (7+0i)
        is_a("*","*") true
        term("(,2,+,3,),EOT")
          factor("(,2,+,3,),EOT")
            is_number("(") false
            is_a("(","(") true
            expression("2,+,3,),EOT")
              term("2,+,3,),EOT")
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
                is_a("+","*") false
                is_a("+","/") false
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
              term("+,3,),EOT") (2+0i)
              is_a("+","+") true
              expression("3,),EOT")
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
                is_a(")","+") false
                is_a(")","-") false
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
              expression("),EOT") (3+0i)
            expression("),EOT") (5+0i) is 2+0i+3+0i
            is_a(")",")") true
          factor("EOT") (5+0i)
          is_a("EOT","*") false
          is_a("EOT","/") false
          factor("(,2,+,3,),EOT")
            is_number("(") false
            is_a("(","(") true
            expression("2,+,3,),EOT")
              term("2,+,3,),EOT")
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
                is_a("+","*") false
                is_a("+","/") false
                factor("2,+,3,),EOT")
                  is_number(2)(2+0i)
                factor("+,3,),EOT") (2+0i)
              term("+,3,),EOT") (2+0i)
              is_a("+","+") true
              expression("3,),EOT")
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
                is_a(")","+") false
                is_a(")","-") false
                term("3,),EOT")
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                  is_a(")","*") false
                  is_a(")","/") false
                  factor("3,),EOT")
                    is_number(3)(3+0i)
                  factor("),EOT") (3+0i)
                term("),EOT") (3+0i)
              expression("),EOT") (3+0i)
            expression("),EOT") (5+0i) is 2+0i+3+0i
            is_a(")",")") true
          factor("EOT") (5+0i)
        term("EOT") (5+0i)
      term("EOT") (35+0i) is 7+0i*5+0i
    expression("EOT") (35+0i)
  expression("EOT") (40+0i) is 5+0i+35+0i
  is_a("EOT","EOT") true
value("") (40+0i)
parse result is 40+0i.