What is Cexp?
-------------

 -- Brought to you by Till Straumann <strauman@slac.stanford.edu> --
 -- README,v 1.14 2004/05/21 21:09:21 till Exp --

The Cexp utility is a C-expression interpreter which
gives its user access to all symbols present in the currently
executing program. Its primary target is RTEMS, an open source
RTOS, but Cexp should compile and run on virtually any platform
supported by the BFD library.

If Cexp is linked to an executable, it is
possible to invoke arbitrary functions and to
read/write variables (and hence virtually any
memory location) by interpreting C-style expressions.

Cexp can access object files using either libelf
or libbfd. The latter is preferred as it is more powerful:
 - not limited to the ELF object format.
 - cexp implements a dynamic/runtime loader using libbfd
 - using BFD / libopcodes, a disassembler comes for free.
However, linking against the BFD library is subject to
the GPL (whereas libelf is under LGPL). See LICENSING
below for more information.

What is New?
------------

 - 2003/02/14: CEXP_1.2.beta ("Valentine") released.
   It supports G++ prioritized execution of static
   constructors/destructors.

 - 2003/02/13: CEXP_1.1.beta is released. It fixes a bug
   of the exception handler registration.
   
   Makefile.syms has been added and modularized copies
   of the RTEMS 'cdtest' sample program.

Description
-----------

CEXP is a simple utility featuring
 - symbol table
 - C expression parser/interpreter
 - type "engine"
 - user definable variables
 - recursive invocation / simple scripts
 - dynamic/runtime 'module' loader
 - disassembler
 - CEXP is reentrant

Cexp knows about the basic C types (no aggregates), i.e.

char,   char*
short,  short*
long,   long*
double, double*
long    (*)()
double  (*)()

and interprets C-style expressions, e.g.

  Cexp> printf("Hello world")

  Cexp> some_variable = 0xdeadbeef

  Cexp> some_double_variable = *(double*)&some_variable

  Cexp> a=printf, a && a("The square root of 2 is %g\n",sqrt(2.0))

Symbol Table
------------
A) Using a Symbol Table File
- - - - - - - - - - - - - - -
On startup, Cexp reads the 'system' symbol table either from 
the executable itself, or from a stripped-down '.sym'
file - such a stripped-down symbol file can be created using
the 'xsyms' utility.

B) Builtin Symbol Table File
- - - - - - - - - - - - - - -
The 'xsyms' tool (BFD version only) can now be used to generate
a symbol table in C-source form for subsequent compilation and
linkage into the final executable. This usually involves linking
the executable twice:
  1) compile and link all sources, libraries etc.; build an executable
  2) xsyms -C executable  mysymtab.c
  3) compile mysymtab.c (set -I to Cexp source directory, since
     cexpsyms.h is included)
  4) link application again, but this time add 'mysymtab.c' to the
     list of sources.
  NOTE: Step 1) linking succeeds without a symtab.o because there's
        a weak NULL-ptr alias for the builtin symtab.
A similar source file can also be generated using the 'ldep' utility
(discussed elsewhere, it is part of RTEMS-GeSys).

Without access to a symbol table file which also provides Cexp
with information about the architecture it is executing on, Cexp
might fail to detect the correct BFD CPU architecture which is
needed for the disassembler. In such cases, the '-a' command
line option can be used to specify the architecture (arch string
can be obtained by running 'objdump -f' on the ELF executable).

Symbol Type
- - - - - -

Once the symbol table is read, Cexp guesses the symbol/variable
types from the symbol sizes. (Guess is only supported on ELF targets.)

In some cases, this guess is obviously wrong, but you can 
always cast the symbol to the correct type.

Arrays are normally 'void' like every symbol for which's
type Cexp cannot make a guess. It is still possible, however, to
use such a symbol, simply by taking its address and
casting to a different pointer type:
    
Cexp> *(((long*)&some_array)+25)

evaluates to the unsigned long array element
some_array[25]. Alternatively, you can redeclare
its type:

Cexp> long *some_array !
Cexp> *(some_array + 25)

Note the exclamation mark: terminating a variable declaration with an
exclamation mark allows you to redefine its type. Only new variables
may be declared without the '!'. This protects you from erroneously
use existing symbols.

Cexp provides a series of lookup functions
for symbols. Most noteworthy:

lkup(char *regexp)
lkaddr(unsigned long addr)

These are just normal C functions which can be
invoked by Cexp. Note that lkup() takes a regular
expression argument allowing for powerful searching.

C-Expression Parser / Interpreter
---------------------------------
The C expression parser has some restrictions:
  - no '?' ':' expression
  - no [], '->'
  - the '.' operator has a special meaning - see below
  - no multilevel pointer dereferencing (since e.g. char**
    is not a valid type, Cexp cannot dereference 
    **addr. You must explicitly cast this:

      Cexp> *(char*)*(long*)some_char_pointer_pointer

which assumes that sizeof(long) == sizeof(char*)
for the particular machine.

However, most of the other operators are available, including
the 'comma' operator and the logical '&&' and '||' with the
correct semantics, i.e. conditional evaluation of expression
parts e.g.

  Cexp> (dd=deviceDetect(_some_device_address)) && driverInit(dd)

will call the driverInit() function only if deviceDetect()
returns a nonzero value.

The '.' operator (structure field access) has a special
meaning to Cexp: symbol "member" (in an OOP sense) access.
Currently, the only member function defined is 'help'. Consult
the 'Help' section about details.

Function calling:
The user is responsible for feeding properly typed arguments;
unused arguments will be filled with integral/scalar 0, which on 
most ABIs is safe.
Since the symbol table (on ELF, Cexp is not using .stab but .symtab)
provides no info about a function's return type, Cexp assumes
all functions to return long. Floating point functions
must be cast appropriately:

  Cexp> ((double(*)())sqrt)(2.345)

Or

  Cexp> double (*sqrt)()
  Cexp> sqrt(2.345)

A lot of type casting can be avoided by re-declaring or
using user defined variables, see below.


Type Engine
-----------
Cexp knows about (and only about) the primitive types listed
above.
NOTE: Cexp treats ALL integral types as UNSIGNED although
no 'unsigned' keyword is used or recognized. The main
reason for this is to save typing. Hence
char is in fact unsigned char etc.

An 'int' type is missing (but could easily added) which
means that on some machines (e.g. an alpha) no 32bit
type is currently available...

User Variables
--------------
Cexp supports (fully typed) user variables which are
visible in a global namespace (i.e. visible to other
instances of Cexp). The name of user variables must
not collide with symbols present in the symbol table.
(When loading object files [AKA 'modules'], name clashes
are ignored. A symbol in a newly loaded module that
conflicts with an already defined user variable is
silently ignored [object file symbols have a higher
priority than user variables]. However, after unloading
the conflicting module, the user variable becomes again
'visible'.)

A user variable is created simply by assigning it
a value which also automatically defines its type.

    hallo="hallo"

creates a 'char*' and assigns it the address of a
string constant. The string is stored in 'malloc()ated'
memory and 'lives' forever. Subsequent use of the
same string (strcmp(a,b)==0) results in re-using the
already stored instance of the string.
String constants must not be written-to!

The type of user variables can be modifed, simply
by re-declaring them, in a C-style manner:

    long hallo

results in 'hallo' maintaining its value (the address
of the string constant) but interpreting it as a long.
Hence

    chppt = &hallo

is an 'disguised' char** and

    *(char*)*chppt

yields 'h'

User variables can also hold function pointers and hence
can be handy abbreviations for long names and casts.
E.g. a convenient variable can be set for 'sqrt()'
(which returns a double value):

    s=((double(*)())sqrt)

It is then possible to automatically get the correct
return type:

    printf("Let's print a double: %g\n",s(44.34))

Help Facility
-------------

The '.' operator (structure field access) has a special
meaning to Cexp: symbol "member" (in an OOP sense) access.
Currently, the only member function defined is 'help'. I.e.
by typing

Cexp>  some_symbol.help()

You get help information about that symbol. If no help
is defined, or if you request 'verbose' help:

Cexp> some_symbol.help(1)

the symbol address, value and type information are printed
(same format as the 'lkup' output).

You can add/modify help information to a symbol (e.g. a user
defined variable) simply by providing a string argument to
the symbol's help member:

Cexp> long myvar
Cexp> myvar.help("I just created a 'long' variable")

Note that help is stored 'per symbol', i.e. when creating
an 'alias' as in the example above:

Cexp> s=(double (*)())sqrt

the help information is not propagated to 's'. If 'sqrt'
had help information, it would have to be copied:

Cexp> s.help(sqrt.help())

This example shows that the return value of the 'help() member'
is the address of the (static) help text string.

How to Start/Invoke Cexp - Simple Scripts
-----------------------------------------
Cexp has two entry points,

    cexp_main(int argc, char **argv)

and

    cexp(char *arg1,...)

i.e. a 'main' style and a 'vararg' style. The
'vararg' version ends up building an argument list
and calling cexp_main(). It is mainly intended
for recursively invoking 'cexp()' e.g. for 
reading a series of lines from a file.
Note that the string argument list submitted to
cexp() must be NULL terminated. When calling
cexp() from cexp() (e.g. to interpret a script),
this is not necessary, however, because any unused
arguments (up to the internal maximum of 10 args)
are filled with zeroes...

The calling syntax of cexp_main/cexp is as follows:

cexp [-s <symbolfile>] [-h] [-d] [-q] [<scriptfile>]

-h and -d have the obvious effect of printing usage
info and enabling debugging information (only available
if configured with --enable-YYDEBUG).

The -q ('quiet') flag instructs cexp not to print
any normal output (errors are still reported to stderr)
on stdout. This behavior can be desirable when evaluating
scripts.

Note that the -s option MUST be used the first time
'Cexp' is started and it must be provided with an
appropriate symbol file. This can be the executable
itself of a stripped version (use the 'xsyms' utility)
to reduce memory usage and loading time on RTEMS systems.
A basic check is made to protect against version mismatch
between the symbol file and the executable.

Once Cexp has loaded the system's symbol table, further
instances will simply use the global system table if
the -s argument is missing.

cexp then reads commands from stdin (using the TECLA
library) or alternatively from a script file.

Cexp ignores any characters present on a line
after it scans the comment tokens
'#' or '//'.

Example for invoking Cexp from Cexp to evaluate
a script:

  Cexp> cexp("script_file")

While the former syntax involves recursive execution of
Cexp, it is now also possible to let the current instance
divert its input to read from a script using the
'<' operator:

  Cexp> < script_file

There are slight syntactic differences when using this syntax,
though. Evaluation of the file name is not performed by
the parser but by a simple preprocessor. The filename doesn't have
to be embedded in double quotes (but it can be). Note that
no escape sequences are recognized. It is possible, however,
to use filenames with embedded whitespace by enclosing the
script name in single or double quotes. In this case, the
entire string between matching quotes is used. E.g., (RHS
string delimiter is '"' -- remember that there are no escape
sequences...):

  < a b c     --> read from file "a"
  < 'a b' "c" --> read from file "a b"
  < "a'b"     --> read from file "a'b"
  < "a\"b"    --> read from file "a\\" (a followed by a single backslash)


Loadable Modules / Runtime Loader
---------------------------------

When built aginst the BFD library, cexp is capable of
dynamically loading object files into a running program.
Cexp keeps track of module dependencies. Note that this
only covers symbol table dependencies. Cexp rejects
unloading a module 'A' if there is still another module,
'B' loaded which had undefined symbols resolved against 'A's
symbols. Obviously, more subtle dependencies, such as
threads using a modules text or data cannot be tracked
easily and are ignored.

An object file (AKA 'module') can be loaded invoking the
command:

  Cexp> someModule=cexpModuleLoad("someModule.o")

The loader returns a 'module ID', which in this example
is stored in the user variable 'someModule'. If errors
occur during the load (such as undefined references or
multiple symbol definitions), they are reported and a NULL
module ID is returned.

After loading, C++ static constructors are executed and
exception handler frames are registered. Note that these
features are probably only supported on ELF and, especially
the exception handling, might only work for gcc compiled
code. Unfortunately, even different versions of gcc and/or
target architectures involve varying implementations of
exception handling - YMMV...

An (unused) module can be unloaded by passing its ID to
cexpModuleUnload() (prior to unloading, the C++ static
destructors are executed):

  Cexp> cexpModuleUnload(someModule)

Two more routines are useful in this context:

  cexpModuleInfo([ID])

prints info about a specific module (ID) or all currently
loaded modules if passed NULL ID.

  cexpModuleFindByName("regexp")

searches the list of modules for a regular expression and
reports the IDs of all matches. It returns the first ID
found (or NULL if there was no match).

Search Path for Loadable Modules
- - - - - - - - - - - - - - - - -
If set, the PATH environment variable is consulted
by the loader when it tries to locate object files
by subsequently prepending colon-separated search
paths as listed in PATH to the file name. The search
starts with the first component and stops as soon
as the file is found. An empty search path (two colons
in a row or a leading or trailing colon) is equivalent
to the current directory ('.'). Note that the current
directory is not searched if PATH is set but does not
contain the CWD. PATH is ignored if the object name
contains a (relative or absolute) directory path already.

Example: search '/tmp', '/TFTP/BOOTP_HOST/blah/bin' and
finally the CWD:

setenv("PATH","/tmp:/TFTP/BOOTP_HOST/blah/bin:",1)

Building loadable modules
- - - - - - - - - - - - -
Note the important difference between a dymamically loaded
object and a _shared_ object (such as a shared library). The 
latter, although also 'dynamically loaded' is different from
the former. A shared object is shared among several entities
using disjunct address spaces (e.g. different UNIX processes).
Supporting shared objects involves PIC and GOTs.

***********************************************************

CEXP does _not_ support shared libraries! An attempt to run
a loaded module that was compiled with '-fpic', '-fPIC'
and/or linked with the '-shared' options will crash hard.

***********************************************************


To illustrate the difference, consider a LINUX system
running two instances of the 'cexp' demo. 'cexp' is linked
against glibc which is a shared library - both instances
of 'cexp' use the _same_ copy of libc residing in physical memory.
Let's now assume the both of the demo programs issue

	cexpModuleLoad("someObjectFile.o")

Cexp's dynamic loader doesn't support shared objects - hence
both instances of 'cexp' will end up with their own copy of
'someObjectFile' which will get loaded twice to physical memory.

The main target of Cexp is RTEMS, a real-time OS which has
a global address space shared by all threads. Hence, there is
no need for shared object support but dynamically loading
code is still very desirable.

The Cexp dynamic loader simply accepts _any_ relocatable object file.
The simplest modules are just object files:

	cross-gcc -c -O some_object.c

Multiple objects can be combined using the '-r' linker option

	cross-ld  -o some_object.o -r some1.o some2.o some3.o

It is also possible to convert entire (static) libraries into
loadable objects:

	cross-ld  -o lib_object.o -r --whole-archive libSome.a

The 'system' symbol table is itself an object file from where
Cexp loads the initial symbol table at startup. The executable
itself may be used for that purpose (unless it is a pure binary
as it is the case on some embedded systems). Alternatively
(for saving memory and time), a stripped-down object file containing
only the symbol table may be generated using the 'xsyms' tool
(available on ELF targets only), e.g:

	xsyms cexp cexp.sym
	cexp -s cexp.sym

Loadable Modules 'Magic'
- - - - - - - - - - - - -
When loading modules, 'cexp' does some magic operations on
special symbols it recognizes. Note that the all-capital names
given here are macros - the actual names can be found in the
header.

	CEXPMOD_INITIALIZER_SYM (defined in cexpmodP.h), when
		present in a loaded module, a routine with this
		name is invoked by cexpModuleLoad() just after calling
		C++ constructors.
		This 'module-constructor' routine [for C++ information
		see below] can be used to initialize a (non-C++) module.

	CEXPMOD_FINALIZER_SYM, when present in a loaded module,
		a routine with this name is invoked just prior to calling
		C++ destructors and unloading the module.
		The FINALIZER may reject the unloading attempt by
		returning a nonzero value.

	CEXP_HELP_TAB (defined in cexpHelp.h). A module may define
		(multiple) 'help' tables (their name must begin with
		the magic CEXP_HELP_TAB string) for providing help
		information about specific symbols contained in a module.
		See cexpHelp.h for more information about the help information.
		cexpModuleLoad() automatically registers this data.

Building the 'Main' Application
- - - - - - - - - - - - - - - -

When linking a traditional application, only objects referenced by the
application will be linked into the executable. E.g. an application which
does not use 'printf()' will not have that routine available. In the
normal case, this is fine. When using a runtime loader like CEXP, there
arises a problem: imagine you want to load a piece of code which _does_
use 'printf()':

  - Cexp resolves the module's undefined symbols, encounters 'printf'
    but doesn't find it in its symbol table - it rejects loading your
    module.

  - It is not possible either to simply link your module against libc!
    The module might reference other libc objects which _are_ present
    in the application already - Cexp would complain that their symbols
    are already defined.

Therefore, the 'primary' or 'system' application should be built to 
include ALL parts of the core libraries (such as the C-library, the
RTEMS executive managers etc.) which will possibly be used by modules
loaded into the running system.

Hence, there is a new task for the system designer which (for a statically
linked application) would be automatically performed by the linker:
You must now tailor/configure the core parts of your system. This
is essentially a memory/functionality tradeoff (Sidenote: vxWorks
is configured in such a way, too). This 'tailoring' essentially
means that you have to tell the linker what parts of the basic
system libraries (RTEMS managers, CPU/BSP support, libc, networking
etc.) should forcibly be included into the link.

Note that it is only necessary to take into account the libraries
needed by CEXP and the OS itself during this process.
Any library CEXP does _not_ depend on, directly or indirectly, can
of course be dynamically loaded any time later.

**************** SOMEHOW OBSOLETE BEGIN **********************

A special makefile is provided (Makefile.syms)
to help you generating the system configuration (RTEMS).
You can give the make process a list of libraries to work on.
'make' then generates a file containing a list of all symbols
exported by the libraries.

You can then browse this lists and delete/comment symbols or
chunks of symbols that you dont want to be present. In some
cases it is very much necessary to exclude certain symbols
from a link because an attempt to forcibly link them results
in linker errors. This can happen, e.g. if you try to enforce
linking a (under normal circumstances NEVER used) routine which 
has unresolved dependencies.

Makefile.syms can then be used to generate a special ELF
object file 'syms.o' containing nothing but symbol
references. 'syms.o' can then be linked with the application
(just add it to the list of the application's objects)
and will force the linker to pull in the referenced
library objects.

**************** SOMEHOW OBSOLETE ELSE ***********************

A new tool 'ldep' is available greatly alleviating the task
of analyzing link file interdependency and generating symbol
lists etc. It is available as part of the 'GeSys' package
(www.slac.stanford.edu/~strauman/rtems/gesys) and comes with
more documentation. Some info is available here:
http://www/~strauman/rtems/epics/README.config

**************** SOMEHOW OBSOLETE END ************************

C++ Information
- - - - - - - -

Dynamically loading C++ modules requires additional support.

	1) static constructors and destructors
	2) multiply defined instances of code (templates, multiple inclusion of headers)
	3) exception handling

Unfortunately, all of these items are highly implementation dependent and it is therefore
unlikely that C++ support works for compilers other than gcc. Alas, there are even
significant differences between different versions of gcc and target ABI's.

IMHO, C++ is unreadable, unportable, bloat-prone and should be avoided if at all
possible. That said, let's dive into the details:

	1) static constructors and destructors take care of initializing/finalizing
	   static objects such as:

		BlahClass	blahObj(a,b,c);
		int			test=testInitialize(x,y);

	   gcc creates code for initializing/finalizing these kinds of objects and
	   tags this code with symbols similar to '__GLOBAL__.I.xxx' / '__GLOBAL__.D.xxx'.
	   If Cexp encounters such symbols, it builds constructor/destructor lists when
	   loading the module and executes the respective code after loading / prior to
	   unloading.
	   Later versions of gcc use the '__cxa_atexit' callback mechanism which requires
	   CEXP to call '__cxa_finalize()' when a module is unloaded. This seems to be
	   a C++ ABI standard and is supported by Cexp.
	   There are other possible implementations (such as creating special ctor/dtor
	   sections) which are not supported - as I said, there is no general way for 
	   handling C++ :-(

	2) C++ allows for multiple definitions/instantiations of code (a header included
	   by more than one 'xxx.cc' file may define class members) plus there is the
	   'template' ''feature'' of C++.
	   These let C++ code size _explode_ keeping hardware manufacturers happy.
	   Gcc deals with this problem by putting every piece and bit of (possibly
	   redundant code and/or data) into a separate '.gnu.linkonce.t.xyzkljsbjj783c'
	   SECTION. The idea is that the linker (which is in our case 'Cexp') eliminates
	   redundant sections. The sad result are bloated object files and Cexp symbol
	   tables (Unfortunately, Cexp must keep _all_ of the zillions of
	   '.gnu.linkonce.x.yzu' symbols around since a redundant copy might be
	   loaded/linked at any time :-0 )

       You can help CEXP, however by using proper linker scripts when linking
       large C++ applications (such as EPICS).
       If you are sure that your large application is the only (C++) object (sharing
       common code), you may provide the linker with a script which instructs it
       to integrate _all_ the "gnu.linkonce.t.xxx" sections into ".text" etc.

       If you don't believe me - just let CEXP load an EPICS IOC and list all the
       'linkonce' symbols:

         lkup("linkonce")

       you will be surprised!
	   
	3) exception handling (while nice at the abstracion level) is another field
	   that is highly architecture/compiler/version dependent and a seed to bloat.
	   Even reentrancy might not be taken for granted.
	   Gcc uses two flavors of exception handling 'longjmp' and 'eh_frame' style
	   which are (to some extent) both supported by Cexp...

As is said: C++ IS THE DEVIL in the detail. While linking C (and even fortran etc.)
code compiled with different compilers (e.g. libraries) is seamless under a common
ABI, it is very complex if not impossible with C++.

Important PowerPC ABI Information
---------------------------------
Cexp currently does not support the short data areas according to the SYSV/EABI
specifications. Short data sections are merged by Cexp into the normal data
segment and hence cannot be accessed via R13/R2. Hence, when compiling loadable
modules, the compiler's -msdata flag must not be set to either of 'eabi', 'sysv'
or 'use'. If you try to load code compiled with an improper -msdata setting,
you will get a 'Relocation of unsupported size/type requested' error for the
relocation type 'R_PPC_EMB_SDA21' or 'R_PPC_SDAREL16' or similar.

LICENSING & DISCLAIMERS
-----------------------

CEXP is released under the terms of the EPICS open license (consult the LICENSE
file for details)

Copyright 2002, Stanford University and
		Till Straumann <strauman@slac.stanford.edu>

Stanford Notice
***************

Acknowledgement of sponsorship
* * * * * * * * * * * * * * * *
This software was produced by the Stanford Linear Accelerator Center,
Stanford University, under Contract DE-AC03-76SFO0515 with the Department
of Energy.

Government disclaimer of liability
- - - - - - - - - - - - - - - - -
Neither the United States nor the United States Department of Energy,
nor any of their employees, makes any warranty, express or implied,
or assumes any legal liability or responsibility for the accuracy,
completeness, or usefulness of any data, apparatus, product, or process
disclosed, or represents that its use would not infringe privately
owned rights.

Stanford disclaimer of liability
- - - - - - - - - - - - - - - - -
Stanford University makes no representations or warranties, express or
implied, nor assumes any liability for the use of this software.

This product is subject to the EPICS open license
- - - - - - - - - - - - - - - - - - - - - - - - - 
Consult the LICENSE file or http://www.aps.anl.gov/epics/license/open.php
for more information.

Maintenance of notice
- - - - - - - - - - -
In the interest of clarity regarding the origin and status of this
software, Stanford University requests that any recipient of it maintain
this notice affixed to any distribution by the recipient that contains a
copy or derivative of this software.
