


SPECPR Users' Manual                                                Page 1.2


_1._1 _I_n_t_r_o_d_u_c_t_i_o_n

     The SPECtrum Processing Routines (SPECPR) is a large scale  interactive
program  for  general  one  dimensional  array processing, and optimized for
reflectance spectroscopy data and  analysis.   The  program  processes  one-
dimensional  arrays  up to 4852 data points and the operations include addi-
tion, subtraction, multiplication, division, trigonometric functions,  loga-
rithmic and exponential functions, and many more specialized routines.

     The specpr user interface is a menu driven,  character  command  system
with  all  user input entered as ascii characters to the program.  User com-
mands are thoroughly checked for the context in which  they  will  be  used.
This  provides  for  essentially  all  user  input mistakes to be caught and
appropriate error messages to be issued.  The user interface also allows for
command  aliasing, variable substitutions, command history and batch command
processing to occur at any point in the program.  The user can also  control
his  or her own variables, increment and decrement them, and monitor certain
internal variables.  This flexibility allows for sophisticated  programming,
e.g. from specific computations, to management of a database.

     All arrays are treated as one dimensional lists of numbers,  each  with
an  appropriate  header  for  identification  and history.  Any array can be
plotted versus another array, and  each  axis  labeled  appropriately.   For
example, you may plot a reflectance spectrum versus wavelength or wavelength
versus reflectance.  The array type is simply a label, so data like tempera-
ture  versus  time  can be treated just as easily as a reflectance spectrum.
Some routines are specific to certain purposes, however.  For  example,  the
Planck  black  body  generator  is  specific to intensity versus wavelength,
whereas a smoothing routine could  operate  equally  on  reflectance  versus
wavelength or temperature versus time.

     Each array element can have an error bar associated with it and  errors
are  propagated through all appropriate routines.  Data points can be marked
deleted and deleted points are tracked appropriately.  These features  allow
the program to be effectively used on real world data.

     This philosophy has resulted in a flexible system for which a user  can
manipulate  data  arrays efficiently.  The program was designed for analysis
of laboratory, field, telescopic, and  spacecraft  spectroscopic  data,  and
although  general  in  nature, there are biases built in.  For example, hor-
izontal axis labels default to wavelength in micrometers, but  the  defaults
can be changed.

     Because of the orientation in specpr to process planetary  and  terres-
trial  data,  and  the need to access such data, specpr has been extended to
have access to non-specpr default file types.   In  particular,  specpr  can
"skewer"  an  3-dimensional  (3D)  data  file  along  any of the three axes.
Further, block skewers can be done with the standard deviation of  the  mean
computed for each channel in the block.  Currently, specpr can access any of
the standard file types common in the terrestrial and planetary remote sens-
ing communities.  Details on 3D file I/O are given in Chapter 6.

     Specpr has multiple record types available within a single specpr  data
file.   Currently defined record types are data and text.  In a data record,



                             February 20, 2002





SPECPR Users' Manual                                                Page 1.3


a standard one dimensional array is held, along with its header information.
The  header  information  includes  a title, history, dates and time of data
acquisition and when the data were last processed, the  user  who  processed
the  data,  information  typical  of a spectrum like temperature and viewing
geometry, as well as pointers to wavelengths, resolution, and  text.   If  a
spectrum  is  more  than  256  channels in length, then the data gets put in
succeeding records in the file, and the following records  are  continuation
records.

     The second record type is the text, where a title and a block  of  text
(up  to  19 kbytes) can be stored.  This record type is typically used for a
description of samples, experiments, instruments, or data processing  for  a
particular data set.  It could also store the actual commands used to create
a spectrum.  The commands for a complicated plot for a publication might  be
stored  in  a text record.  That way, you can easily regenerate the plot, or
even write the commands to a non-specpr file, modify them  with  any  editor
and reexecute them.

     The file types, text and command processing, combined with the math and
special  functions,  provide  for  very  powerful and general analysis tool.
These facilities also provide the framework for database management.


_1._2  _H_i_s_t_o_r_y _o_f _S_p_e_c_p_r

     The program began in 1975 at  the  M.I.T.  Wallace  Observatory,  on  a
Harris  2024  computer,  as  a short routine to subtract or divide 2 spectra
from the Remote Sensing Laboratory  Circular  Variable  filter  spectrometer
("The  Wedge").    Due  to  lack of processing software, the program grew to
fill the Wedge data reduction needs.  Over the years, the program has  grown
to  serve  more  and  more  applications.  The reason specpr has survived is
because of the philosophy behind each application routine that was added: it
should  be of a general nature, have checking of user input so that data are
not inadvertently destroyed, give a history of  all  operations,  and  track
deleted points because most data are not perfect.

     After the initial development of specpr at the MIT Remote  Sensing  Lab
(RSL),  the  lab  moved  to the Institute for Astronomy of the University of
Hawaii in June 1977.  At that time, there was no software for  reduction  of
spectral  data.  It was decided that the Wedge data reduction program should
be put on the new computer (TI 980 B) as a general spectrum and  one  dimen-
sional  array processing system.  In the next 3 years, the program was writ-
ten to handle the many types of astronomical photometry  being  obtained  by
the group at the Institute.

     In 1980, Specpr was moved from the overworked TI 980  B  to  the  newly
acquired LSI 11/23 system running the UNIX operating system.

     The historical development of specpr up to this point  solidified  some
of the commands used in the program.  The initial specpr routines (pre 1980)
were designed with data reduction strategy for a particular method of astro-
nomical  observing.   That strategy is still there, but must be modified for
other methods, for example reduction of laboratory spectroscopy  data.   The
TI 980 B and LSI 11/23 versions of specpr were based on 16 bit words whereas



                             February 20, 2002





SPECPR Users' Manual                                                Page 1.4


the MIT version was based on a 24 bit word machine,  and  that  resulted  in
certain  array  sizes  that are still in place.  During the change to the TI
980 B, many of the letter command codes and array sizes were changed to  fit
more  general  cases.   Also during the change to the LSI version all of the
routines were rewritten in Ratfor with a few in C and the program was broken
up  into  about  25  separate  programs  due to the lack of a decent overlay
linker for the Fortran compiler.  This  Ratfor  conversion  apparently  took
place because the programmer who did it was a C programmer, and liked Ratfor
better than FORTRAN 66 (this happened before FORTRAN 77 became widely avail-
able).

     In 1984, the program was moved  to  the  U.  S.  Geological  Survey  in
Denver.  The new spectrometers at the USGS necessitated many changes, mostly
because the USGS machines produced more data points per spectrum.   Research
directions also played a role in the analysis routines developed.  The study
of high resolution spectra, absorption  band  analysis,  radiative  transfer
mixing  models,  and  the  desire  to manage spectral databases necessitated
major changes to the functionality of specpr.

     The latest analysis area for specpr development is  access  to  imaging
spectrometer data cubes.  This access allows a user to query large data sets
for spectra and then analyze those spectra in detail.


_1._3  _S_p_e_c_p_r _D_e_s_i_g_n _P_h_i_l_o_s_o_p_h_y

     Specpr is designed to meet the needs of the remote sensing spectroscop-
ist  for  detailed  analysis  of  spectral  data.   In that sense, it is not
designed to analyze whole image cubes (hundreds of  thousands  of  spectra),
but  smaller  numbers (ones to several thousands) where the user must become
intimately knowledgeable about the data quality and results.

     One of the main goals of the program is to be uncrashable.  Since users
sit  at  the terminal for many hours processing data, they are bound to make
mistakes, so another goal is for the program to  be  somewhat  "intelligent"
and to try and catch mistakes.  The program is completely free format allow-
ing quick and easy input of  commands.   Spaces  are  required  between  two
numbers only when there is no character between them.  Otherwise, spaces can
be completely left out or inserted wherever the user wishes.

     Most commands are a single  character.   With  the  many  programs  and
options  in  specpr, an individual letter is used for more than one command.
For example in one section of the program, the command "l" may mean  list  a
file,  but  in  another,  say a plot routine, it may mean line type.  Thus a
command is context dependent.  The menus in each routine give  the  commands
and options available to the users.

     In many batch processing systems, the user types commands  in  a  file,
and  then  directs  the files into a program for processing.  If there was a
mistake, the job must be run again, and this wastes time.  In  the  interac-
tive  version,  as  the commands are typed in, they are executed immediately
and the results displayed.  In this way, all intermediate  steps  are  seen,
and  decisions  can  be made to change the processing in order to obtain the
best results.  Specpr  commands  can  be  read  from  a  file  starting  and



                             February 20, 2002





SPECPR Users' Manual                                                Page 1.5


terminating at any point in the program, thus giving batch capability.  Com-
mands can also be saved in a text file as they are typed  in.   Then,  if  a
mistake is made, the text file can be edited and executed again.

     Specpr maintains a complete history (within reason) of each  operation.
This  has  proved  invaluable  for  figuring  out what was done to some data
whether it is yours or someone else's.  Any time there is a question of  the
results of a particular analysis, anyone can trace and verify whether or not
everything was done correctly.  However, for specpr to properly record  his-
tories,  certain  steps  must be followed, for it is possible to subvert any
history mechanism.  One possible way to subvert the history is to put inter-
mediate products in a temporary file, then delete the file and only keep the
final product.  Specpr records the history of the  last  operation  of  each
spectrum,  so  to  trace  a complete history, all intermediate steps must be
kept.  This is not really a limitation, because spectral data are  small  in
comparison to the size of modern computer disk space.

     An interactive "smart" program of this nature requires an almost shock-
ing  amount  of  code.   Specpr is now about 50,000 lines of Ratfor and C in
length, contains over 400 subroutines, and  takes  about  1.5  megabytes  of
memory to run.

     For reference, specpr was first described by Clark (1980,  _P_u_b_l_i_c_a_t_i_o_n_s
_o_f _t_h_e _A_s_t_r_o_n_o_m_i_c_a_l _S_o_c_i_e_t_y _o_f _t_h_e _P_a_c_i_f_i_c, _9_2, 221-224).

































                             February 20, 2002


