Running ACIS Extract to build a final sourcelist for a target
Patrick Broos & Leisa Townsley

validation_procedure.txt $Revision: 1304 $ $Date: 2022-02-14 13:36:12 -0700 (Mon, 14 Feb 2022) $

Tested on CIAO 4.12 and CALDB 4.9.2.1.

THIS PROCEDURE MUST BE EXECUTED IN A csh OR tcsh SHELL!

THIS PROCEDURE PROBABLY REQUIRES THE "GNU" VERSION OF THE Unix "screen" utility (to accept the "^M" syntax we use for the newline character).

Use the local path to access your data.
On Hiawatha, 
  cd /Volumes/hiawatha1/targets/<target>/data/


=============================================================================
CONFIGURE YOUR COMPUTING ENVIRONMENT
=============================================================================
Once and forever, edit your .cshrc to add the following lines so that $PROMPT, if available, will set your unix prompt to show the ObsID number being processed in each 'screen' window:
  if ($?PROMPT == 1) then
    set prompt="$PROMPT"
  endif


*****
BEWARE OF OVERFLOWING IDL'S COMMAND LINE BUFFER!
If characters are sent to the IDL prompt too quickly then its command line buffer will overflow and drop some of those characters.  This problem can easily occur if you paste a large block of IDL commands into terminal window where IDL is running.

In this recipe, blocks of IDL commands are sent to IDL processes in two distinct ways:

1. When the same commands need to be sent to several IDL sessions (e.g. to perform a task for each ObsID), we use features of the "screen" utility to send those "paste" commands to several screen windows.  The "screen" session is configured to slow down those "paste" operations, to reduce the chance of buffer overflow.

2. When commands need to be sent to just one IDL session, this recipe directs you to directly "paste" those commands in the appropriate screen window, using your computer's normal cut-and-paste operations.  In this case, the best way to prevent overflow of the IDL command buffer is to configure your terminal application's "paste" command to slow down the rate that characters are delivered.  

For example the Mac terminal application called iTerm2 has a menu item called "Paste Slowly" for just this situation.  If you have iTerm2, then the most convenient approach is to navigate to iTerm2->Preferences->Keys->Global Shortcut Keys.  There, you can configure your normal "paste" key (e.g. CMD-V) to the menu item "Paste Slowly".

Other terminal applications may have similar options to slow down pasting.  If that is not available, you may find that you have to cut-and-paste IDL commands in smaller chunks.
****




A note about speeding up AE runs:
---------------------------------------------------------------------
1. AE will run faster and be more reliable if you maintain mirrors of CIAO and MARX on each machine you're using.   See Pat for a script to do this, and for the alterations needed in .cshrc to use those mirrors.

2. As described in the AE manual, configure .cshrc to skip slow-running commands in AE shells, e.g.

  if ($?FAST_START == 0) then
    source $HEADAS/headas-init.csh

    set headas=$HEADAS
    source $ciao_dir/bin/ciao.csh -o -q
    setenv HEADAS $headas
  endif



#################################################################################
#################### SECTION I: SET UP #################### 
#################################################################################
Make symlinks to access target-level data products, and symlinks to access various region files that AE will build.

In <target>/data/ directory:

  mkdir extract

  ln -s point_sources.noindex/all.reg point_sources.noindex/xray_positions.reg point_sources.noindex/polygons.reg point_sources.noindex/review.reg point_sources.noindex/tables/xray_properties.fits extract

  ln -s extract/point_sources.noindex/all.reg extract/point_sources.noindex/xray_positions.reg extract/point_sources.noindex/polygons.reg extract/point_sources.noindex/review.reg extract/point_sources.noindex/tables/xray_properties.fits .

  ln -s ../all.reg ../xray_positions.reg ../polygons.reg ../review.reg ../xray_properties.fits counterparts

  foreach dir (counterparts extract)
    pushd $dir
    ln -s ../fullfield.target.evt       target.evt
    ln -s ../fullfield.target.soft.evt  target.soft.evt
    ln -s ../fullfield.target.hard.evt  target.hard.evt
    ln -s ../fullfield.target.vhard.evt target.vhard.evt
    ln -s ../fullfield.target.emap      target.emap
    popd
  end



Start a new screen session. 

  cd extract
  setenv PROMPT "%m (%c1): "
  setenv SCREEN_ARCH ''
  if (`uname -m` == arm64)  setenv SCREEN_ARCH 'arch -arm64 tcsh'

  setenv SCREEN_NAME "`basename $PWD`"
  screen -S $SCREEN_NAME -t "AE setup"  $SCREEN_ARCH
  screen -S $SCREEN_NAME -X logfile ae_setup.screen.txt
  screen -S $SCREEN_NAME -X log on
  xclock &

This will cause screen to log its output to a file called ae_setup.screen.txt.  
xclock is launched to verify that X11 is working; kill it once it comes up.



GATHER ObsID DATA PRODUCTS NEEDED BY AE
=======================================

  idl
    .run ae
    run_command, file_which('setup_AE_for_obsid_data.csh')+' "../pointing_*/obsid_*" AE'
    exit, /NO_CONFIRM
  'ls' -l obs[0-9]*

The script above makes a new subdirectory (obsXXXX) in the extract directory for each ObsID, and fills it with a bunch of symlinks back to the data products that AE will need.  Those single-ObsID data products were produced by the L1->L2 workflow.  The event lists and emaps are expected to lie in directory paths of the form ../pointing_*/obsid_*/AE/.  If the data products you used for source searching live in a DIFFERENT subdirectory, then you must alter the parameter "AE" in the setup_AE_for_obsid_data.csh call appropriately.

** 
The key issue here is that the field-of-view of the data products AE sees MUST MATCH the field-of-view on which you performed source detection.  Unless AE knows about all the point sources in the data it's been given, the accuracy of AE background spectra can be compromised, because those unreported point sources may contaminate the background spectra of nearby sources that AE does know about.
**

Front-illuminated and back-illuminated CCDs have to be treated as separate virtual ObsIDs, because those two device designs have very different instrumental backgrounds.




CREATE DIRECTORIES WE WILL NEED FOR POINT SOURCES and (later) diffuse analysis
=========================================================================

  if (! -e diffuse_sources.noindex                   ) mkdir diffuse_sources.noindex
  if (! -e diffuse_sources.noindex/extra_maskfile.reg) touch diffuse_sources.noindex/extra_maskfile.reg 
  ~/bin/acis_lkt4 diffuse_sources.noindex/ diffuse_sources.noindex/extra_maskfile.reg

  mkdir point_sources.noindex
  cd    point_sources.noindex
  ~/bin/acis_lkt4 .
  mkdir tables  ../checkpoint_for_backup

  wget http://personal.psu.edu/psb6/TARA/procedures/generic.rmf


Begin recording notes about your data processing in a file named validation_notes.txt, so that when your work is interrupted you can come back and understand where you left off.  The following call creates new notes files; instead, I usually copy them from the last target I analyzed and edit the notes as needed.  This promotes consistency between targets.

  touch validation_notes.txt photometry_notes.txt
  /bin/ls $PWD/validation_notes.txt

>>>>>>>>>>>>
Record in your notes the version of this procedure that you are executing, shown below:

Starting candidate validation using validation_procedure.txt:
  $Revision: 1304 $ $Date: 2022-02-14 13:36:12 -0700 (Mon, 14 Feb 2022) $
>>>>>>>>>>>>

The source extraction we will be doing in this validation recipe requires source RMF files, only to map energies to PI channels. Since the actual response in the RMF is not used, and since building individual RMF files takes a lot of CPU time, we instead supply AE with a single "generic" RMF (generic.rmf above, obtained from an ACIS source in another target).



BUILD AN OBSERVING LOG and AIMPOINT REGION FILE
=======================================================================
The preliminary observing log we build below can serve as a reference when you need to look up the date or exposure time of an ObsID.

  pushd tables
  idl
    .run ae
    .run
    hmsfr_tables, '', file_which('/hmsfr_tables.tex')
    exit, /NO_CONFIRM
    end

  lualatex  xray_properties; lualatex  xray_properties 
  lualatex -jobname=target_observing_log "\includeonly{./observing_log}\input{xray_properties}"  
  lualatex -jobname=target_observing_log "\includeonly{./observing_log}\input{xray_properties}"  
  open target_observing_log.pdf
  popd

NOTE -- the "lualatex" commands above do not work on Ouray, Osceola, or Sequoia!  Run from a hiawatha or cochise iterm, cd'd to /tables .  Don't be surprised if $DISPLAY is messed up so "open" sends the PDF file back to Hiawatha or Cochise.




BUILD AN AE CATALOG FROM THE CELESTIAL COORDINATES OF YOUR PROPOSED SOURCES
===========================================================================
The starting point for AE is a list of candidate source positions from the standard reduction procedure:
  proposed.target.fits (a match_xy catalog with RA and DEC columns, ready for ACIS Extract)
  proposed.target.reg  (a ds9 region file, not used here).

The 'c' label in the catalog stands for "candidate".

  idl -queue |& tee -a get_sourcelist.log
    .run ae
    .run
    target_cat = mrdfits('../../recon_detect/proposed.target.fits', 1, /SILENT)

    forprint, target_cat.LABEL, target_cat.POSNTYPE, target_cat.PROVENAN,  F='(%"%10s  %20s  %s")'

    ae_source_manager, /ADD, RA=target_cat.ra, DEC=target_cat.dec, LABEL=target_cat.LABEL, PROVENANCE=target_cat.PROVENAN, POSITION_TYPE=target_cat.POSNTYPE
    exit, /NO_CONFIRM
    end


If you have previously identified "by-eye" candidates that were missed by the detection procedure (i.e. that are missing from proposed.target.fits above), then you may add them to your AE workflow now, using additional calls to ae_source_manager.  

For a small number of "by-eye" candidates, it's convenient to call ae_source_manager for each, explicitly specifying the coordinates in the call.  For example,
  idl -queue |& tee -a get_sourcelist.log
    .run ae
    .run
    ae_source_manager, /ADD,  RA=xx.xxxxxD, DEC=xx.xxxxxD, LABEL='pass1_a1', PROVENANCE='eye', POSITION_TYPE='eye'
    ae_source_manager, /ADD,  RA=xx.xxxxxD, DEC=xx.xxxxxD, LABEL='pass1_a2', PROVENANCE='eye', POSITION_TYPE='eye'
    ...
    exit, /NO_CONFIRM
    end

  * Replace xx.xxxxx with the decimal RA and DEC coordinates for each source.  BE SURE TO PUT A 'D' AFTER EACH DECIMAL COORDINATE, SO THAT IDL INTERPRETS THE NUMBER AS DOUBLE-PRECISION.

  * Be sure to invent a unique source label for each source -- here I use "a" for "added."


For a large number of "by-eye" candidates, it's convenient to read the decimal RA and DEC coordinates from an ASCII file, and use code to generate source labels.  For example,
  idl -queue |& tee -a get_sourcelist.log
    .run ae
    .run
    readcol, COUNT=num_sources, 'by-eye.txt', ra, dec, FORMAT='D,D', COMMENT=';'    ; read coordinates as DOUBLE precision
    if (num_sources GT 0) then begin
      label = string(1+indgen(num_sources), F='(%"pass1_a%d")')
  
      help, ra, dec, label
  
      ae_source_manager, /ADD,  RA=ra, DEC=dec, LABEL=label, PROVENANCE='eye', POSITION_TYPE='eye'
    endif
    exit, /NO_CONFIRM
    end





Terminate the screen session by typing "^a\".




CREATE A LIST OF ObsIDS FOR THE TARGET, AND A SHELL FOR EACH ObsID
==================================================================
Create a new 'screen' session named after your target (***replace <target> below with your target name***), and create a 'screen' window for each ObsID_device set (usually there's an ObsID_FI and an ObsID_BI):

  cd <target>/data/extract/point_sources.noindex/
  setenv TARGET <target>

  'rm' CURRENT_PASS; ln -s pass1 CURRENT_PASS
  setenv           PASS `readlink CURRENT_PASS`
  mkdir           $PASS
  cp all.srclist  $PASS

  mkdir IDL
  
cat > IDL/exit.pro << \STOP
    PRO ae_code_block
    print, 'Exiting IDL at ' + now() 
    exit, /NO_CONFIRM 
    end
\STOP
  
  
  setenv OBS_LIST ""
  foreach dir (`'ls' -d ../obs[0-9]* | sort --field-separator=_ --key=2r --key=1.7g`)
    if (! -d $dir) continue
    set obs=`basename $dir | sed -e "s/obs//"`
    setenv OBS_LIST "$OBS_LIST $obs"
    end
  echo $OBS_LIST  

  setenv SCREEN_ARCH ''
  if (`uname -m` == arm64)  setenv SCREEN_ARCH 'arch -arm64 tcsh'

  setenv SCREEN_NAME      AE_point_${TARGET}
  setenv PROMPT              '%m::%${TARGET} (%$PASS) %% '
  screen -S $SCREEN_NAME -d -m -t "${TARGET}"  $SCREEN_ARCH 
  screen -S $SCREEN_NAME -X defslowpaste 10

  screen -S $SCREEN_NAME -X setenv PROMPT "%m::${TARGET} (window1) %% "
  screen -S $SCREEN_NAME -X screen -t window1  $SCREEN_ARCH
  screen -S $SCREEN_NAME -X setenv PROMPT "%m::${TARGET} (window2) %% "
  screen -S $SCREEN_NAME -X screen -t window2  $SCREEN_ARCH
  screen -S $SCREEN_NAME -X setenv PROMPT "%m::${TARGET} (window3) %% "
  screen -S $SCREEN_NAME -X screen -t window3  $SCREEN_ARCH
  sleep 1
  foreach obs ($OBS_LIST)
    screen -S $SCREEN_NAME -X setenv OBS     ${obs}
    screen -S $SCREEN_NAME -X setenv PROMPT "${obs} %% "
    screen -S $SCREEN_NAME -X screen -t   obs${obs}  $SCREEN_ARCH
    sleep 0.5
  end
  screen -S $SCREEN_NAME -r
  sleep 4
  screen -S $SCREEN_NAME -X select 0
  screen -S $SCREEN_NAME -X at \# slowpaste 10
  screen -S $SCREEN_NAME -X at \# stuff 'unset correct^M'
  screen -S $SCREEN_NAME -p 0 -X stuff 'echo $OBS_LIST | wc -w^M'
  screen -S $SCREEN_NAME -X windowlist -b


Double-check that you have a screen window for every ObsID.  There should also be screen windows named 'window1', 'window2', 'window3', plus the usual top-level $TARGET window #0.  Use ^a" to see the list of screen windows.  The last window index should be THREE LARGER than the number of ObsIDs (which was printed by the "echo" command above).  

If you are missing some screen windows, then read the appendix titled THE "SCREEN" UTILITY in this document.



"screen" help:
  ^a"  select from list of windows using arrow keys
  ^an  next window
  ^ap  previous window
  ^ak  kill the current window
  ^aw  list the existing windows
  ^a^c create a new window and switch to it
  ^aA  name the current window
  ^a?  help

  screen -S $SCREEN_NAME -r    attach to existing screen windows
  screen -ls                   find all existing screen sessions
  ^ad                          detach from "screen" session

  When your data processing is completely finished, you can destroy each "screen" window by exiting its shell in the normal way, e.g. via "exit" or "^d".  All screen windows can be destroyed with "^a\".


When you have a large number of screen windows (ObsIDs), mistakes/problems can leave you needing to terminate IDL sessions in a large number of windows.  The quick way to do that is to paste an IDL "exit" command in all the windows at once, by running the following from window #0 (named $TARGET):

  screen -S $SCREEN_NAME -X at \# stuff 'exit,/NO_CONFIRM^M'



A note about running on multiple computers:
---------------------------------------------------------------------

Most tasks in AE will generally run fastest on the computer that is hosting the disk on which the data are stored.  However, if you have more ObsIDs than the number of "cores" on that computer, and if your data are visible to other computers (via a networked filesystem), then you will want to spread your ObsID-based processing across multiple computers.  

In window #0 run the following script to print the commands needed to ssh to each machine you want to use:


  foreach machine (hiawatha_data ouray_data osceola_data tecumseh_data sequoia_data ouachita_data cochise_data)
    printf '\n\nTo set up %s run:\n\n' $machine

    printf '  setenv WD_DESIRED "`echo $PWD|sed -e %s`"\n\n'  "'s|/Volumes/|/bulk/|'" 

    printf '  ssh -t %s env OBS_LIST=\\"$OBS_LIST\\" OBS=\\"$OBS\\" PROMPT=\\"%s $PROMPT\\" WD_DESIRED=\\"$WD_DESIRED\\"  caffeinate -s %s \n\n' $machine $machine  $SHELL

    printf '  cd "$WD_DESIRED" ; pwd\n\n'
  end


Then, in the screen window for each ObsID that you want to be processed on a remote machine, paste the set of commands printed by the script above for the appropriate remote machine.  

*****
If the pasted "cd" command fails with "Permission denied", then on the machine hosting the data you need to run:
  sudo nfsd restart

If you need to conserve IDL licenses, then the foreach loop needs to identify a single DISPLAY value for each machine
    set fixed_display=`ssh $machine 'echo $DISPLAY'`
and needs to setenv DISPLAY in each remote shell
    env .... DISPLAY=%s .... $fixed_display .... $SHELL
*****



NOTE -- The best procedure to terminate the screen session is:

1. In screen window #0 run the following to paste the 'exit' command in every window, which will cleanly terminate ssh connections you have to other computers (avoiding zombie processes):.
  screen -S $SCREEN_NAME -X at \# stuff 'exit^M'

2. If that doesn't end the screen session, then type the screen command "^a\".


If you forgot to "exit" the ssh connections as instructed above, and instead simply killed the screen session with ^a\, the remote shells (which are running "caffeinate" to keep the machine from sleeping) will NOT DIE.  In that case then, the easiest way to kill them (and recover from your mistake) is to ssh to the machine again and run
  killall caffeinate


To put a remote machine to sleep when you're completely done with your AE analysis and don't need its processors anymore, execute
  sudo pmset sleepnow; exit


{For reference, in the ssh commands used above explicitly setting the DISPLAY environment variable is a trick to force all the ssh sessions to use the same value for $DISPLAY, which allows multiple IDL sessions on the remote machine to consume only one IDL license.  Note that 'X11 forwarding' must be configured in your ssh preferences.  Note that using absolute machine names in DISPLAY, e.g. 'cochise:0', disables the security protections afforded by the X11 forwarding mechanism in ssh and is not recommended.}




#################################################################################
#################### THINK CAREFULLY ABOUT YOUR BACKUP SYSTEMS #################### 
#################################################################################
This validation recipe will re-extract your catalog numerous times, and none of those extractions are of scientific interest.  If you allow your backup systems (e.g. TimeMachine, CrashPlan, rsync, etc.) to see this AE workspace, then it will waste a great deal of resources (I/O bandwith and CPU time) making numerous backups of this rapidly-changing directory tree.

We recommend excluding the point_sources.noindex/ directory from your backup process during the execution of this procedure.  If point_sources.noindex/ is excluded and your disk fails, the essential data product produced by this procedure (the source catalog) will be preserved, since this procedure frequently copies it to extract/checkpoint_for_backup/.  

CrashPlan is a terrible memory hog, even when it has nothing to do!   We strongly recommend killing CrashPlan while doing AE runs, with:

  sudo launchctl unload /Library/LaunchDaemons/com.crashplan.engine.plist

To restart, use
  sudo launchctl load /Library/LaunchDaemons/com.crashplan.engine.plist



#################################################################################
############# SECTION II: REMOVE DUPLICATE SOURCE CANDIDATES ##################
#################################################################################

Source detection often produces candidates that are too closely-spaced to be extracted.  In this section we identify and remove such "duplicate" sources.

This requires extracting the catalog in each ObsID (omitting background calculations for now), attempting to merge those extractions, determining which sources are unacceptably crowded in every ObsID, and then deciding which of those to prune.

Multiple cycles of this processing are executed automatically, without human review of the pruning decisions.
This takes many hours (many days for large sourcelists).

=================================================================================================
First we launch IDL sessions to extract the catalog from each ObsID. 
From screen window #0 (named $TARGET), run:

cat > IDL/extract.pro << \STOP
    PRO ae_code_block
    ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
    pass_name = file_readlink('CURRENT_PASS')
    print, pass_name, systime(0), F='(%"\n\n=====================================================================\nSTARTING %s @%s\n")'
    extract_and_pileup_block1, ae_get_target_parameters(), EXTRACT_BACKGROUNDS=0
    obsdir = "../obs"+getenv("OBS")+"/"
    file_move, obsdir+"ae_lock", obsdir+"ae_finished", /OVERWRITE 
    print, pass_name, F='(%"\n\n=====================================================================\nFINISHED %s\n")'
    return
    end 
\STOP

cat > block.txt << \STOP
  
  nice idl -queue |& tee -a ae_validation.${OBS}.log 
    .run ae 
    file_delete,    "waiting_for_"+getenv("OBS"), /ALLOW_NONEXISTENT
    ae_processing_server, "../obs"+getenv("OBS")+"/ae_code_block.pro"
\STOP
  screen -S $SCREEN_NAME                -X readreg A block.txt

  foreach obs ($OBS_LIST)
    set file=../obs${obs}/ae_code_block.pro
    if (-e $file) 'rm' $file
    set marker=waiting_for_${obs}; touch ${marker}
    screen -S $SCREEN_NAME -p obs${obs} -X   paste A
    sleep 1
    while ( -e ${marker} )
      echo "Launching IDL for ObsID ${obs}..."
      sleep 1
    end
    sleep 2
  end

***DON'T STOP HERE -- KEEP READING...***


=================================================================================================
Launch an IDL session that will:
  * manage the ObsID extractions
  * merge the extractions 
  * prune sources based on overlap, and prune source not extracted from any ObsID

When the prompt returns, from screen window #0 (named $TARGET), run:

  touch SKIP_RESIDUALS REUSE_NEIGHBORHOOD VALIDATION_MODE
  'rm'  DISCARD_PSF      >& /dev/null
  'rm'  CHECK_FOR_PILEUP >& /dev/null 
  'rm'  ABORT            >& /dev/null

  nice idl -queue |& tee -a merge_and_prune_overlap.log
    .run ae
    .run
    cd, CURRENT=cwd
    obsname = strtrim(strsplit(getenv('OBS_LIST'), /EXTRACT, COUNT=num_obs), 2)
    obsdir  = '../obs'+obsname+'/'
    if ~array_equal( file_test(obsdir), 1) then message, 'ERROR: some obsXXXX dirs do not exist.'

    ; Look up the number of events in each ObsID.
    obsid_event_tally = lonarr(num_obs)
    for ii=0,num_obs-1 do obsid_event_tally[ii] =  psb_xpar( headfits(obsdir[ii]+'spectral.evt', EXT=1), 'NAXIS2')

    obsid_finished_flags = obsdir + 'ae_finished'
    obsid_IDL_files      = obsdir + 'ae_code_block.pro'

    ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
    pass_name = file_readlink('CURRENT_PASS')
    is_first_iteration = 1B

    ; Manage pruning passes until stopping criterion met.
    repeat begin
      if ~is_first_iteration then begin
        ; Increment PASS name
        ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
        lexeme = stregex(pass_name,'([^0-9]*)([0-9]+)',/SUB,/EXT)
        pass_name = strjoin( [lexeme[1], strtrim(lexeme[2]+1,2)] )
        file_delete,          'CURRENT_PASS'
        file_link, pass_name, 'CURRENT_PASS'
      endif

      print, pass_name, systime(0), F='(%"\n\n=====================================================================\nSTARTING %s @%s\n")'
      tic

      ; Verify that no ObsID processing subroutine files remain.
      if ~array_equal( file_test(obsid_IDL_files), 0) then begin
        print, 'ERROR: the following files should not exist yet:'
        forprint, obsid_IDL_files, SUBSET=where(file_test(obsid_IDL_files))
        retall
      endif

      ; Remove the flag files that signal completion of ObsID extractions.
      file_delete, /ALLOW_NONEXISTENT, obsid_finished_flags
  
      ; LAUNCH EXTRACTIONS (largest ObsIDs first)
      foreach ii, reverse(sort(obsid_event_tally)) do begin
        file_copy, /OVERWRITE, 'IDL/extract.pro', obsid_IDL_files[ii]
        wait, 5
      endforeach

      ; Wait for extractions to finish.
      wait_until_files_exist, 60, obsid_finished_flags, /DELETE

      overlap_limit = 0.10

      acis_extract, 'all.srclist', MERGE_NAME='position', /MERGE_OBSERVATIONS, /MERGE_FOR_POSITION, MIN_NUM_CTS=3, OVERLAP_LIMIT=overlap_limit, EBAND_LO=0.5, EBAND_HI=7.0, ENERGY_RANGE=[0.5,7.0], /SKIP_PSF, /SKIP_NEIGHBORHOOD, /SKIP_APERTURE, /SKIP_TIMING, GENERIC_RMF_FN='generic.rmf'

      acis_extract, 'all.srclist', MERGE_NAME='position', COLLATED_FILENAME='tables/position.collated', LABEL_FILENAME='label.txt'

      ae_analyze_pb, /ANALYZE     , /IGNORE_PB, 'tables/position.collated', OVERLAP_LIMIT=overlap_limit, GENERIC_RMF_FN='generic.rmf'
      ae_analyze_pb, /MAKE_REGIONS, /IGNORE_PB, DS9_TITLE=getenv('TARGET')+', '+pass_name

      fn = 'prune.srclist'
      if logical_true( file_lines(fn) ) then begin
        readcol, COUNT=num_pruned, fn, sourcename, FORMAT='A', COMMENT=';'
        if (num_pruned GT 0) then begin
          ae_source_manager, /REMOVE, NAME=sourcename, TRASH_DIR='/tmp/$TARGET/'+pass_name+'/invalid_sources/'
          ae_source_manager, ORPHAN_SOURCES=os
        endif
      endif else num_pruned = 0

      ; Stopping criterion:   <1% of sources are pruned
      done = (num_pruned LE file_lines('label.txt')/100.) || file_test('ABORT')

      ; Rename some files to document this pass.
      wait, 10   ; ds9 may still be loading Pbslice.reg

      foreach fn, ['label.txt', 'prune.srclist', 'Pbslice.reg'] do begin
        if ~file_test(fn) then continue
        fdecomp, fn, disk, item_path, item_name, item_qual
        file_move, /VERBOSE, fn, item_path+item_name+'.'+pass_name+'.'+item_qual
      endforeach

      is_first_iteration = 0B

      print, pass_name, round(toc()/60), num_pruned, F='(%"\n\n=====================================================================\nFINISHED %s (%d minutes elapsed, %d pruned)\n")'
    endrep until done
    file_delete, /ALLOW_NONEXISTENT, 'ABORT'

    ; Terminate the ObsID IDL sessions, so we can archive their log files.
    print, now(),F='(%"\n\n=====================================================================\n(%s) TERMINATING ObsID IDL sessions.\n")'
    file_copy, /OVERWRITE, replicate('IDL/exit.pro',num_obs), obsid_IDL_files
    exit, /NO_CONFIRM
    end


To check the progress of the passes, run:
  egrep 'STARTING|minutes elapsed' merge_and_prune_overlap.log | egrep -v 'print,'

To make sure that the IDL processes are still running, go to the screen named "window1" and run:
  while 1
    printf "\n" 
    date "+%B-%d %H:%M"
    ls -ltr ae_validation*.log
    sleep 300
  end

If you wish to interrupt the automated passes at the end of the current pass, then:
  touch ABORT



---------------------------------------------------------------------
Review warning/error messages in log files.

  find [0-9]* -maxdepth 1 -type d \( -name position -or -name trash \) -print0 -prune | xargs -0 -n1000 rm -r

  egrep -i "WARNING|ERROR|DEBUG|halted|Stop"         ae_validation.*.log | egrep -v "DISPLAY variable|LS_COLORS|Program caused arithmetic error|error=|ARF was computed to be zero|has no rows|cannot intersect|no in-band|not observed|segments with ends|brute force"

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop|IMMORTAL" merge*.log  | egrep -v "DISPLAY variable|LS_COLORS|Program caused arithmetic error|cannot intersect|is missing|accepting only|Trying again|No HDUNAME|Skipping source|OBJECT keyword|DETNAM has different|segments with ends|no in-band|excessive OVERLAP|PSF2CAT|ERR_DATA|ERR_RA|Soft|Hard|array_equal|topping criterion|print,"

  printf "\n\n"
  egrep 'minutes elapsed' merge_and_prune_overlap.log | egrep -v 'print,'

Investigate any messages you don't understand.

=================================================================================================
HAND-DELETE SOURCE CANDIDATES IF NEEDED
=================================================================================================
Before re-extracting and calculating backgrounds (twice), you may want to remove more sources than just decrowding achieved.  Here's a way to do that.  In point_sources.noindex/ in screen window0, 

Collect polygons from whatever AE region file you have, e.g.
  echo "global select=0"          > /tmp/polygon.reg
  grep polygon Pbslice.pass1.reg >> /tmp/polygon.reg


First we need to build a collation of the current AE catalog and a match_xy region file to review in ds9 (hand_prune.reg).  From screen window #0 (named $TARGET), run:

  idl
    .run /bulk/cochise1/psb6/TARA/code/utilities/match_xy.pro
    .run ae
    .run
    AE_catalog_filename = '/tmp/all.collated'
    acis_extract, 'all.srclist', COLLATED_FILENAME= AE_catalog_filename
    
    event2wcs_astr   = get_astrometry_from_eventlist('../../tangentplane_reference.evt')
    acis_cat = build_AE_cat(AE_catalog_filename, event2wcs_astr)
    catalog_ds9_interface, acis_cat, /WRITE_REGFILE, ASTROMETRY=event2wcs_astr, 'hand_prune.reg'
    end

This will produce a flurry of warning messages -- ignore those.
If you get the error message
  build_AE_cat: ERROR!  Some position uncertainties (X_ERR, Y_ERR) are not positive. 
type .continue

LEAVE IDL RUNNING.


In screen window1 or any other shell, display the match_xy regions and the AE polygons.
  ds9 -title 'hand pruning interface' ../target.evt -region /tmp/polygon.reg -region hand_prune.reg &

REMOVE "cross" regions (tagged as "ACIS") for sources you want to prune.
For convenience you can move those regions without affecting the AE source position.
The polygons are irrelevant.
Re-save edited regions to hand_prune.reg (in any coordinate system, with or without polygons).


In a shell, discard the polygons:

  grep point hand_prune.reg > /tmp/temp.reg

In the existing IDL session, 

  .run
  catalog_ds9_interface, acis_cat, '/tmp/temp.reg' , /PRUNE_CATALOG, PRUNE_LABELS=prune_labels 
  ae_source_manager, /REMOVE, LABEL=prune_labels,TRASH_DIR='/tmp/$TARGET/$PASS/invalid_sources/'
  end
  
  exit

This finds the labels for the sources missing from hand_prune.reg, records the labels we are pruning (to hand_prune.txt), and prunes the AE catalog.


=================================================================================================
ARCHIVE THE PASSES RUN ABOVE
=================================================================================================

Do this even if you have converged (have no sources to prune)!

**FROM SCREEN WINDOW #0 (named $TARGET)**, run:

  setenv PASS `readlink CURRENT_PASS`
  mkdir $PASS
  rsync -a ./{*.txt} tables/*.collated removed_sources.* ../checkpoint_for_backup/
  (mv by-eye.txt add.txt move.txt delete.txt              $PASS |& cat > /dev/null )
  rsync -a --info=NAME ./{*.txt} --exclude=block.txt --exclude='label.*' $PASS  
  rsync -a label.${PASS}.txt label.txt
  gzip *.log
  mv `find *.srclist \! -name 'all*.srclist'` label.* *.log *.log.gz *.reg *.sav *.ps screenrc.* *.collated tables/*.collated  $PASS  

  set filelist=`find $PASS -name label.txt -o -name all.reg -o -name polygons.reg -o -name xray_positions.reg -o -name psf_hook.reg -o -name removed_sources.reg -o -name fov_ptsrcs.reg -o -name modified_aperture.srclist`
  if ( "$filelist" != '' ) rsync -a --info=NAME $filelist  .

  chmod -R a-w      $PASS
  setenv PASS `echo $PASS | awk '{sub("pass",""); printf "pass%d", $0+1}'` ; printf "\n\nSTARTING $PASS\n\n" 
  'rm' CURRENT_PASS; ln -s $PASS CURRENT_PASS
  mkdir                    $PASS
  rsync -a all.srclist     $PASS
  touch SKIP_RESIDUALS REUSE_NEIGHBORHOOD VALIDATION_MODE



(Pat and Leisa only) Copy vital files to archive disk so they will be backed up.
-------------------------------------------------------
IF you're working on the computer where the copy will be stored, run:
  idl
    relocate_target, /CHECKIN, /BACKUP, getenv("TARGET"), ['data/extract/checkpoint_for_backup/','data/extract/point_sources.noindex/pass*'], ARCHIVE_VOLUME='hiawatha2', FAST_VOLUME='osceola_fast'

  exit   

IF INSTEAD you're working on a remote computer but the copy will be stored on a different computer, run something like the following FROM THE DESTINATION COMPUTER (fill in appropriate volumes; it doesn't matter where you're cd'd on the destination computer):
  idl
    relocate_target, /CHECKIN, /BACKUP, getenv("TARGET"), ['data/extract/checkpoint_for_backup/','data/extract/point_sources.noindex/pass*'], ARCHIVE_VOLUME='hiawatha2', FAST_VOLUME='ouray_fast'
    
    exit 





############# NOTES ################
In the section above no background spectra were built, because we were pruning based only on aperture overlap.  In the next section every pass will build background spectra, compute Pb, and use Pb in pruning decisions.

Alas, the FIRST set of background spectra we build is going to be inaccurate, because the BETTER_BACKGROUNDS algorithm is an iterative process, for two reasons:

  1. The algorithm builds a spatial model for each source, which requires photometry for each source.  Accurate photometry requires background subtraction, which requires that we already have background estimates.
  
  2. As described in the "Adjust BACKSCAL Stage" section of the AE manual, it is important that all extractions of a source use background regions whose scalings (BACKSCAL keyword) are similar.  For each source, we must iteratively *search* for a range of allowed scalings that is appropriate.

Thus, we really should not rely on Pb to make pruning decisions until we have performed several cycles of extraction and background estimation.  If processing time was unimportant, we would have computed backgrounds in all the extractions above to help them stabilize.

Instead, we strike a compromise between perfection and progress by running TWO CYCLES of background estimation in the first Pb pruning pass.  (That is done in the procedure extract_and_pileup_block1, in Section IV below.)  

This background generation used to be SECTION III of the validation procedure; it has been deprecated in this automated version.




#################################################################################
############# SECTION IV: PRUNE INSIGNIFICANT SOURCE CANDIDATES ##################
#################################################################################

#################################################################################
(optional) Include VeryHard Band in Source Validation

If you wish the significance of source candidates to be evaluated in the "very hard" (4-7 keV) energy band, as well as the three normal bands, then create/edit the ASCII file named target_parameters.par and insert the following line:

  PB_IN_VHARD_BAND     = 1  ; include "very hard" band in Pb calculations

#################################################################################

#################################################################################
(optional) Change P_B Validity Threshold

For targets with bright, highly structured diffuse emission, we have decided (6 Dec 2019) to lower the P_B validity threshold from 1e-2 to 1e-3.  (So far, this applies only to 30 Dor and Arches+Quintuplet.)  This seems to be the fairest way to minimize over-reconstructions of that diffuse emission from polluting the point source catalog.  To do this, create an ASCII file named target_parameters.par (if it doesn't already exist) and insert the following line:

INVALID_THRESHOLD = 0.001 ; source is valid if Pb < 0.001

#################################################################################


=================================================================================================
IDENTIFY INSIGNIFICANT SOURCE CANDIDATES
=================================================================================================


=================================================================================================
First we launch IDL sessions to extract the catalog from each ObsID. 
From screen window #0 (named $TARGET), run:

cat > IDL/extract.pro << \STOP
    PRO ae_code_block
    ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
    pass_name = file_readlink('CURRENT_PASS')
    print, pass_name, systime(0), F='(%"\n\n=====================================================================\nSTARTING %s @%s\n")'
    par = ae_get_target_parameters()
    obsdir = "../obs"+getenv("OBS")+"/"
    if keyword_set(par.BACKGROUND_MODEL_FILENAME) then $
      par.BACKGROUND_MODEL_FILENAME = obsdir + par.BACKGROUND_MODEL_FILENAME
    extract_and_pileup_block1, par, EXTRACT_BACKGROUNDS=1, BACKGROUND_MODEL_FILENAME=par.BACKGROUND_MODEL_FILENAME
    file_move, obsdir+"ae_lock", obsdir+"ae_finished", /OVERWRITE 
    print, pass_name, F='(%"\n\n=====================================================================\nFINISHED %s\n")'
    return
    end 
\STOP

cat > block.txt << \STOP
  
  nice idl -queue |& tee -a ae_validation.${OBS}.log 
    .run ae 
    file_delete,    "waiting_for_"+getenv("OBS"), /ALLOW_NONEXISTENT
    ae_processing_server, "../obs"+getenv("OBS")+"/ae_code_block.pro"
\STOP
  screen -S $SCREEN_NAME                -X readreg A block.txt

  find [0-9]* -maxdepth 1 -type d \( -name theta_00-03 -or -name theta_02-06 -or -name theta_05-09 -or -name theta_08-14 -or -name theta_13-99 -or -name most_valid -or -name trash \) -print0 -prune  | xargs -0 -n1000 rm -r 

  foreach obs ($OBS_LIST)
    set file=../obs${obs}/ae_code_block.pro
    if (-e $file) 'rm' $file
    set marker=waiting_for_${obs}; touch ${marker}
    screen -S $SCREEN_NAME -p obs${obs} -X   paste A
    while ( -e ${marker} )
      echo "Launching IDL for ObsID ${obs}..."
      sleep 1.0
    end
  end
  
***DON'T STOP HERE -- KEEP READING...***


=================================================================================================
DECIDE WHETHER TO "SHAKE THE TREE"

If you wish to recalculate PSFs for every source, then:

  touch DISCARD_PSF

Rebuilding PSF images tends to cause some marginal sources to fail validation.
We assume this is because the apertures often move a little bit when PSFs are rebuilt.
This is an expensive way to prune sources, and we would like to get rid of this practice, so this step is NOT RECOMMENDED at this point.

Currently, the source repositioning section of this recipe optionally touches the file DISCARD_PSF, which causes all PSFs to be rebuilt during the first extraction after repositioning.


=================================================================================================
DECIDE WHETHER TO REVIEW CATALOG PRUNING

If you wish to review the sources nominated for pruning in each pass, then:

  touch REVIEW_PRUNING

If you get tired of this, just:

  rm REVIEW_PRUNING from any window in the /point_sources.noindex directory.


=================================================================================================
DECIDE WHETHER CORRECTIONS FOR PILE-UP SHOULD BE DONE NOW

Before the first "REPOSITION SOURCE CANDIDATES" round, don't worry about pile-up.


After the first "REPOSITION SOURCE CANDIDATES" round, check for pile-up:

  touch CHECK_FOR_PILEUP 


REPEATING the pile-up corrections is warranted when the SIZE of the extraction apertures for any piled sources may have changed since the last correction.  This typically occurs when a crowded neighbor is repositioned or pruned, allowing the aperture of the piled source to significantly change size.

Note that we are not yet concerned with pile-up effects on the piled source itself; we are concerned only with accurately modeling how much light from the piled source contaminates its neighbor's apertures.  




=================================================================================================
Launch an IDL session that will:
  * manage the ObsID extractions and adjustment of each source's BACKSCAL range 
  * merge the extractions 
  * prune sources based on overlap and Pb

From screen window #0 (named $TARGET), once the prompt has returned, run:

  'rm'  ABORT            >& /dev/null
  nice idl -queue |& tee -a merge_and_prune.log
    .run ae
    .run
    cd, CURRENT=cwd
    obsname = strtrim(strsplit(getenv('OBS_LIST'), /EXTRACT, COUNT=num_obs), 2)
    obsdir  = '../obs'+obsname+'/'
    if ~array_equal( file_test(obsdir), 1) then message, 'ERROR: some obsXXXX dirs do not exist.'

    ; Look up the number of events in each ObsID.
    obsid_event_tally = lonarr(num_obs)
    for ii=0,num_obs-1 do obsid_event_tally[ii] =  psb_xpar( headfits(obsdir[ii]+'spectral.evt', EXT=1), 'NAXIS2')

    obsid_finished_flags = obsdir + 'ae_finished'
    obsid_IDL_files      = obsdir + 'ae_code_block.pro'

    ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
    pass_name = file_readlink('CURRENT_PASS')
    is_first_iteration = 1B

    ; Manage pruning passes until stopping criterion met.
    repeat begin
      if ~is_first_iteration then begin
        ; Increment PASS name
        ; The authoritative value of the current pass name is the string in symlink CURRENT_PASS
        lexeme = stregex(pass_name,'([^0-9]*)([0-9]+)',/SUB,/EXT)
        pass_name = strjoin( [lexeme[1], strtrim(lexeme[2]+1,2)] )
        file_delete,          'CURRENT_PASS'
        file_link, pass_name, 'CURRENT_PASS'
      endif

      print, pass_name, systime(0), F='(%"\n\n=====================================================================\nSTARTING %s @%s\n")'
      tic

      ; Verify that no ObsID processing subroutine files remain.
      if ~array_equal( file_test(obsid_IDL_files), 0) then begin
        print, 'ERROR: the following files should not exist yet:'
        forprint, obsid_IDL_files, SUBSET=where(file_test(obsid_IDL_files))
        retall
      endif

      ; Remove the flag files that signal completion of ObsID extractions.
      file_delete, /ALLOW_NONEXISTENT, obsid_finished_flags
  
      ; LAUNCH EXTRACTIONS (largest ObsIDs first)
      foreach ii, reverse(sort(obsid_event_tally)) do begin
        file_copy, /OVERWRITE, 'IDL/extract.pro', obsid_IDL_files[ii]
        wait, 2
      endforeach

      ; Manage adjustment of BACKSCAL range.
      manage_extract_and_pileup_block1
      
      ; Verify extractions.
      ; Assess source validity, prune catalog.
      validation_block1, pass_name, VERIFY_EXTRACTIONS=is_first_iteration, NUM_PRUNED=num_pruned


      ; Manage target parameters.
      ; Only the first pass in this loop is allowed to:
      ;   discard PSFs (aka "shake the tree")
      ;   check for afterglows
      ;   check for pile-up
      ;   display residual images
      file_delete, /ALLOW_NONEXISTENT, /VERBOSE, ['DISCARD_PSF','CHECK_FOR_AFTERGLOWS','CHECK_FOR_PILEUP']
      file_copy, /OVERWRITE, '/dev/null', 'SKIP_RESIDUALS'
      
      ; Stopping criterion:  zero sources are pruned
      done = (num_pruned EQ 0) || file_test('ABORT')

      ; Rename some files to document this pass.
      wait, 10   ; ds9 may still be loading Pbslice.reg

      foreach fn, ['label.txt', 'prune.srclist', 'Pbslice.reg'] do begin
        if ~file_test(fn) then continue
        fdecomp, fn, disk, item_path, item_name, item_qual
        file_move, /VERBOSE, fn, item_path+item_name+'.'+pass_name+'.'+item_qual
      endforeach

      is_first_iteration = 0B

      print, pass_name, round(toc()/60), num_pruned, F='(%"\n\n=====================================================================\nFINISHED %s (%d minutes elapsed, %d pruned)\n")'
    endrep until done
    file_delete, /ALLOW_NONEXISTENT, 'ABORT'

    ; Terminate the ObsID IDL sessions, so we can archive their log files.
    print, now(),F='(%"\n\n=====================================================================\n(%s) TERMINATING ObsID IDL sessions.\n")'
    file_copy, /OVERWRITE, replicate('IDL/exit.pro',num_obs), obsid_IDL_files
    exit, /NO_CONFIRM
    end



To check the progress of the passes, run:
  egrep 'STARTING|minutes elapsed' merge_and_prune.log | egrep -v 'print,'

To make sure that the IDL processes are still running, go to the screen named "window1" and run:
  while 1
    printf "\n" 
    date "+%B-%d %H:%M"
    ls -ltr ae_validation*.log
    sleep 300
  end

If you wish to interrupt the automated passes at the end of the current pass, then from any window in /point_sources.noindex:
  touch ABORT



-------------------------------------------------------------------------------------------
PILE-UP CORRECTION
  
The extraction script for each ObsID will report piled extractions, and will pause while you go off and compute pileup-corrected photometry using the procedure pileup_reconstruction.txt.  You will then resume the extraction script, which will carry on with computing background spectra.

In each ObsID's screen window, the ae_pileup_screening tool prints a table of likely-piled extractions, and prepares the files needed for pile-up reconstruction.  If any of the rows (extractions) in the table printed above end with "STATUS=0", then you should stop and contact Patrick Broos!!!


NOTES ABOUT PILE-UP CORRECTION

Photon pile-up in bright sources can cause AE to over-estimate the significance of nearby weak sources.  This occurs when the wings or readout streak of the piled source contribute significant background within the aperture of the weak neighbor, but AE's "better backgrounds" algorithm underestimates that background (because pile-up biases photometry downward).


When necessary, we improve the photometry of piled extractions by constructing a simulated event list that approximates the events ACIS would have detected if pile-up effects were absent, and then performing photometry on that simulated data.  

Pile-up is detected and corrected on a single-ObsID basis, since pile-up of the same source can vary significantly among observations (and even within a single observation, but we're ignoring that).  The log files from the previous step (ae_validation.*.log) show a table of the extractions at risk of pile-up, determined by thresholding the extraction property RATE_3x3 (an estimate of the OBSERVED count rate in a cell of size 3x3 CCD pixels centered on the source position). 

There is, of course, no magic threshold on RATE_3x3 where pile-up "begins".  We have observed that the pile-up tool has produced an event rate correction of 1.11 for an extraction where RATE_3x3 was 0.075 ct/frame (source 111509.34-611602.0=p1_3368 in ObsID 0633 of NGC 3603).  Thus, we recommend that you consider pile-up correction for extractions with RATE_3x3 > 0.05 (ct/frame).  

Each pile-up reconstruction you perform will save corrected photometry to a file <src>/EPOCH_XXXX/model.photometry, which will be used in future runs of the background algorithm instead of <src>/EPOCH_XXXX/source.photometry.  Thus, in subsequent pruning iterations, AE will better model the wing/streak light that falls in neighboring apertures.





-------------------------------------------------------------------------------------------
SCAN FOR MISSED SOURCES

Note that even very bright sources can sometimes be accidentally removed from the catalog when mistakes are made in the execution of this procedure.

On the first re-extraction after source positions were updated, the procedure above will spawn a ds9 session for each ObsID showing the smoothed residuals (../obsXXX/inband_residual.img) remaining after the source & streak models are subtracted from the observed data.  That residual image is displayed with the "rainbow" color map, and is scaled so that zero is the color green.  Positive residuals (e.g. missing sources) are yellow/red; negative residuals (e.g. sources with pile-up correction, streak models with overestimated flux) are blue/purple.

*** Two more ds9 frames (hidden -- Frame -> tile frames will reveal them) contain the observed events (first frame) and the point source and streak models (second frame, ../obs1875/observation_counts_model.img), to help you understand the origin of any features you see in the residual image. ***

Do not rescale the image.  Don't worry about little bits of light inside extraction regions or at the edge of tight groups of sources -- that will work itself out.  You're looking for big, bright chunks of light with no extraction region -- these indicate bright sources that have been missed.  If you find such a source, note its position so you can add it back in as a "by-eye" source on the next iteration through the recipe (~line 1040 below).



-------------------------------------------------------------------------------------------
AFTERGLOW REVIEW
  
In a pass following source repositioning, the code above will direct you to review sources that appear to be severely contaminated with afterglow events (Pb_revised > 0.01).   Rarely, a bright source will generate enough spurious afterglow identifications to cause it to be mistakenly flagged.  YOU SHOULD VISUALLY REVIEW EACH SOURCE LISTED IN agr_prune.srclist. 

During this review, you'll want to consult the log messages in merge_and_prune.log ("P_b rises to") to see what the ae_afterglow_report tool had to say about each source it's proposing to prune (search merge_and_prune.log for "ae_afterglow_report").  You should suspect a false positive when the event energy for a claimed pair of afterglow events exhibits *rising* over time.

If you want to retain a source that ae_afterglow_report proposes to prune, prepend a semicolon to the appropriate line in agr_prune.srclist.



-------------------------------------------------------------------------------------------
PRUNING REVIEW (optional, if REVIEW_PRUNING specified earlier)

IF you decided beforehand to review catalog pruning, then a dialog box will prompt you to do so in each pass.


The ds9 session that comes up at the end of ae_analyze_pb color-codes the sources by Pb; the condemned sources are red, magenta, and orange.  Alternatively, there are several ways you could examine *individual* condemned sources (prune.srclist):

1. Display each extraction of a source in a separate frame.

   idl
   acis_extract, COLLATED_FILENAME='tables/most_valid_merge.collated', SRCLIST_FILENAME='prune.srclist', /SHOW_REGIONS, /INCLUDE_PRUNED_OBSIDS, OMIT_SINGLE_OBSID=0

   In the above call, use OMIT_SINGLE_OBSID=1 to avoid creating a ds9 frame for each extraction.
   In the above call, add the option /OMIT_BKG_REGIONS  to suppress display of background regions.

AND/OR (in a separate IDL session):

2. Display Pbslice.reg on the full-band target-level event list; pan to each source. 

    idl
    acis_extract, COLLATED_FILENAME='tables/most_valid_merge.collated', SRCLIST_FILENAME='prune.srclist', /SHOW_REGIONS, DISPLAY='../target.evt', REGION_FILE='Pbslice.reg', DS9_OPTION_STRING='-bin buffersize 8192'


AND/OR (in a separate IDL session):

3. Display Pbslice.reg on the full-band and narrow-band event lists; pan to each source. 

    idl
    acis_extract, COLLATED_FILENAME='tables/most_valid_merge.collated', SRCLIST_FILENAME='prune.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt','../target.hard.evt','../target.soft.evt','../target.vhard.evt'], REGION_FILE=replicate('Pbslice.reg',4), DS9_OPTION_STRING='-bin buffersize 8192'

Pay particular attention to condemned sources (red, magenta, and orange) that are crowding piled-up sources -- the pile-up corrector will have to be run again on any piled source that loses its "satellite" in Pb pruning, so make a note if that is about to happen.

To look at the bigger picture, you can review recon images (if they exist) from the last repositioning this way:

   idl
   acis_extract, COLLATED_FILENAME='tables/most_valid_merge.collated', merge_name='position', /SHOW_REGIONS, OMIT_SINGLE_OBSID=1

(set OMIT_SINGLE_OBSID=0 if you want to see the events from each ObsID).


If you want to remove extra sources or revive sources, edit prune.srclist.  Document your reasons very well!  

If you need to move sources now, look at SECTION VI: REPOSITION SOURCE CANDIDATES for inspiration.  To add by-eye sources now, see ADD/MOVE SOURCE CANDIDATES "BY-EYE" around line 1120 below.  They will be validated in the next pass.



=================================================================================================
CHECK LOGS FOR PROBLEMS

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop" ae_validation.*.log | egrep -v "DISPLAY variable|LS_COLORS|no in-band data|No HDUNAME|Program caused arithmetic error|error=|ARF was computed to be zero|has no rows|spans multiple|not observed|spectra will be reused|sources were not extracted|sources not in this observation|ran out of candidate|one emap pixel|(GRATTYPE|CCD_ID)' in supplied header|Background spectrum|BACKGROUND_IMBALANCE_THRESHOLD|Zero length|cannot intersect|finite width|subset of full catalog|saved from previous session|adopted region|StopTime|severely crowded|brute force|single extraction has excessive OVERLAP|reserve region|reached hard limits" |more


  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop|IMMORTAL" merge*.log | egrep -v "DISPLAY variable|LS_COLORS|no in-band data|No HDUNAME|Program caused arithmetic error|cannot intersect|finite width|different value|off-axis angle range|Merge is empty|excessive OVERLAP|position estimate skipped|too few events|DETNAM has different|RECONRES|Wrote source properties|enlarged enough|conflicting votes|ARF files listed above are obsolete|array_equal|topping criterion|print,|IMMORTAL EMAP_AVG"

  printf "\n\n"
  egrep 'minutes elapsed' merge_and_prune.log | egrep -v 'print,'
  
  
  
  
For a message like this:
  ERROR: Source 022606.83+615343.5 (4_542) is not in field of view, but extraction 022606.83+615343.5/446/ exists!
you should delete the offending extraction subdirectory (ObsID 446 in this example).  This comes about when a source moves a little, just onto or off of a given ObsID.


These messages are ok:
  WARNING: all 4 extractions have excessive OVERLAP; accepting only those with OVERLAP comparable to the best extraction (i.e. OVERLAP<=0.1).




-------------------------------------------------------------------------------------------
BACKSCAL Range Adjustment

The ae_adjust_backscal_range tool (called by the manage_extract_and_pileup_block1 code) makes 20 attempts to adjust the "BACKSCAL range" of each source.  Often, the BACKSCALE range for a **few** sources is unstable.  If this occurs for more than 1% of the catalog, the code prints the following warning:

  WARNING: BACKSCAL range adjustments have not converged for more than XX% of sources.

You should investigate such a warning.  The shell commands below will show, for each source not converged, the evolution of the BACKSCAL goal (the middle number inside the square brackets).

  foreach src ( `cat rerun.srclist | cut -f1 -d ' '` )
    grep "$src" merge*.log | grep BKSCL_GL
    printf '\n\n'
    end

When a cluster is observed on-axis in one ObsID and far off-axis in another, you may find that the BETTER_BACKGROUNDS processing for the off-axis ObsID consumes an unreasonable amount of time.  The only strategy we current have to shorten that run time involves discarding an observer-selected portion of the slow ObsID.  See Pat to execute that strategy, if you wish.  




=================================================================================================
ARCHIVE THE PASSES RUN ABOVE
=================================================================================================

Do this even if you have converged (have no sources to prune)!

**FROM SCREEN WINDOW #0 (named $TARGET)**, run:

  setenv PASS `readlink CURRENT_PASS`
  mkdir $PASS
  rsync -a ./{*.txt} tables/*.collated removed_sources.* ../checkpoint_for_backup/
  (mv by-eye.txt add.txt move.txt delete.txt              $PASS |& cat > /dev/null )
  rsync -a --info=NAME ./{*.txt} --exclude=block.txt --exclude='label.*' $PASS  
  rsync -a label.${PASS}.txt label.txt
  gzip *.log
  mv `find *.srclist \! -name 'all*.srclist'` label.pass* *.log *.log.gz *.reg *.sav *.ps screenrc.* *.collated tables/*.collated  $PASS  

  set filelist=`find $PASS -name all.reg -o -name polygons.reg -o -name xray_positions.reg -o -name psf_hook.reg -o -name removed_sources.reg -o -name fov_ptsrcs.reg -o -name modified_aperture.srclist`
  if ( "$filelist" != '' ) rsync -a --info=NAME $filelist  .

  chmod -R a-w      $PASS
  setenv PASS `echo $PASS | awk '{sub("pass",""); printf "pass%d", $0+1}'` ; printf "\n\nSTARTING $PASS\n\n" 
  'rm' CURRENT_PASS; ln -s $PASS CURRENT_PASS
  mkdir                    $PASS
  rsync -a all.srclist     $PASS
  touch SKIP_RESIDUALS REUSE_NEIGHBORHOOD VALIDATION_MODE



(Pat and Leisa only) Copy vital files to archive disk so they will be backed up.  Choose appropriate volumes (call below is just an example).  This needs to be run from the *archive* machine (hiawatha in this example).  Be sure $TARGET is defined on that machine.
-------------------------------------------------------
  idl
    relocate_target, /CHECKIN, /BACKUP, getenv("TARGET"), ['data/extract/checkpoint_for_backup/','data/extract/point_sources.noindex/pass*'], ARCHIVE_VOLUME='hiawatha2', FAST_VOLUME='ouray_fast'

  exit   





=================================================================================================
ADD/MOVE SOURCE CANDIDATES "BY-EYE" (optional)
=================================================================================================
If desired, visually review the stable catalog produced by the iterative pruning above, looking for opportunities to add unique source candidates by eye.

  ds9 ../target.evt -region all.reg -region removed_sources.reg &

DS9 note:  you can position the cursor where you want then type "c" and ds9 will spawn a new window showing you that position.  Set WCS coords to "degrees" and you'll have what you need to assign a "by-eye" position.

Record your by-eye positions (decimal RA,DEC) in an ASCII file `by-eye.txt'.

Then from any window in data/extract/point_sources.noindex/ (e.g. window1 in the screen session), run:

  setenv PASS `readlink CURRENT_PASS`
  idl -queue |& tee -a add_sources.log
    .run ae
    .run
    readcol, COUNT=num_sources, 'by-eye.txt', ra, dec, FORMAT='D,D', COMMENT=';'    ; read coordinates as DOUBLE precision
    if (num_sources GT 0) then begin
      label = getenv('PASS') + string(1+indgen(num_sources), F='(%"_%d")')

      forprint, label, ra, dec

      ae_source_manager, /ADD,  RA=ra, DEC=dec, LABEL=label, PROVENANCE='eye', POSITION_TYPE='eye'
    endif
    exit, /NO_CONFIRM
    end




=================================================================================================
BRANCH IN THIS PROCEDURE!
=================================================================================================
A. If you added sources just above, then run another round of pruning passes by returning to the top of the section

       "PRUNE INSIGNIFICANT SOURCE CANDIDATES"), roughly line 700.


- - - - - -  OR - - - - - -


B. If the catalog is completely stable, then you must choose one of two courses: 

  (1a) check the astrometry of your event data (section "CHECK THE ASTROMETRY OF YOUR EVENT DATA"), and then
  (1b) estimate new source positions (section "REPOSITION SOURCE CANDIDATES") 
  --OR--
  (2) declare the catalog to be converged (go to section "PREPARE CATALOG DATA PRODUCTS TO RELEASE", around line 2270).  

We strongly recommend that you calculate new position estimates for every source at least once during this validation procedure.  Perfect convergence is achieved when your last three passes are:

    * Pb pruning, with all sources found to be valid
    * Repositioning (which can affect Pb calculations)
    * Pb pruning, with all sources found to be valid

However, if pruning has converged but only a few sources have been pruned since the last repositioning pass, then you should ponder whether another full repositioning round is worth the labor required.  That decision depends on how the removal of those sources has affected surviving source apertures.  To investigate, you may have to look at the sources removed since the last repositioning (found in removed_sources.reg).  If the extraction aperture of any surviving source has been significantly altered since the last repositioning, then you must ponder whether its historic position estimate is acceptable to you.

    - For example, a classic situation is when one of a pair of crowded sources is pruned, and the aperture of the surviving source expands to full size.  The  historic RECON position estimate for the survivor (appropriate when we believed there were two sources) may no longer be appropriate, now that we believe there is only one source.


- - - - - -  OR - - - - - -


C. If the catalog is *almost* stable (<1%, but not zero sources removed in the last pass), then you may wish to jump ahead with the astrometry/repositioning work (option B1 above), as if the pruning had produced a stable catalog (option A above).  This compromise strategy is particularly appropriate for the *first* astrometry/repositioning round; that first repositioning will usually produce more sources to prune anyway, so there is little benefit from spending lots of time achieving a stable catalog beforehand.





#################################################################################
############# SECTION V: CHECK/IMPROVE THE ASTROMETRY OF YOUR TARGET ############
#################################################################################
Before we estimate new source positions (next section), we should do our best to remove astrometic errors in the event data.  Each ObsID should have been registered to an astrometric standard in the L1->L2 workflow, but that astrometric correction is not perfect.  Now that we've extracted the catalog with AE, we have single-ObsID source position estimates for a larger and better-validated catalog.  Thus, we have an opportunity to very easily estimate the astrometric offsets among the ObsIDs, which directly influences the tightness of the multi-ObsID PSF realized by on-axis sources.  Improving that multi-ObsID PSF will improve the image reconstructions we will use in the next section to estimate positions for crowded sources.  Here, we will also estimate the astrometric offsets between the ObsIDs and our astrometric reference. 

The only type of single-ObsID source position estimates available at this point are mean positions of the events in each aperture.  We ignore a position estimate if the aperture is >5' off-axis, if the aperture is reduced (indicating crowding), or if the source is not reliably detected in the single ObsID.  Uncertainties on single-ObsID position estimates depend on the local PSF size and on the number of counts in the aperture.  Inter-ObsID shift estimates are weighted averages of the single-source offsets (see eq. 5-6 in Bevington).  


The astrometric *check* below takes little time to run, and we recommend doing this every time you reach this point in the workflow.  Hopefully, you will need to *apply* astrometric corrections to the event data (the eight-step process below) infrequently.

***CAVEAT***
If you have a target with lots of crowding and/or bright diffuse emission, you may have a lot of spurious source candidates in early iterations of this recipe.  You may not want to adjust astrometry until much of that flotsam has been cleared away by P_B passes.  


=================================================================================================
First we launch IDL sessions to handle ObsID-specific processing. 
From screen window #0 (named $TARGET), run:

cat > IDL/rebuild_photometry.pro << \STOP
    PRO ae_code_block
    print, F='(%"REBUILDING SINGLE-OBSID PHOTOMETRY needed by ae_interObsID_astrometry...\n")'

    obsdir = "../obs"+getenv("OBS")+"/"
    file_delete, obsdir+"ae_finished", /ALLOW_NONEXISTENT 
    semaphore = wait_for_cpu()

    ae_standard_extraction, getenv("OBS"), EVTFILE="spectral.evt", EXTRACT_SPECTRA=0, TIMING=0, GENERIC_RMF_FN="generic.rmf", /BETTER_BACKGROUNDS, /PHOTOMETRY_STAGE_ONLY 

    release_cpu
    file_move, obsdir+"ae_lock", obsdir+"ae_finished", /OVERWRITE 
    return
    end 
\STOP


cat > IDL/extract.pro << \STOP
    PRO ae_code_block
    print, F='(%"\nRUNNING SINGLE-OBSID EXTRACTIONS to propagate shifted data...\n")'

    par = ae_get_target_parameters()
    par.CHECK_FOR_PILEUP  =0
    par.SKIP_RESIDUALS    =1
    par.REUSE_NEIGHBORHOOD=0  ; PROPAGATE revised event list
    obsdir = "../obs"+getenv("OBS")+"/" 
    file_delete, obsdir+"ae_finished", /ALLOW_NONEXISTENT 
    semaphore = wait_for_cpu()

    if keyword_set(par.BACKGROUND_MODEL_FILENAME) then $
      par.BACKGROUND_MODEL_FILENAME = obsdir + par.BACKGROUND_MODEL_FILENAME
    extract_and_pileup_block1, par, ADJUST_BACKSCAL=0, BACKGROUND_MODEL_FILENAME=par.background_model_filename

    release_cpu
    file_move, obsdir+"ae_lock", obsdir+"ae_finished", /OVERWRITE 
    return
    end 
\STOP


cat > block.txt << \STOP
  
  nice idl -queue |& tee -a ae_interObsID_astrometry.${OBS}.log 
    .run ae 
    file_delete,    "waiting_for_"+getenv("OBS"), /ALLOW_NONEXISTENT
    ae_processing_server, "../obs"+getenv("OBS")+"/ae_code_block.pro"
\STOP
  screen -S $SCREEN_NAME                -X readreg A block.txt

  foreach obs ($OBS_LIST)
    set file=../obs${obs}/ae_code_block.pro
    if (-e $file) 'rm' $file
    set marker=waiting_for_${obs}; touch ${marker}
    screen -S $SCREEN_NAME -p obs${obs} -X   paste A
    sleep 1
    while ( -e ${marker} )
      echo "Launching IDL for ObsID ${obs}..."
      sleep 1
    end
    sleep 2
  end


{Notes for future reference:

The standard extraction workflow is to build and extract source apertures, build the single-ObsID EPOCH_xxxx merges (to estimate single-ObsID photometry), then build and extract background regions.  Thus, the photometry in the single-ObsID EPOCH_xxxx merges normally uses source and background regions that are from different passes (which often have different source lists).  This inconsistency is uncomfortable, but the ae_better_backgrounds algorithm requires single-ObsID photometry, which requires some estimate of background.

Since the astrometry work done in the ae_interObsID_astrometry tool (run below) is quite important (and difficult), we want the single-ObsID source properties it uses (RA_DATA,DEC_DATA,SRC_CNTS,BKG_CNTS,BACKSCAL,BACKGRND,FLATGRND,SELFCNTS) to be as accurate and "current" as possible.  For technical reasons, the most straightforward and reliable way to collate fresh values for all those source properties is to rebuild and re-collate the single-ObsID EPOCH_xxxx merges.
}



***DON'T STOP HERE -- KEEP READING...***  


=================================================================================================
CHECK THE ASTROMETRY OF YOUR EVENT DATA

The above code will launch a bunch of IDL sessions, then return the prompt.
From screen window #0 (named $TARGET), then run:

  idl -queue |& tee -a ae_interObsID_astrometry.log
    .run match_xy
    .run ae
    .run
    ;------------- Launch single-ObsID processing -------------
    cd, CURRENT=cwd
    obsname = strtrim(strsplit(getenv('OBS_LIST'), /EXTRACT, COUNT=num_obs), 2)
    obsdir  = '../obs'+obsname+'/'
    if ~array_equal( file_test(obsdir), 1) then message, 'ERROR: some obsXXXX dirs do not exist.'

    ; Look up the number of events in each ObsID.
    obsid_event_tally = lonarr(num_obs)
    for ii=0,num_obs-1 do obsid_event_tally[ii] =  psb_xpar( headfits(obsdir[ii]+'spectral.evt', EXT=1), 'NAXIS2')

    obsid_finished_flags = obsdir + 'ae_finished'
    obsid_IDL_files      = obsdir + 'ae_code_block.pro'

    ; Verify that no ObsID processing subroutine files remain.
    if ~array_equal( file_test(obsid_IDL_files), 0) then begin
      print, 'ERROR: the following files should not exist yet:'
      forprint, obsid_IDL_files, SUBSET=where(file_test(obsid_IDL_files))
      retall
    endif

    ; Remove the flag files that signal completion of ObsID extractions and emap construction.
    file_delete, /ALLOW_NONEXISTENT, obsid_finished_flags
    file_list = file_search('../../pointing_*/obsid*/ae_lock')
    if keyword_set(file_list) then file_delete, /ALLOW_NONEXISTENT, file_list

    ; LAUNCH COLLATIONS (largest ObsIDs first)
    print, F='(%"REBUILDING SINGLE-OBSID PHOTOMETRY needed by ae_interObsID_astrometry...\n")'
    foreach ii, reverse(sort(obsid_event_tally)) do begin
      file_copy, /OVERWRITE, 'IDL/rebuild_photometry.pro', obsid_IDL_files[ii]
      wait, 5
    endforeach
  
    ; Wait for extractions to finish.
    wait_until_files_exist, 60, obsid_finished_flags, /DELETE
    
    ;------------- Check Astrometry -------------
    significance_threshold = 0.90
    run_command, /QUIET
    event2wcs_astr = get_astrometry_from_eventlist('../../tangentplane_reference.evt')

    ref_cat = build_FITS_cat('../../counterparts/astrometric_reference_catalog.fits.gz', event2wcs_astr, /ONLY_COORDINATES, RA_EXPRESSION='tb.ra_epochACIS', DEC_EXPRESSION='tb.dec_epochACIS', RA_ERROR_EXPRESSION='tb.ra_epochACIS_error', DEC_ERROR_EXPRESSION='tb.dec_epochACIS_error', NAME='Reference')

    ; When 2MASS is reference catalog:
    ; ref_cat = build_FITS_cat('../../counterparts/Twomass/2mass_highqual.fits', event2wcs_astr, RA_EXPRESSION='tb.RAJ2000', DEC_EXPRESSION='tb.DEJ2000', RA_ERROR_EXPRESSION='(tb.errmaj+tb.errmin)/2', DEC_ERROR_EXPRESSION='(tb.errmaj+tb.errmin)/2', NAME='TWOMASS')

    msg   = string( getenv("TARGET"), F='(%"Target %s is ready to display astrometry analysis.")')
    print, msg
    TimedMessage, tm_id, msg, TITLE=getenv('TARGET')+': astrometry', QUIT_LABEL='Close', PRESSED_ID=trash
    obsname = strtrim(strsplit(getenv('OBS_LIST'), /EXTRACT), 2)
    ae_interObsID_astrometry, obsname, ASTROMETRY=event2wcs_astr, REF_CATALOG=ref_cat, REF_NAME='GaiaNIR',$
                              SIGNIFICANCE_THRESHOLD=significance_threshold, TARGET_NAME=getenv('TARGET'),$
                              recommendation
    end



A dialog box will appear (in several minutes) telling you that a flurry of windows is about to be created, so you can prepare.

** LEAVE THE IDL SESSION RUNNING IN WINDOW 0 **


--------------------------
Verify that there were no errors reported by ae_interObsID_astrometry:
--------------------------

** LEAVE THE IDL SESSION RUNNING IN WINDOW 0 **

In any other screen window, grep the logs:

  egrep -i "WARNING|ERROR|DEBUG|halted|Stop" ae_interObsID_astrometry.*.log | egrep -v "DISPLAY variable|LS_COLORS|no in-band data|No HDUNAME|Program caused arithmetic error|error=|ARF was computed to be zero|has no rows|spans multiple|not observed|spectra will be reused|sources were not extracted|sources not in this observation|ran out of candidate|one emap pixel|Background spectrum|BACKGROUND_IMBALANCE_THRESHOLD|Zero length|cannot intersect|finite width|subset of full catalog|saved from previous session|adopted region|severely crowded|brute force|single extraction has excessive OVERLAP|Skipping source" |more

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop" ae_interObsID_astrometry.log | egrep -v "^-|DISPLAY variable|LS_COLORS|Program caused arithmetic error|1-sigma|excessive OVERLAP" 
  
  
If all ObsIDs have "NO adjustment recommended" then go to line ~1452 below to close out the IDL sessions.  

  
--------------------------
ds9 SESSION LAUNCHED BY THE TOOL
--------------------------
The "proposed shifts" ds9 session created above shows each ObsID's CURRENT event data, with arrow regions depicting the shift proposed for those data.  These arrow regions will help you judge the legitimacy of the shift recommendations in two ways.

1. For each ObsID separately, regions are shown only for sources that are judged to be useful for interObsID astrometric analysis.  

If some beautiful, bright, on-axis source has no region then you should stop to investigate why!


2. For ObsIDs with a recommended shift, the regions are vectors depicting the recommended shift.  By blinking a bright, on-axis source your brain may be able to sanity check the recommended shifts.



--------------------------
PLOTS PRODUCED BY THE TOOL
--------------------------
The most informative are perhaps the scatter plots showing "weight" vs "master - slave" (separate plots for X and Y) for the  matches to our astrometric reference.  Ideally, the match with the highest weight will be near zero offset.  If you find a single high-weight match pulling one way, and most of the other matches pulling the other way, then you should suspect that there is something wrong with the high-weight match (e.g. the reference position is crap or the source has high proper motion).


A scatter plot "xshift" vs "yshift" (with error bars) is created for each ObsID, showing all the available shift estimates, with their 1-sigma uncertainties depicted by error bars.  "One-hop" shift estimates come from each single-ObsID catalog match to our astrometric reference.  "Two-hop" estimates come from adding one-hop shifts to inter-ObsID shift estimates.  The final shift recommended by the code is a weighted average of these estimates, which is displayed with a five-sided star symbol.

Uncertainties on one-hop shift estimates are "inflated", as required, so that the set of shift estimates plotted for an ObsID is "consistent" (i.e. chi^2 is one).  THUS, THE APPARENT STATISTICAL CONSISTENCY OF POINTS IN THESE PLOTS IS NOT EVIDENCE THAT ALL IS WELL.  The inflation factors required are reported in subtitles of these plots, and in a column in the summary table.

IDEALLY, NO INFLATION OF UNCERTAINTIES WOULD BE REQUIRED TO ACHIEVE CONSISTENCY.



------------------------------------
LOG INFORMATION PRODUCED BY THE TOOL
------------------------------------
The  tool prints two tables titled "ALIGNMENT SUMMARY" reporting the ObsID-ObsID offsets and reporting the X-ray/reference offsets.  You can review these at the end of ae_interObsID_astrometry.log.

Recommended shifts (weighted averages) with signal-to-noise ratios ("S") are shown at the far right of these tables.

The matrix has X's along the diagonal because no numbers should ever appear there.  The -'s indicate missing shifts -- these appear most often when ObsID's are not overlapping enough for matching/shifts to be done.  The matrix is always symmetric, with matching shifts across the diagonal (but with opposite sign of course).



------------------------------------
REGION FILES PRODUCED BY THE TOOL
------------------------------------
The region files shift.twomass.to.xxxxx.reg show the matches between each ObsID's positions and the Reference Catalog, with the 20 brightest unmatched sources marked with yellow circles.  You can visualize all the ObsID/reference matches in the same ds9 session with the code below, BUT JUDGING WHETHER THERE IS A PROBLEM MAY BE VERY DIFFICULT.

  gzcat shift.*.to.*.reg.gz | egrep 'global|fk5|obs'  | egrep -v 'domain' > /tmp/matches.reg
  ds9 -title 'ObsID/reference Matches' ../target.evt -region /tmp/matches.reg &



------------------------------------
CORRUPTED SOURCES IN ASTROMETRIC REFERENCE CATALOG
------------------------------------
If you know there is a bright X-ray source that has high proper motion, then excluding it from this analysis may be very important.  Alas, there is no simple procedure for that.  The best option is to remove whatever stars in the Reference Catalog are at risk of matching the problematic X-ray source (section "PROCEDURE TO EDIT ASTROMETRIC REFERENCE CATALOG" in the reduction procedure).  A more cumbersome approach would be to manually remove the offending X-ray source from each of the X-ray collations used by ae_interObsID_astrometry.  





=================================================================================================
CHOOSE A SET OF ASTROMETRIC ADJUSTMENTS TO APPLY TO THE ObsID EVENT DATA 

We have chosen a policy to apply the recommended ObsID shift when its signal-to-noise ratio (S) is >1 (i.e. if the original zero-point of the coordinate system is not within the 1-sigma uncertainty of the proposed shift).  For S<1, the error on the proposed shift is bigger than the shift itself, so applying the shift would just be randomizing the alignments.

After the ALIGNMENT SUMMARY table, the code prints calls to the shell script "adjust_astrometry.csh" that will apply the suggested shifts.  The code implements our policy above, i.e. it will show deltax=0 or deltay=0 if the skyx or skyy shift in the table has S<1.


***  BUYER BEWARE HERE!!!  ***
You need to use your brain now -- don't be afraid.  These ``recommended'' shifts may not always be perfect.  Sometimes there are few astrometric reference matches participating in the decision, or there's a mismatch throwing everything off, or whatever.  We have seen a situation (G34) where two ObsIDs are well-aligned with each other, one says it's also well-aligned to the reference, but the other wants to go way off in the weeds (>0.5 sky pixels) to align to the reference.  Doing so would disrupt the good inter-ObsID alignment.  In this case, we chose not to apply the big shift, because it was late in the analysis and we had already done several alignments.  So think carefully about what's going on and take notes to justify your decisions.



IF YOU DECIDE THAT NO ObsIDs SHOULD BE SHIFTED, then clean up by pasting the following to screen window #0 (where the ae_interObsID_astrometry.log IDL session is still running):

    .run
    ; Terminate the ObsID IDL sessions, so we can archive their log files.
    print, now(),F='(%"\n\n=====================================================================\n(%s) TERMINATING ObsID IDL sessions.\n")'
    file_copy, /OVERWRITE, replicate('IDL/exit.pro',num_obs), obsid_IDL_files
    run_command, /IGNORE_STATUS, 'screen -S adjust_astrometry -X quit'
    exit, /NO_CONFIRM
    end

and MOVE ON TO SECTION VI: REPOSITION SOURCE CANDIDATES. 




=================================================================================================
APPLY YOUR CHOSEN SET OF ASTROMETRIC ADJUSTMENTS TO THE EVENT DATA, AND PROPAGATE TO AFFECTED DATA STRUCTURES

------------------------------------
In a new shell in a "tall" window, paste the "screen" command printed at the end of ae_interObsID_astrometry.log.  That will launch a "screen" session, with a "window" for each ObsID recommended for adjustment.



------------------------------------
IF YOU DEVIATE FROM THE RECOMMENDED SHIFTS, you must:

  1. In the screen session you just launched, edit the pre-pasted "adjust_astrometry.csh" calls to reflect the shifts you are applying.  Exit screen windows for ObsIDs that you are not shifting.

  2. At the IDL prompt, edit the ACTION_XSHIFT/ACTION_YSHIFT fields of the structure array "recommendation" to reflect the shifts you are applying.  Set those fields to zero if you are not shifting an ObsID.



------------------------------------
Launch all the pre-pasted "adjust_astrometry.csh" calls in the "adjust_astrometry" screen session.



------------------------------------
When the "adjust_astrometry.csh" calls finish with "SCRIPT EXITED NORMALLY", launch IDL sessions to update a variety of files that contain astrometry information, such as aspect histograms, exposure maps, event lists.  So in each ObsID window where you see "SCRIPT EXITED NORMALLY", execute this code:

    idl -queue |& tee -a ae_make_emap.log 
     .run ae 
     .run L1_2_L2_emaps.pro 
     .run
     file_delete, "ae_finished", /ALLOW_NONEXISTENT 
     semaphore = wait_for_cpu()
     ; We DISCARD existing aspect histograms!
     L1_2_L2_emaps, OUTPUT_DIR="AE", REUSE_ASPHIST=0, REUSE_INSTMAP=1, /INCLUDE_FI_AND_BI_PRODUCTS 
     release_cpu
     file_move, "ae_lock", "ae_finished", /OVERWRITE 
     exit, /NO_CONFIRM
     end

{We use the tool 'L1_2_L2_emaps' for that task.  It embodies many assumptions about our L1->L2 workflow, directory structure, and file naming conventions.}

{ For reasonable shifts, the time-consuming task of rebuilding the emaps themselves is not very important.  However, the L1_2_L2_emaps tool performs another task that MUST be done now---it rebuilds the event lists that AE uses as input (files AE/spectral*.evt) from the ObsID event lists that we just shifted.  We could save time by modifying the L1_2_L2_emaps tool to reuse the existing emaps.  However, there would be no guarantee that the emaps and event lists would have the same CCD and spatial filters applied---the observer would have to be very careful to supply the same CCD and spatial filters every time L1_2_L2_emaps was called.  We take the conservative (and time-consuming) approach of rebuilding both the emaps and AE event lists any time the ObsID is shifted.} 



------------------------------------
Rebuild target-level emaps and event lists.

Once the L1_2_L2_emaps sessions have been launched above, move to screen window #1 ("target-level") of that same screen session and run:

  idl -queue |& tee -a fullfield_mosaic.log  
    .r ae
    .run
    wait_while_files_exist, 60, 'pointing_*/obsid*/ae_lock'
    build_target_emap_and_eventlist, 'pointing*/obsid*/'
    exit, /NO_CONFIRM
    end


    
------------------------------------
Calculate new source positions that attempt to "follow" the movement of the relevant event data.
Propagate the shifted event positions into the AE point source extractions.

Find screen window #0 from the original screen session (where the ae_interObsID_astrometry.log IDL session is still running) and paste the following code.  This can be done while the emaps are still running above.

    .run
    ;------------- Move sources to follow the shifting data -------------
    ; Ask the observer to review the ACTION_XSHIFT/ACTION_YSHIFT fields in recommendation structure, which
    ; they are supposed to have updated if they deviation from the recommendations.
    forprint, recommendation.OBSID, recommendation.ACTION_XSHIFT, recommendation.ACTION_YSHIFT, F='(%"Assuming ObsID %6s has been shifted by %0.3f,%0.3f skypix")'

    msg = 'Do the ObsID shifts shown in the IDL window describe what you have actually done?'
    print, msg, F='(%"\n%s\n")'
    TimedMessage, tm_id, msg, TITLE=getenv('TARGET')+': Following The Light', QUIT_LABEL='No, let me edit recommendation.ACTION_XSHIFT, recommendation.ACTION_YSHIFT.', BUTTON_LABEL='Yes, estimate new source positions', BUTTON_ID=button_id, QUIT_ID=quit_id, PRESSED_ID=pressed_id

    if (pressed_id EQ quit_id) then begin
      print, F='(%"\n\nEdit recommendation.ACTION_XSHIFT, recommendation.ACTION_YSHIFT at the IDL prompt, and then type .go at the IDL prompt.")'
      retall
    endif

    ae_reposition_to_follow_obsid_shifts, event2wcs_astr, recommendation

    ;------------- Wait until emaps have been rebuilt. -------------
    wait_while_files_exist, 60, '../../pointing_*/obsid*/ae_lock'


    ;------------- Launch single-ObsID extractions -------------
    ; Remove the flag files that signal completion of ObsID extractions.
    file_delete, /ALLOW_NONEXISTENT, obsid_finished_flags

    ; LAUNCH EXTRACTIONS (largest ObsIDs first)
    foreach ii, reverse(sort(obsid_event_tally)) do begin
      file_copy, /OVERWRITE, 'IDL/extract.pro', obsid_IDL_files[ii]
      wait, 5
    endforeach
    print, F='(%"\nRUNNING SINGLE-OBSID EXTRACTIONS to propagate shifted data...\n")'
    checkpoint = systime(1)

    ; Wait for extractions to finish.
    wait_until_files_exist, 60, obsid_finished_flags, /DELETE

    ; VERIFY that new extractions were completed.
    obsid_collation_time = (file_info(obsdir + 'all.collated')).MTIME
    if ~array_equal( (obsid_collation_time GT checkpoint),1 ) then message, 'FAILURE: extractions did not run!'

    ; Terminate the ObsID IDL sessions, so we can archive their log files.
    print, now(),F='(%"\n\n=====================================================================\n(%s) TERMINATING ObsID IDL sessions.\n")'
    file_copy, /OVERWRITE, replicate('IDL/exit.pro',num_obs), obsid_IDL_files

    msg   = string( getenv("TARGET"), F='(%"Astrometry correction to target %s has been completed.")')
    print, msg
    TimedMessage, tm_id, msg, TITLE=getenv('TARGET')+': astrometry', QUIT_LABEL='Close', PRESSED_ID=trash
    exit, /NO_CONFIRM
    end


When prompted, confirm that the "recommendation" structure reflects the shifts you have applied.

Distributions of the proposed source position shifts will be plotted; sanity-check that they are consistent with the ObsID shifts, then confirm that you want to move the sources.


{The re-extraction of ALL ObsIDs is necessary because we have altered the celestial positions of sources.  The REUSE_NEIGHBORHOOD=0 option ensures that existing neighborhood event files (with old astrometry) will be discarded.  We recalculate backgrounds because they influence the optimized merge that's going to be done in the upcoming "REPOSITION SOURCE CANDIDATES" section.}


When IDL exits, check for error messages.

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop" ae_interObsID_astrometry.*.log | egrep -v "DISPLAY variable|LS_COLORS|no in-band data|No HDUNAME|Program caused arithmetic error|error=|ARF was computed to be zero|has no rows|spans multiple|not observed|spectra will be reused|sources were not extracted|sources not in this observation|ran out of candidate|one emap pixel|Background spectrum|BACKGROUND_IMBALANCE_THRESHOLD|Zero length|cannot intersect|finite width|subset of full catalog|saved from previous session|adopted region|severely crowded|brute force|single extraction has excessive OVERLAP|reserve region|reached hard limits|Skipping source" |more

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop" ae_interObsID_astrometry.log | egrep -v "^-|DISPLAY variable|LS_COLORS|Program caused arithmetic error|1-sigma|excessive OVERLAP" 
 

The second screen session that build emaps can be killed once both ObsID-level and target-level emaps are done.


=================================================================================================
ARCHIVE THIS PASS
=================================================================================================
**FROM SCREEN WINDOW #0 (named $TARGET)**, run:

  setenv PASS `readlink CURRENT_PASS`
  rsync -a ./{*.txt} tables/*.collated removed_sources.* ../checkpoint_for_backup/
  (mv by-eye.txt add.txt move.txt delete.txt              $PASS |& cat > /dev/null )
  rsync -a --info=NAME ./{*.txt} --exclude=block.txt --exclude='label.*' $PASS  

  gzip *.log
  mv `find *.srclist \! -name 'all*.srclist'`  *.log *.log.gz *.reg *.sav *.ps screenrc.* *.collated tables/*.collated  $PASS  

  set filelist=`find $PASS -name all.reg -o -name polygons.reg -o -name xray_positions.reg -o -name psf_hook.reg -o -name removed_sources.reg -o -name fov_ptsrcs.reg -o -name modified_aperture.srclist`
  if ( "$filelist" != '' ) rsync -a --info=NAME $filelist  .

  chmod -R a-w      $PASS
  setenv PASS `echo $PASS | awk '{sub("pass",""); printf "pass%d", $0+1}'` ; printf "\n\nSTARTING $PASS\n\n" 
  'rm' CURRENT_PASS; ln -s $PASS CURRENT_PASS
  mkdir                           $PASS
  rsync -a all.srclist label.txt  $PASS
  touch SKIP_RESIDUALS REUSE_NEIGHBORHOOD VALIDATION_MODE



=================================================================================================
*** CHECK ASTROMETRY AGAIN TO SEE IF IT HAS CONVERGED.***
=================================================================================================

IF you just shifted any ObsIDs that OVERLAP other ObsIDs, then repeat this whole section:

   SECTION V: CHECK/IMPROVE THE ASTROMETRY OF YOUR TARGET

which starts near line 1200.





#################################################################################
############# SECTION VI: REPOSITION SOURCE CANDIDATES ##################
#################################################################################

We strongly recommend that you calculate new position estimates for every source at least once during this validation procedure.  We typically reposition twice, interleaved with the pruning loop (above) as follows:

1. Loop over pruning until <1% of sources are removed on each pass.
2. Reposition
3. Loop over pruning until NO sources are removed.
4. Reposition
5. Loop over pruning until NO sources are removed.
6. Declare victory.


This exercise is long and detailed and should be done with a clear head.  Expect to spend at least half a day on it, for a large project...

For isolated sources < 5' off-axis we use mean data positions.
For isolated sources > 5' off-axis we use correlation positions.
For crowded sources (PSF fraction < 0.87) we reconstruct the source neighborhood (using 400 iterations by default) and find the peak in the recon image, called ML for maximum likelihood positions.




HERE, WE NEED A VISUAL SANITY CHECK ON THE ALIGNMENT OF OBSIDS!!!!!
Find a way for the SHOW stage to display circles that mark single-ObsID positions and uncertainties.
Sort the catalog by net counts.



=================================================================================================
Compute "mean data," "correlation," and "maximum likelihood" position estimates.
Choose a position estimate for each source.
=================================================================================================
From screen window #0 (named $TARGET), paste the following:

  touch           add.txt move.txt delete.txt
  touch  merge_position_lock  positions_data_lock  positions_corr_lock  positions_ml_lock
cat > block.txt << \STOP
  
  nice idl -queue |& tee -a check_positions_data.log
    .run ae
    .run 
    wait_while_files_exist, 60, 'merge_position_lock'
    ; Compute DATA positions.
    acis_extract, 'need_data.srclist', MERGE_NAME='position', /CHECK_POSITIONS, /SKIP_RECONSTRUCTION, /SKIP_CORRELATION
    file_delete, 'positions_data_lock', /ALLOW_NONEXISTENT
    exit, /NO_CONFIRM
    end
    
\STOP
  screen -S $SCREEN_NAME      -X readreg A block.txt
  screen -S $SCREEN_NAME -p 1 -X   paste A
sleep 3
cat > block.txt << \STOP
  
  nice idl -queue |& tee -a check_positions_corr.log
    .run ae
    .run 
    wait_while_files_exist, 60, 'merge_position_lock'
    ; Compute CORR positions.
    acis_extract, 'need_corr.srclist', MERGE_NAME='position', /CHECK_POSITIONS, /SKIP_RECONSTRUCTION
    file_delete, 'positions_corr_lock', /ALLOW_NONEXISTENT
    exit, /NO_CONFIRM
    end
    
\STOP
  screen -S $SCREEN_NAME      -X readreg A block.txt
  screen -S $SCREEN_NAME -p 2 -X   paste A
sleep 3
cat > block.txt << \STOP
  
  nice idl -queue |& tee -a check_positions_ml.log
    .run ae
    .run 
    wait_while_files_exist, 60, 'merge_position_lock'
    ; Compute ML (and CORR) positions.
    acis_extract, 'need_recon.srclist', MERGE_NAME='position', /CHECK_POSITIONS
    file_delete, 'positions_ml_lock', /ALLOW_NONEXISTENT
    exit, /NO_CONFIRM
    end
    
\STOP
  screen -S $SCREEN_NAME      -X readreg A block.txt
  screen -S $SCREEN_NAME -p 3 -X   paste A

  find [0-9]* -maxdepth 1 -type d  \( -name theta_00-05 -or -name position -or -name trash \)  -print0 -prune  | xargs -0 -n1000 rm -r
  
  nice idl -queue |& tee -a check_positions_subdivide.log
    .r ae
    reposition_block1


*** Note that the ae_improve_positions tool will plot the distributions of the proposed changes in RA and DEC, for each of the three types of position estimates, and print the plots to ae_improve_positions_ra.ps and ae_improve_positions_dec.ps. ***


{The "check_positions_subdivide" session runs POSITION merges and then decides which type of position estimate is needed for each source.  It then waits for the other three IDL sessions to calculate those estimates in parallel.  The session above then decides which source position estimate will actually be *used* for each source, based on the preferences established above and on the availability of the estimates.}

{These three "check_positions" sessions wait for the subdivide to finish, and then they estimate new source positions.  To save time, we compute data positions only for the sources for which we're going to prefer the data position (rate = 0.028 minutes per source on Hiawatha) and we compute correlation positions only for the sources for which we're going to prefer the CORR position (rate = 0.040 minutes per source on Hiawatha).  Sometimes ML positions fail, so for those sources we compute CORR positions as well.  ML positions used to take MUCH longer (rate = 0.73 minutes per source on Hiawatha) but Pat has recently sped up the code by reducing the size of the neighborhood being reconstructed and reducing the size of the PSF image (may need to beware of this, look for ringing or other artifacts in the recon).}



Review warning/error messages in log files.
-------------------------------------------------------
Review histograms showing position changes to look for something systematic.  Talk to Pat if you find something.  Exit IDL when ready.  Then, check the log files:

  ~/bin/acis_lkt4 . no_estimate.srclist use*.srclist add.txt move.txt delete.txt

  egrep -ni "WARNING|ERROR|DEBUG|halted|Stop" check_positions*.log | egrep -v "DISPLAY variable|LS_COLORS|no in-band data|No HDUNAME|Program caused arithmetic error|different value|not merged|accepting only|Zero length|cannot intersect|finite width|excessive OVERLAP|specified off-axis angle"


You may see this warning, which occurs when the ML algorithm could not calculate a position:
  WARNING!  Brightest recon pixel near source is not a local maximum; ML position estimate skipped.





=================================================================================================
REVIEW NEIGHBORHOODS OF PILED SOURCES
=================================================================================================
Below, image reconstructions of the position-optimized merges for piled sources are displayed.
Review the position proposed for each neighbor that may be affected by the wings of these very bright sources.

  idl
    .run
    fn = 'possibly_piled.srclist'
    if logical_true( file_lines(fn) ) then begin
      acis_extract, COLLATED_FILENAME='tables/position.collated', /SHOW_REGIONS, /OMIT_BKG_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='psf_hook.reg', SRCLIST_FILENAME=fn
    endif else print, 'NO PILED SOURCES FOUND.'
    exit, /NO_CONFIRM
    end



=================================================================================================
IDENTIFY "HOOK SOURCES"
=================================================================================================
The "hook" is defined as a second local maximum in reconstructed images of on-axis Chandra images.  Public information about the PSF hook is sparse.  In the few images publicly studied, the hook feature contains about 6% of the full-band power in the normal part of the PSF.  We know nothing about whether that 6% figure is energy dependent.  We do not know if the hook's properties are stable over time or stable over thermal conditions on the observatory.

With enough counts from a source ~<5' off-axis, our source detection process may resolve the PSF peak and its hook artifact, producing an AE source candidate for each.  Your task here is to try to identify, and remove, AE source candidates that you can reasonably attribute to hook artifacts.

We have available a model for the approximate spatial zone where spurious AE sources arising from hook artifacts should lie (the "panda" regions discussed below).  Our policy is that only AE sources inside these zones are eligible to be called "hook sources" (to be pruned).

However, simply pruning every AE source candidate that lies in a hook zone would be unwise.  In crowded fields, there can be many such sources, some of which are detectable astrophysical objects in a chance coincidence with a hook artifact.  Thus, we require two criteria to declare an AE source to be a "hook source" (to be pruned):

1. The source position must lie within a hook zone.
AND
2. The estimated power in the AE source candidate must be less than TWICE THE PREDICTED POWER IN THE HOOK.


An algorithm in the processing run above has identified sources that are likely to produce significant and resolvable hook artifacts (bright_psf_hook.srclist), and has identified sources that *may* lie in one of those "hook zones" (based only on their distance from hook-generating sources, not their azimuth).  It has also estimated the number of counts produced by each hook artifact.  

Beware that pile-up in the bright central source can make the hook brighter than you might think.

Visual review by an intelligent human is required to judge which AE sources, if any, meet our criteria for pruning.  As you do this review, make notes about these potential hook artifacts, including your decisions on whether they should be kept as real or deleted as hook artifacts.

THERE ARE NOW TWO WAYS TO REVIEW HOOKS!

FIRST, you can step through the list of possible "hook sources".  Many will not coincide with any panda regions and can be easily judged not to be hook artifacts.

  idl
    .run
    fn = 'near_psf_hook.srclist'
    if logical_true( file_lines(fn) ) then begin
      acis_extract, COLLATED_FILENAME='tables/theta_00-05.collated', /SHOW_REGIONS, /OMIT_BKG_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='psf_hook.reg', SRCLIST_FILENAME=fn
    endif else print, 'NO SOURCES ARE NEAR PSF HOOKS.'
    exit, /NO_CONFIRM
    end

SECOND, you can step through each bright, hook-generating source, and look for neighboring sources coincident with its hook regions ("pandas"). 

  idl
    .run
    fn = 'bright_psf_hook.srclist'
    if logical_true( file_lines(fn) ) then begin
      acis_extract, COLLATED_FILENAME='tables/theta_00-05.collated', /SHOW_REGIONS, /OMIT_BKG_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='psf_hook.reg', SRCLIST_FILENAME=fn
    endif else print, 'NO SOURCES GENERATE SIGNIFICANT PSF HOOKS.'
    exit, /NO_CONFIRM
    end


Both reviews display data from each ObsID participating in a theta < 5' merge, the merged data, and reconstruction of a merged data image.

Both reviews show the same types of regions:

* Each ObsID frame shows annular "panda" regions (group "hook panda") that depict the zones where the CXC expects power from the PSF hook to appear **IN IMAGE RECONSTRUCTIONS** of that ObsID's data.  The label on each panda reports a prediction of the number of counts expected in that hook artifact.

* The merged data frame and image reconstruction frame show all the single-ObsID hook regions (group "hook panda"), plus a large text label that reports the total number of hook counts expected from all ObsIDs (group "mulit-ObsID hook counts").

* As in all /SHOW_REGIONS sessions, polygon extraction apertures (polygons) are shown for the "current source," which is summarized in the IDL window.


----------------------------------------------------------
NOTE THAT PSF hooks are NOT RESOLVED in the data images!
They can be seen only in image reconstructions, where they will appear as power inside a panda region.  In ideal conditions, the panda region will contain a clear peak in the reconstructed image.
----------------------------------------------------------

For each AE source candidate you find inside a panda region (the first of our criteria), you should perform the following analysis steps:

A. Identify all the panda regions that enclose the source position.  These represent hook artifacts that lie close to our AE source candidate.

  Pandas may overlap, so you must move aside the ones of interest to see what lies underneath.
  It may be helpful to remove irrelevant pandas.

B. SUM the number of counts predicted to be generated by those hook artifacts, which are shown as labels on each of the enclosing pandas.  

C. Estimate the observed counts in the AE source under suspicion.

  Although AE has photometry estimates for the source under suspicion, we prefer to manually sum the pixels over the reconstruction peak associated with these hook artifacts.

D. IF the estimated power in the reconstruction peak (from C) is less than twice the predicted power in the hook (from B), then REMOVE THE SOURCE (by adding its telephone number to the file delete.txt).





NOTES:

If a model of the hook was publicly available, then we would simply incorporate the hook into the observatory PSF model, and no special procedures would be required.  

When an astrophysical source and a hook artifact are coincident, the hook contributes to the background of the astrophysical source.  Since we have no way to model or subtract that background, all source properties estimated by AE are biased.

When a bright source is observed at several roll angles (within 5' off-axis), the reconstructed image of the merged data will contain multiple distinct hook artifacts determined by those roll angles.  If the source is very bright, the image may resolve multiple hook peaks; weaker sources will produce an unresolve halo of light around the main peak.

The color used for a set of panda regions encodes the number of counts expected in the hook across all ObsIDs in the "most valid" merge, as follows:

hook counts    color
-----------------------
    <5        'green'
    5:10      'cyan'
    10:20)    'yellow'
    20:40)    'red'
    >40       'magenta'






=================================================================================================
Visually review the recommendations made by the ae_improve_positions tool.
=================================================================================================

The ae_improve_positions tool divides the source list into these six groups, depicted in review.reg:

     SRCLIST FILE    :  COLOR :      GROUP
 ----------------------  ------------------------------------------------
 no_estimate.srclist :  red   : the preferred position estimate is MISSING
     use_eye.srclist :  yellow: position previously defined by the observer (FITS keyword POSNTYPE=='eye')
   use_recon.srclist :  green : source will move to RECON position
    use_corr.srclist :  blue  : source will move to CORRELATION position
    use_data.srclist :  cyan  : source will move to MEAN DATA position
 use_catalog.srclist :  tan   : source will not move (initially empty)

Subsections below show how to visually review the position estimates for each group of sources.  To prioritize your review of these recommendations, the use_recon.srclist/use_corr.srclist/use_data.srclist files are sorted by a "suspicion metric".

  1. The basic suspicion metric is the distance the source will move, normalized by the aperture size.  
  
  2. Sources with a reasonably close neighbor are at some risk that their position CORR and DATA estimates are corrupted, and are thus assigned a higher suspicion than isolated ones.
  
  3. Among the recon sources, we may wish the observer to review the existence of sources with large Pb, so they are assigned a higher suspicion than more significant sources.



As you conduct this review, you should record any decisions you make to over-ride the recommendations by annotating the relevant sourcelists directly, following each source name and label with a semicolon, then notes about what position that source should use or other pertinent information.  I review all the sourcelists making notes about where sources should move (I append a semicolon to every source I review, even if I don't make notes about it, so I'll know which ones I looked at).  After all the reviewing is done, I go back and do all the text editing to put sources into different sourcelists if I want to use a different position.  I never remove a source from a given sourcelist, I just comment it out once I've added it (to the end) of a different sourcelist.

Begin by loading your sourcelists into a text editor, e.g.

  tw no_estimate.srclist use*.srclist add.txt move.txt delete.txt




Each subsection below shows several ways the SHOW_REGIONS stage can be used to conduct the visual review.  The first displays each extraction of a source in ds9, and the second displays the region file review.reg on top of the project-level event list.  In these ds9 sessions, region shape encodes position type

  +       = catalog position
  diamond = mean data position
  circle  = correlation position
  box     = ML recon position



DS9 note:  you can position the cursor where you want then type "c" and ds9 will spawn a new window showing you that position.  Set WCS coords to "degrees" and you'll have what you need to assign a "by-eye" position.



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Visually review the BY-EYE POSITIONS (+) you determined before
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(None will exist if this is your first pass through repositioning).  You should look at all of these to make sure that your decision from the last time through is still valid.  Ideally, it's good to move these to one of the calculated positions (RECON, DATA, or CORR) if at all possible.

  idl
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_eye.srclist', /SHOW_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg', /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OMIT_SINGLE_OBSID=1

   In the above call, use OMIT_SINGLE_OBSID=0 to create a ds9 frame for each extraction.


AND/OR (in a separate IDL session):

  idl
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_eye.srclist', /SHOW_REGIONS, DISPLAY=['../target.soft.evt','../target.hard.evt'], REGION_FILE=['review.reg','review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

OR
   idl 
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_eye.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt'], REGION_FILE=['review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'
    
NOTE -- the above call sometimes barfs if it takes ds9 too long to draw the region files.  Just ".continue" in the IDL window and it will figure it out.

It's useful to overlay psf_hook.reg and removed_sources.reg in the second ds9 session, especially if you use these displays to nominate by-eye sources.  This is done automatically now.

If you want to adjust the position again, append a semicolon to the line listing the source and record your new position (in decimal degrees); you will later use that information to manually reposition the source.



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Visually review the sources that are missing their preferred position estimates
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='no_estimate.srclist', /SHOW_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg', /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OMIT_SINGLE_OBSID=1

   In the above call, use OMIT_SINGLE_OBSID=0 to create a ds9 frame for each merged ObsID.
   Use INCLUDE_PRUNED_OBSIDS=1 to show ds9 frames for ObsIDs excluded from the merge.


AND/OR (in a separate IDL session):

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='no_estimate.srclist', /SHOW_REGIONS, DISPLAY=['../target.soft.evt','../target.hard.evt'], REGION_FILE=['review.reg','review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

OR

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='no_estimate.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt'], REGION_FILE=['review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'


Catalog positions (+) are shown for all sources.


You should investigate why these sources have no position estimate available. 
  * ML positions (boxes) will be missing for sources that report this error message:
    "WARNING! Brightest recon pixel near source is not a local maximum; ML position estimate skipped."

  * CORR positions (circles) will be missing for sources that report this error message:
    "WARNING!  Correlation position cannot be computed; peak lies on search boundary."

  * DATA positions (diamonds) should never be missing.


For crowded sources missing their ML positions, this is often because they are trying to climb up the wings of their bright neighbor.  Sometimes we choose a secondary recon peak for the source position or even delete the source altogether if there is no recon peak to associate with it.  This is accomplished with ae_soure_manager, /MOVE or ae_source_manager, /REMOVE -- this will be shown below.  Another good idea is to see how the source in question fares in the recon of a nearby source (e.g. the bright one it's trying to climb).

YOU SHOULD MAKE A DECISION ON WHAT POSITION TO ADOPT FOR EACH SOURCE (RECON, DATA, CORR, current catalog, or a by-eye estimate) and record that decision by annotating no_estimate.srclist.  

It's useful to overlay psf_hook.reg and removed_sources.reg to the second ds9 session, especially if you use these displays to nominate by-eye sources.



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Visually review the RECON POSITIONS (box)
- - - - - - - - - - - - - - - - - - - - -
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_recon.srclist', /SHOW_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg', /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OMIT_SINGLE_OBSID=1

   In the above call, use OMIT_SINGLE_OBSID=0 to create a ds9 frame for each extraction.


AND/OR (in a separate IDL session):

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_recon.srclist', /SHOW_REGIONS, DISPLAY=['../target.soft.evt','../target.hard.evt'], REGION_FILE=['review.reg','review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

OR

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_recon.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt'], REGION_FILE=['review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'


The proposed repositioning you are reviewing is from the cross (catalog) to the box (reconstruction).

WHEN THE PIXELS IN THE RECONSTRUCTED IMAGE ARE TOO SMALL, YOU WILL SEE MULTIPLE PEAKS THAT ARE TOO CLOSE TO BE RESOLVABLE SOURCES.  I SUGGEST USING A LITTLE SMOOTHING IN DS9 TO PRODUCE A LOCAL MAXIMUM, AND THEN DEFINING A "BY-EYE" POSITION AT THAT PEAK.

The file use_recon.srclist consists of two blocks of sources, each sorted by the magnitude of the proposed shift.  The first block (suspicion > 10) are low-significance sources (large PROB_NO_SOURCE).  When you get tired of looking at the first block, advance to the first source in the second block by entering its label; then use the "r" (rejoin the sequence) command to get the SHOW tool to advance down the list from there.

You will see that some weak sources in this list seem to be positioned far from any recon peak in the _current_ reconstruction.  Presumably the "recon_detect" reconstructions that produced these candidate sources did have peaks at these positions.  We believe this is simply an illustration of the fact that low-level features in reconstructions will vary with changes in the bin phase (or bin size), or with changes in the PSF. 

It's useful to overlay psf_hook.reg and removed_sources.reg to the second ds9 session, especially if you use these displays to nominate by-eye sources.

If you find a source that should NOT use the RECON position, then append a semicolon followed by your preferred position type (DATA, CORR, current catalog) or a new by-eye set of coordinates (in decimal degrees).




- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Visually review the CORR POSITIONS (circle)
- - - - - - - - - - - - - - - - - - - - - - 
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_corr.srclist', /SHOW_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg', /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OMIT_SINGLE_OBSID=1

   In the above call, use OMIT_SINGLE_OBSID=0 to create a ds9 frame for each extraction.


AND/OR (in a separate IDL session):

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_corr.srclist', /SHOW_REGIONS, DISPLAY=['../target.soft.evt','../target.hard.evt'], REGION_FILE=['review.reg','review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

OR

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_corr.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt'], REGION_FILE=['review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'


The proposed repositioning you are reviewing is from the cross (catalog) to the circle (correlation).

The file use_corr.srclist consists of two blocks of sources, each sorted by the magnitude of the proposed shift.  The first block (DISTANCE_REG2REG < SRC_RAD) are sources that are at some risk that their position estimates are corrupted by a reasonably close neighbor.  When you get tired of looking at the first block, advance to the first source in the second block by entering its label; then use the "r" (rejoin the sequence) command to get the SHOW tool to advance down the list from there.

If you find a source that should NOT use the CORR position, then append a semicolon followed by your preferred position type (RECON, DATA,  current catalog) or a new by-eye set of coordinates (in decimal degrees).




- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Visually review the DATA POSITIONS (diamond)
- - - - - - - - - - - - - - - - - - - - - - -
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_data.srclist', /SHOW_REGIONS, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg', /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OMIT_SINGLE_OBSID=1

   In the above call, use OMIT_SINGLE_OBSID=0 to create a ds9 frame for each extraction.


AND/OR (in a separate IDL session):

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_data.srclist', /SHOW_REGIONS, DISPLAY=['../target.soft.evt','../target.hard.evt'], REGION_FILE=['review.reg','review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

OR

    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='use_data.srclist', /SHOW_REGIONS, DISPLAY=['../target.evt'], REGION_FILE=['review.reg'], DS9_OPTION_STRING='-bin buffersize 8192'

The proposed repositioning you are reviewing is from the cross (catalog) to the diamond (mean data).

The file use_data.srclist consists of two blocks of sources, each sorted by the magnitude of the proposed shift.  The first block (DISTANCE_REG2REG < SRC_RAD) are sources that are at some risk that their position estimates are corrupted by a reasonably close neighbor.  When you get tired of looking at the first block, advance to the first source in the second block by entering its label; then use the "r" (rejoin the sequence) command to get the SHOW tool to advance down the list from there.

If you find a source that should NOT use the DATA position, then append a semicolon followed by your preferred position type (RECON, CORR, current catalog) or a new by-eye set of coordinates (in decimal degrees).


We sometimes find that sources extracted with ~90% extraction regions (thus about to be assigned a mean data position) are in fact pretty crowded with nearby sources.  You can do a reconstruction of those sources if you want.

To do recon's on extra sources:
Make a small sourcelist, extra_recons.srclist, containing the sources that need reconstruction, e.g.

    egrep "'cc624'|'cc427'|'cc587'|'cc30'|'cc351'|'cc289'|'cc290'" label.txt > extra_recons.srclist

then merge and estimate positions.

  nice idl -queue |& tee -a check_positions.log
    .run
    acis_extract, 'extra_recons.srclist', MERGE_NAME='position', /CHECK_POSITIONS
    acis_extract, COLLATED_FILENAME='tables/position.collated', SRCLIST_FILENAME='extra_recons.srclist', /SHOW_REGIONS, /OMIT_BKG_REGIONS, INCLUDE_PRUNED_OBSIDS=0, OBSID_REGION_GLOB_PATTERN='../obs*/psf_hook.reg', MERGE_REGION_FILENAME='removed_sources.reg'
    exit, /NO_CONFIRM
    end

Then copy these sources to the appropriate use_*.srclist.  Generally they should move to their recon positions, so they should be appended to use_recon.srclist.  Be sure to comment them out of use_data.srclist if you want them to use a different position.




- - - - - - - - - - - - 
Modify your sourcelist to reflect the decisions you have made.
- - - - - - - - - - - - 
During your reviews you should have recorded your decisions to change AE's recommended position update by adding notes to rows in use_catalog.srclist, no_estimate.srclist, use_recon.srclist, use_corr.srclist, use_data.srclist, add.txt, move.txt, and delete.txt.  

Now you must implement your decisions by "moving" sources to the correct srclist file as needed.  We implement such a "move" by COPYING the source to the correct file and then COMMENTING OUT (with ';') the row where the source originally appeared.

A series of 'grep' commands like those below may help you efficiently find the various lines you need to add to the end of each use*.srclist file.  Similar multi-file searches in TextWrangler can help you find each line you need to comment out.

  set srclists='use_corr.srclist use_data.srclist use_eye.srclist use_recon.srclist no_estimate.srclist'
  rsync -av                 $srclists /tmp/
  egrep -hi "cat"           $srclists | egrep -v '^;'                  >> use_catalog.srclist
  egrep -hi "corr"          $srclists | egrep -v '^;|AE correlation'
  egrep -hi "data"          $srclists | egrep -v '^;|AE mean data'
  egrep -hi "recon|ML"      $srclists | egrep -v '^;|AE reconstruction'

  egrep -hi "move|peak"     $srclists | egrep -v '^;'                   >> move.txt
  egrep -hi "add"           $srclists | egrep -v '^;'                   >> add.txt
  egrep -hi "remove|delete" $srclists | egrep -v '^;'                   >> delete.txt

  egrep -hiv ";"            $srclists | sort        

To figure out the name of a source when you only have its label, use an egrep on label.txt, e.g.:
  egrep 'c1091)' label.txt


- - - - - - - - - - - - 
Verify your sourcelists
- - - - - - - - - - - - 
When you think that the source lists 
  use_eye.srclist    
  use_recon.srclist  
  use_data.srclist   
  use_corr.srclist   
  use_catalog.srclist
  no_estimate.srclist
  add.txt 
  move.txt 
  delete.txt
have been correctly edited AND YOU HAVE SAVED THEM TO DISK, look for mistakes.


First, try to verify that together they contain the correct number of uncommented sources.  

  egrep -v '^[[:space:]]*(;|$)'   all.srclist                                         | wc -l
  egrep -v '^[[:space:]]*(;|$)'  use*.srclist no_estimate.srclist delete.txt move.txt | wc -l
  dmlist tables/position.collated block |grep Table
  
Those three calls should report the same number of sources.  If not, then you can look for uncommented sources with comments that you have not already dealt with:

  egrep -v '^[[:space:]]*(;|$)'  use*.srclist no_estimate.srclist delete.txt move.txt | more



=================================================================================================
Apply the position review changes
=================================================================================================

First, add any new sources, then remove sources that you want to delete.  Then move the sources in use_recon.srclist, use_corr.srclist, and use_data.srclist. (The POSNTYPE property of these sources will be changed by ae_source_manager below.)  Finally, apply the "by-eye" positions that you came up with for the new sources that you just added to use_eye.srclist and any sources that were in use_eye.srclist from a previous pass and that need to be moved again.  Use the convention of POSITION_TYPE='eye' so that the ae_improve_positions tool will not override your decision in the future.  BE VERY CAREFUL TO USE DOUBLE-PRECISION FOR THE SOURCE COORDINATES (the "D" appended to the numbers in the example below)!  

COPY AND EDIT THE FOLLOWING TEMPLATE TO IMPLEMENT YOUR DECISIONS (or just run as written if appropriate): 

From screen window #0 (named $TARGET), run:

  idl -queue |& tee -a update_positions.log
    .run ae
    .run
    ;Add sources:
    fn = 'add.txt'
    if logical_true( file_lines(fn) ) then begin
      readcol, COUNT=num_sources, fn, ra, dec, FORMAT='D,D', COMMENT=';'    ; read coordinates as DOUBLE precision
      if (num_sources GT 0) then begin
        print, num_sources, F='(%"INFORMATION: ADDING %d SOURCES:")'
        label = getenv('PASS') + string(1+indgen(num_sources), F='(%"_%d")')
        forprint, label, ra, dec
        ae_source_manager, /ADD,  RA=ra, DEC=dec, LABEL=label , PROVENANCE='eye', POSITION_TYPE='eye'
      endif
    endif

    ;Delete sources:
    fn = 'delete.txt'
    if logical_true( file_lines(fn) ) then begin
      readcol, COUNT=num_sources, fn, srcname, FORMAT='A', COMMENT=';'    ; read source name
      if (num_sources GT 0) then begin
        print, num_sources, F='(%"INFORMATION: DELETING %d SOURCES:")'
        forprint, srcname
        ae_source_manager, /REMOVE, NAME=srcname, TRASH_DIR='/tmp/$TARGET/$PASS/invalid_sources/'
      endif
    endif

    ;Move sources to user-determined positions:
    fn = 'move.txt'
    if logical_true( file_lines(fn) ) then begin
      readcol, COUNT=num_sources, fn, srcname, ra, dec, FORMAT='A,D,D', COMMENT=';'    ; read coordinates as DOUBLE precision
      if (num_sources GT 0) then begin
        print, num_sources, F='(%"INFORMATION: MOVING %d SOURCES to BY-EYE POSITIONS:")'
        forprint, srcname, ra, dec
        ae_source_manager, /MOVE, NAME=srcname, RA=ra, DEC=dec, POSITION_TYPE='eye'
      endif
    endif

    ;Reposition the uncommented sources that need to move to an AE-estimated position:
    for ii = 0, 2 do begin
      fn =    (['use_recon.srclist',              'use_corr.srclist',              'use_data.srclist'])[ii]
      if logical_true( file_lines(fn) ) then begin
        readcol, COUNT=num_sources, fn, sourcename, FORMAT='A', COMMENT=';'
        if (num_sources GT 0) then begin
          print, num_sources, fn, F='(%"\nINFORMATION: IMPLEMENTING %d NEW POSITIONS IN %s.\n")'
          ae_source_manager, MERGE_NAME='position',  NAME=sourcename, $
                   UPDATE_POSITIONS_RECON=(ii EQ 0), UPDATE_POSITIONS_CORR=(ii EQ 1), UPDATE_POSITIONS_DATA=(ii EQ 2)
        endif
      endif
    endfor; ii
    ae_source_manager, ORPHAN_SOURCES=os
    exit, /NO_CONFIRM
    end

  egrep -i "WARNING|ERROR|DEBUG|INFORMATION|halted|Stop|QUANTCOR" update_positions.log 

For the notes:
  egrep -i "INFORMATION" update_positions.log


=================================================================================================
Archive this pass.
Remove file REUSE_NEIGHBORHOOD, so that neighborhood event lists will be re-centered on the sources.
Touch  file DISCARD_PSF to "shake the tree" on the next extraction.  This regenerates PSFs for all sources, not just the ones that moved a lot.
=================================================================================================

From screen window #0 (named $TARGET), run:

  setenv PASS `readlink CURRENT_PASS`
  rsync -a ./{*.txt} tables/*.collated removed_sources.* ../checkpoint_for_backup/
  (mv by-eye.txt add.txt move.txt delete.txt              $PASS |& cat > /dev/null )
  rsync -a --info=NAME ./{*.txt} --exclude=block.txt --exclude='label.*' $PASS  

  gzip *.log
  mv `find *.srclist \! -name 'all*.srclist'`  *.log *.log.gz *.reg *.sav *.ps screenrc.* *.collated tables/*.collated  $PASS  

  set filelist=`find $PASS -name all.reg -o -name polygons.reg -o -name xray_positions.reg -o -name psf_hook.reg -o -name removed_sources.reg -o -name fov_ptsrcs.reg -o -name modified_aperture.srclist`
  if ( "$filelist" != '' ) rsync -a --info=NAME $filelist  .
  rsync -a --info=NAME $PASS/theta_00-05.collated tables/

  chmod -R a-w      $PASS
  setenv PASS `echo $PASS | awk '{sub("pass",""); printf "pass%d", $0+1}'` ; printf "\n\nSTARTING $PASS\n\n" 
  'rm' CURRENT_PASS; ln -s $PASS CURRENT_PASS
  mkdir                           $PASS
  rsync -a all.srclist label.txt  $PASS
  'rm'  SKIP_RESIDUALS       >& /dev/null
  'rm'  REUSE_NEIGHBORHOOD   >& /dev/null

***Consider whether you want to do the following:***
  touch CHECK_FOR_AFTERGLOWS 
  touch DISCARD_PSF
  
In early iterations you may not want to check for afterglows yet.  We "shake the tree" with DISCARD_PSF -- this will recompute PSFs for all sources, not just the ones that moved a lot.  Again, in early iterations you may not want to take the time for that -- it's best to wait until later iterations, when most of the flotsam has been cleared out of your sourcelist.

{REUSE_NEIGHBORHOOD is removed so that neighborhoods are rebuilt in the next extraction, to keep them centered on the source position.}


(Pat and Leisa only) Copy vital files to archive disk so they will be backed up.  Edit archive and fast volumes as needed.
-------------------------------------------------------
  idl
    relocate_target, /CHECKIN, /BACKUP, getenv("TARGET"), ['data/extract/checkpoint_for_backup/','data/extract/point_sources.noindex/pass*','data/pointing*'], ARCHIVE_VOLUME='hiawatha2', FAST_VOLUME='osceola_fast'

  exit 


=================================================================================================
JUMP BACKWARDS IN THIS PROCEDURE!
=================================================================================================

  Return to the top of Section IV, "PRUNE INSIGNIFICANT SOURCE CANDIDATES" (around line 700 in this procedure).




#################################################################################
############# SECTION VII: PREPARE CATALOG DATA PRODUCTS TO RELEASE ##################
#################################################################################

At the end of Pb pruning (just above "BRANCH IN THIS PROCEDURE"), you archived the last pass, so here you should be starting a new pass.


Verify that no bright sources were missed (optional)
---------------------------------------------------------------------
Even very bright sources can sometimes have been removed from the catalog when mistakes are made in the execution of this procedure.  You *should* have been reviewing the residual images that the background code displays, looking for missing sources.  If you have not been doing that, then run the code below to display that residual image once again.

In each ObsID's screen window, run:
  idl
    .run ae
    par = ae_get_target_parameters()
    ae_standard_extraction, getenv("OBS"), SOURCE_NOT_OBSERVED=source_not_observed, EVTFILE='spectral.evt', TIMING=0, GENERIC_RMF_FN='generic.rmf', EXTRACT_SPECTRA=0, /BETTER_BACKGROUNDS, NEIGHBOR_INVALID_THRESHOLD=par.neighbor_invalid_threshold, /BUILD_MODELS_ONLY
    exit, /NO_CONFIRM





=================================================================================================
SORT THE CATALOG BY RA
REGENERATE FINAL DATA PRODUCTS
=================================================================================================
We assume here that the catalog and source positions have not been altered since the last Pb pruning step.  However so far theta-range merges exist for only a subset of sources (those with failing single-ObsID Pb values); we perform those merges here on *every* source so that the Pb_min value published will reflect the best significance achieved in any merge and band, AND so that we can identify "occasional" sources.  

Run this:

  find [0-9]* -maxdepth 1 -type d \( -name theta_00-03 -or -name theta_02-06 -or -name theta_05-09 -or -name theta_08-14 -or -name theta_13-99 -or -name most_valid -or -name trash \) -print0 -prune  | xargs -0 -n1000 rm -r 


Then from screen window #0 (named $TARGET), run the following (may have to paste in several parts):
  
  'rm'  CHECK_FOR_AFTERGLOWS >& /dev/null
  touch REVIEW_PRUNING
  nice idl -queue |& tee -a validation_data_products.log
    .run ae
    .run
    ae_source_manager, /SORT_RA

    ; Re-collate the single-ObsID merges since the catalog was re-ordered above.
    print, 'Re-collating the single-ObsID merges ...'
    obsname = strtrim(strsplit(getenv('OBS_LIST'), /EXTRACT), 2)
    obsdir  = '../obs'+obsname+'/'
    single_ObsID_collation_fn = obsdir + 'all.collated'
    for ii=0,n_elements(obsname)-1 do $
      acis_extract, 'all.srclist', obsname[ii], MERGE_NAME='EPOCH_'+obsname[ii], COLLATED_FILENAME=single_ObsID_collation_fn[ii], MATCH_EXISTING=1, VERBOSE=0


    ; Repeat the validation process to generate various data products with the sorted catalog,
    ; including tables/most_valid_merge.collated
    pass_name = file_readlink('CURRENT_PASS')
    validation_block1, pass_name, /VERIFY_EXTRACTIONS, /CALCULATE_BEST_PB, NUM_PRUNED=num_pruned

    if (num_pruned GT 0) then begin
      title = 'WARNING! WARNING! WARNING! THE CATALOG IS NOT CONVERGED'
      msg   = 'THE CATALOG FOR '+getenv("TARGET")+' IS NOT CONVERGED!  You may want more pruning passes!'
      print, msg, F='(%"\n%s\n")'
      TimedMessage, tm_id, msg, TITLE=title, QUIT_LABEL='Close', PRESSED_ID=trash
    endif 

    ; Construct a FITS table that contains the important source properties relevant to validation.
    ; Load data structures saved by ae_analyze_pb tool: 
    sObj = OBJ_NEW('IDL_Savefile', 'Pbslice.sav')
    sObj->Restore, ['NUM_SOURCES','CATALOG_NAME','VALIDATION_REPORT','BANDS'], /VERBOSE

    ; Extract some source properties from "most_valid" merge.
    run_command, 'dmcopy "tables/most_valid_merge.collated[cols CATALOG_NAME,LABEL,POSNTYPE,RA,DEC,ERR_RA,ERR_DEC]" temp.collated clob+'
    position_table = mrdfits('temp.collated',1, theader, /SILENT)

    failure = (n_elements(position_table) NE num_sources) || $
               ~array_equal(CATALOG_NAME, strtrim(position_table.CATALOG_NAME))
                                                                          
    if failure then begin
      print, 'ERROR!  Pbslice.sav and most_valid_merge.collated were made with different source lists!'
      retall
    endif

    ; Build a single table suitable for releasing the catalog.
    f_nan = !VALUES.F_NAN
    validation_template = $
      { ERR_POS          : f_nan,$
        Pb_min           : f_nan,$
        Pb_min_MERGE_NAME: ''   ,$
        Pb_min_band      : ''   ,$
        Pb_min_SRC_CNTS  : -1L  ,$

        Pb_min_full_band     : f_nan,$
        Pb_min_soft_band     : f_nan,$
        Pb_min_hard_band     : f_nan,$
        Pb_min_vhard_band    : f_nan, $
        
        is_occasional               :0B,$
        tally_singleObsID           :0 ,$
        tally_singleObsID_validation:0 ,$
        list_singleObsID_validation :''}

    cat = replicate(create_struct(position_table[0], validation_template), num_sources)

    struct_assign, position_table, cat
    cat.ERR_POS           = sqrt( cat.ERR_RA^2 + cat.ERR_DEC^2 )
    
    cat.Pb_min            =              validation_report.best_phot.Pb
    cat.Pb_min_MERGE_NAME =              validation_report.best_phot.MERGE_NAME
    cat.Pb_min_band       = (bands.name)[validation_report.best_phot.BAND_INDEX]
    cat.Pb_min_SRC_CNTS   =              validation_report.best_phot.SRC_CNTS

    cat.Pb_min_full_band  =              validation_report.Pb_min_full_band 
    cat.Pb_min_soft_band  =              validation_report.Pb_min_soft_band 
    cat.Pb_min_hard_band  =              validation_report.Pb_min_hard_band 
    cat.Pb_min_vhard_band =              validation_report.Pb_min_vhard_band
    
    cat.is_occasional                =   validation_report.is_occasional
    cat.tally_singleObsID            =   validation_report.tally_singleObsID           
    cat.tally_singleObsID_validation =   validation_report.tally_singleObsID_validation
    cat.list_singleObsID_validation  =   validation_report.list_singleObsID_validation 

    prefix = 'ERROR: null/zero values found in column '
    if ~array_equal(finite(cat.RA     ) AND (cat.RA      NE 0),1) then message, prefix+'RA'
    if ~array_equal(finite(cat.DEC    ) AND (cat.DEC     NE 0),1) then message, prefix+'DEC'
    if ~array_equal(finite(cat.ERR_RA ) AND (cat.ERR_RA  NE 0),1) then message, prefix+'ERR_RA'
    if ~array_equal(finite(cat.ERR_DEC) AND (cat.ERR_DEC NE 0),1) then message, prefix+'ERR_DEC'
    if ~array_equal(finite(cat.ERR_POS) AND (cat.ERR_POS NE 0),1) then message, prefix+'ERR_POS'
    if ~array_equal(finite(cat.PB_MIN ),1) then message, 'ERROR: null values found in column PB_MIN'
                                 
    cat_fn = 'catalog.fits'
    mwrfits, cat, cat_fn, theader, /CREATE
    print, 'Wrote '+cat_fn

    ; Make some plots to summarize the catalog.
    ind = where(/NULL,  validation_report.is_occasional)
    if isa(ind, /INTEGER) then begin
      this_Pb       = (validation_report.best_phot.Pb      )[ind] > 1e-10
      this_NET_CNTS = (validation_report.best_phot.NET_CNTS)[ind]
      dataset_1d, id3, DATASET='occasional', BINSIZE=0.1, alog10(this_Pb)
      dataset_1d, id2, DATASET='occasional', BINSIZE=1.0, this_NET_CNTS
      dataset_2d, id1, DATASET='occasional', this_NET_CNTS, alog10(this_Pb), PSYM=3
      endif
      
    ind = where(/NULL, ~validation_report.is_occasional)
    if isa(ind, /INTEGER) then begin
      this_Pb       = (validation_report.best_phot.Pb      )[ind] > 1e-10
      this_NET_CNTS = (validation_report.best_phot.NET_CNTS)[ind]
      dataset_1d, id3, DATASET='not occasional', BINSIZE=0.1, alog10(this_Pb), XTIT='log Pb_min'
      dataset_1d, id2, DATASET='not occasional', BINSIZE=1.0, this_NET_CNTS  , XTIT='NET_CNTS corresponding to Pb_min'
      dataset_2d, id1, DATASET='not occasional', this_NET_CNTS, alog10(this_Pb), PSYM=3, YTIT='log Pb_min', XTIT='NET_CNTS corresponding to Pb_min'
    endif
    end

exit, /NO_CONFIRM

    
    

Check for error messages.

  printf "\n\n"
  egrep -i "WARNING|ERROR|DEBUG|halted|Stop|IMMORTAL" validation_data_products.log | egrep -v "No HDUNAME|different value|cannot intersect|finite width|band/ not found|not match the OBJECT keyword|no in-band|excessive OVERLAP|extractions outside|arithmetic error|ERR_DATA|^-"


Make important region files.

  set color_change='-e /PILED/s/DodgerBlue/magenta/ -e /occasional/s/DodgerBlue/blue/'
  grep "{cat}"      all.reg | sed $color_change  -e '/occasional/s/cross/diamond/' > xray_positions.reg
  grep polygon      all.reg | sed $color_change                                    > polygons.reg
  grep "{fraction}" all.reg | sed $color_change                                    > fractions.reg
  cat xray_positions.reg polygons.reg fractions.reg tables/aimpoints.reg           > review.reg
  foreach obs ($OBS_LIST)
    grep EPOCH_${obs} occasional.reg > occasional.${obs}.reg
  end
  

VERIFY THAT every source has a "most_valid" merge, i.e. minimum value of MERGNUM is not zero: 

  dmstat "tables/most_valid_merge.collated[cols MERGNUM]" | egrep 'min|null'

VERIFY THAT catalog.fits has NULL values only in the PB_SOFT_BAND, PB_HARD_BAND, PB_VHARD_BAND columns:

  dmstat "catalog.fits[cols -CATALOG_NAME,-LABEL,-POSNTYPE,-Pb_min_MERGE_NAME,-Pb_min_band,-IS_OCCASIONAL,-LIST_SINGLEOBSID_VALIDATION]" | egrep '^[A-Z]|null'



=================================================================================================
CLOSE OUT THIS PASS, ARCHIVING IMPORTANT FILES.  
=================================================================================================
Various symlinks point to these final products, so we use a standard name ("pass_final_catalog") for this final pass.

From screen window #0 (named $TARGET), run:

  setenv PASS `readlink CURRENT_PASS`
  rsync -a ./{*.txt} tables/*.collated removed_sources.* ../checkpoint_for_backup/
  (mv by-eye.txt add.txt move.txt delete.txt              $PASS |& cat > /dev/null )
  rsync -a --info=NAME ./{*.txt} --exclude=block.txt  $PASS  

  gzip *.log
  mv `find *.srclist \! -name 'all*.srclist'`  *.log *.log.gz *.reg *.sav *.ps screenrc.* *.collated tables/*.collated  $PASS  
  mv catalog.fits $PASS

  set filelist=`find $PASS -name modified_aperture.srclist`
  if ( "$filelist" != '' ) rsync -a --info=NAME $filelist  .

  printf "\n%s --> pass_final_catalog\n\n" $PASS
  mv    $PASS pass_final_catalog
  setenv PASS 'phot'
  'rm' CURRENT_PASS; ln -s $PASS CURRENT_PASS

  ln -s        pass_final_catalog/*.reg .
  chmod -R a-w pass_final_catalog
  
  find ../../pointing_*/obsid_*/instmap -name "*.instmap" -o -name "*.timemap" | egrep -v 'full|1000eV' |xargs -p rm


Above, we discard .timemap files because we won't need them again.
We discard .instmap files because the photometry procedure is going to recompute them and it's pointless to copy them to the archive disk (below).
I cannot recall why we retain the 'full|1000eV' instmaps ...




(Pat and Leisa only) Copy vital files to archive disk so they will be backed up.
-------------------------------------------------------
ON THE ARCHIVE MACHINE, run:
  setenv TARGET ....
  idl
    relocate_target, /CHECKIN, /BACKUP, getenv("TARGET"), ['data/extract/checkpoint_for_backup/','data/extract/point_sources.noindex/pass*','data/pointing*'], ARCHIVE_VOLUME='hiawatha2', FAST_VOLUME='osceola_fast'

  exit   

-------------------------------------------------------
Write-protect the validation and "position" merges that we have created so far in the source directories so they won't be mistakenly overwritten later:

  find [0-9]* -maxdepth 1 -type d  \( -name theta_00-05 -or -name trash \)  -print0 -prune  | xargs -0 -n1000 rm -rv

  find [0-9]* -maxdepth 1 -type d  \( -name theta_00-03 -or -name theta_02-06 -or -name theta_05-09 -or -name theta_08-14 -or -name theta_13-99 -or -name most_valid -or -name position \) -print0 -prune  | xargs -0  chmod a-w

**** We may consider just deleting those merges, to save disk space!










---------------------------------------------------------------------
(optional) Review "occasional sources"---those that fail to validate in all multi-ObsID merges tried, but do validate in one or more single-ObsID merges.  These are reviewed in Pb order (most-valid sources first).

THE "COMMENT" SHOWN AFTER THE SOURCE NAME REPORTS the ObsIDs in which each occasional source was valid.

  idl
   acis_extract, COLLATED_FILENAME='tables/most_valid_merge.collated', SRCLIST_FILENAME='occasional.srclist', /SHOW_REGIONS, /INCLUDE_PRUNED_OBSIDS, /OMIT_BKG_REGIONS

AND/OR

  foreach obs ($OBS_LIST)
    ds9 -title ${obs}  ../obs${obs}/validation.evt -region  occasional.${obs}.reg -region xray_positions.reg&
  end



---------------------------------------------------------------------
(optional) Review each source in the energy band that produced its best Pb value.

We do not currently have a tool that displays each source on whatever combination of ObsIDs and energy band produced Pb_min for that source.

The target-level event lists used in the visualizations below are only an approximation of the ObsID combinations
used to validate individual sources (recorded in the "most valid merge XXXX" ds9 tags).


  ds9 -title "full " ../target.evt       -region Pb_min_full_band.reg  &
  ds9 -title "hard " ../target.hard.evt  -region Pb_min_hard_band.reg  &
  ds9 -title "soft " ../target.soft.evt  -region Pb_min_soft_band.reg  &
  ds9 -title "vhard" ../target.vhard.evt -region Pb_min_vhard_band.reg &

OR

  ds9  ../target.evt -region Pb_min_full_band.reg   ../target.hard.evt  -region Pb_min_hard_band.reg   ../target.soft.evt  -region Pb_min_soft_band.reg    ../target.vhard.evt -region Pb_min_vhard_band.reg &

OR

Worst Pb slice (yellow):

    acis_extract, SRCLIST_FILENAME='Pbslice_0.6-1.0.srclist', /SHOW_REGIONS, MERGE_NAME='most_valid', COLLATED_FILENAME='tables/most_valid_merge.collated',  DISPLAY=['../target.evt','../target.hard.evt','../target.soft.evt','../target.vhard.evt'], REGION_FILE=['Pb_min_full_band.reg','Pb_min_hard_band.reg','Pb_min_soft_band.reg','Pb_min_vhard_band.reg'], DS9_OPTION_STRING='-linear -scale limits 0 10'


    acis_extract, SRCLIST_FILENAME='Pbslice_0.3-0.6.srclist', /SHOW_REGIONS, MERGE_NAME='most_valid', COLLATED_FILENAME='tables/most_valid_merge.collated',  DISPLAY=['../target.evt','../target.hard.evt','../target.soft.evt','../target.vhard.evt'], REGION_FILE=['Pb_min_full_band.reg','Pb_min_hard_band.reg','Pb_min_soft_band.reg','Pb_min_vhard_band.reg'], DS9_OPTION_STRING='-linear -scale limits 0 10'


    acis_extract, SRCLIST_FILENAME='Pbslice_0.1-0.3.srclist', /SHOW_REGIONS, MERGE_NAME='most_valid', COLLATED_FILENAME='tables/most_valid_merge.collated',  DISPLAY=['../target.evt','../target.hard.evt','../target.soft.evt','../target.vhard.evt'], REGION_FILE=['Pb_min_full_band.reg','Pb_min_hard_band.reg','Pb_min_soft_band.reg','Pb_min_vhard_band.reg'], DS9_OPTION_STRING='-linear -scale limits 0 10'

Best Pb slice (blue):

    acis_extract, SRCLIST_FILENAME='Pbslice_0-0.1.srclist', /SHOW_REGIONS, MERGE_NAME='most_valid', COLLATED_FILENAME='tables/most_valid_merge.collated',  DISPLAY=['../target.evt','../target.hard.evt','../target.soft.evt','../target.vhard.evt'], REGION_FILE=['Pb_min_full_band.reg','Pb_min_hard_band.reg','Pb_min_soft_band.reg','Pb_min_vhard_band.reg'], DS9_OPTION_STRING='-linear -scale limits 0 10'








---------------------------------------------------------------------

If you want to release a tarball to collaborators, the following tar command gathers a set of basic data products available at this point:

  tar -cvzHf /tmp/${TARGET}_`date '+%Y%b%d'`.tar.gz  -C pass_final_catalog/ label.txt catalog.fits review.reg xray_positions.reg polygons.reg Pbslice.reg   -C ../.. target.evt target.emap


{The columns in catalog.fits can be seen with dmlist:
  dmlist catalog.fits cols

--------------------------------------------------------------------------------
Columns for Table Block EXTRACTION RESULTS
--------------------------------------------------------------------------------

ColNo  Name                 Unit   
   1   CATALOG_NAME                  source name in catalog
   2   LABEL                         source label
   3   POSNTYPE                      type of source position
   4   RA                   deg      source position
   5   DEC                  deg      source position
   6   ERR_RA               arcsec   1-sigma uncertainty around (RA,DEC)
   7   ERR_DEC              arcsec   1-sigma uncertainty around (RA,DEC)
   8   ERR_POS                     

   9   PB_MIN                      
  10   PB_MIN_MERGE_NAME           
  11   PB_MIN_BAND                 
  12   PB_MIN_SRC_CNTS             

  13   PB_MIN_FULL_BAND            
  14   PB_MIN_SOFT_BAND            
  15   PB_MIN_HARD_BAND            
  16   PB_MIN_VHARD_BAND           

  17   IS_OCCASIONAL               
  18   TALLY_SINGLEOBSID           
  19   TALLY_SINGLEOBSID_VALIDATION
  20   LIST_SINGLEOBSID_VALIDATION 


These are:
  1   Source name (built from sexagesimal J2000 coordinates)
  2   Source label (a unique source identifier built from the pointing number and a running sequence number)
  3   Origin of position (image reconstruction, mean data within the aperture, or PSF correlation)
  4,5 Source position in J2000 decimal RA and Dec
  6,7 RA and Dec position error, in arcsec
  8   Position error as radius of "error circle", in arcsec
  
  9   Smallest detection p-value found across all merges considered and all energy bands considered.
 10   Name of the merge that produced that p-value.
 11   Energy band that produced that p-value.
 12   Gross extracted counts in the merge and energy band that produced that p-value.
  
 13-16 Smallest detection p-value found across all merges considered (Full, Soft, Hard, and VeryHand bands).
  
 17   IS_OCCASIONAL is "true" (1) if source was NOT validated in any multi-ObsID merge considered.
 18   Number of single-ObsID merges in which Pb was calculated.
 19   Number of single ObsID merges in which the source was valid.
 20   List of single ObsIDs in which the source was valid.
}


---------------------------------------------------------------------
At this point the symlinks you created in the SET UP section should allow the useful region files review.reg, polygons.reg, xray_positions.reg, and all.reg to be accessed from the following directories:

   <target>/data/
   <target>/data/extract/
   <target>/data/counterparts/





=============================================================================
Moving On
=============================================================================
The next phase of your data reduction should be to EXECUTE THE PROCEDURE IN PHOTOMETRY_PROCEDURE.TXT.

If you're doing that now, then retain the screen session you've been using here.  

Otherwise, terminate the screen session:

1. In screen window #0 run the following to paste the 'exit' command in every window, which will cleanly terminate ssh connections you have to other computers (avoiding zombie processes):.
  screen -S $SCREEN_NAME -X at \# stuff 'exit^M'

2. If that doesn't end the screen session, then type the screen command "^a\".


If you forgot to "exit" the ssh connections as instructed above, and instead simply killed the screen session with ^a\, the remote shells (which are running "caffeinate" to keep the machine from sleeping) will NOT DIE.  In that case then, the easiest way to kill them (and recover from your mistake) is to ssh to the machine again and run
  killall caffeinate




#################################################################################
#################### THINK CAREFULLY ABOUT YOUR BACKUP SYSTEMS #################### 
#################################################################################
During this validation procedure, you may have chosen to exclude this AE extraction tree from your backup systems (e.g. TimeMachine, CrashPlan, rsync, etc.), to avoid creating incremental backups of AE extractions that have very little long-term value.  If you are proceeding to the photometry procedure, then we recommend leaving this AE extraction tree  excluded from backups until after the catalog is re-extracted in that procedure.

However, if you are stopping here for a while, then you may wish to re-enable backup of this extraction tree.




#################################################################################
NOTES ON OPTIONAL ANALYSIS STEPS

The material below is a collection of notes on analysis tasks that occassionally may be needed, not a procedure that should be strictly followed for all targets.
#################################################################################

Build a target-level event list from a subset of ObsIDs.
========================================================
For some visual reviews it is sometimes helpful to build some target-level event lists from certain subsets of the ObsIDs, e.g. a group of ObsIDs that share a similar aimpoint.

Below is example code for that task.  In this example, the ObsIDs are explicitly listed (rather than selecting by a wildcard pattern); the event list produced is named 'w43main.evt'.

  cd <target>/data/

  idl -queue |& tee -a W43main_mosaic.log  
    .run ae
    .run
    L2_pattern = 'pointing*/{obsid_9612,obsid_17716,obsid_17717,obsid_18867,obsid_18868,obsid_18869,obsid_18870,obsid_18887,obsid_18888}/AE/validation.AI.evt'

    build_scene, 'fullfield', $
                 PATTERN=L2_pattern, EMAP_BASENAME='', $
                 MERGED_EVENTFILE='w43main.evt', MERGED_COLUMNLIST='sky,energy', MERGED_FILTERSPEC="energy=500:7000", $
                 TEMPLATE_FN='fullfield_template.img'

    run_command, "ciao;dmcopy 'w43main.evt[energy<2000]' w43main.soft.evt clob+"
    run_command, "ciao;dmcopy 'w43main.evt[energy>2000]' w43main.hard.evt clob+"
    run_command, "ciao;dmcopy 'w43main.evt[energy>4000]' w43main.vhard.evt clob+"

    exit, /NO_CONFIRM
    end



#################################################################################
######################   APPENDIX: THE "SCREEN" UTILITY    ###################### 
#################################################################################
This appendix discusses what you should do if the screen session you built at the top of this procedure does not have the correct number of screen windows.

The first thing to do is to try to manually create another screen window in your existing session, by typing ^a^c.  If you see the message "No more windows", then you've run into a limit compiled into your version of "screen".  For example, the version that comes in OS-X (/usr/bin/screen) has a limit of 40 screen windows.  Luckily, the version available via macports has a higher limit.  Follow the instructions below to install it.


=====================================================================
If you see the message "No more PTYs", then you have run out of "pseudo terminals" in the OS kernel.  On a Mac (at least) you can check the total supply of "pseudo terminals" with the following command
  sysctl -a | grep tty
You can increase the supply with the following command
  sudo sysctl -w kern.tty.ptmx_max=255

=====================================================================
If you start to see failures with the message "no more processes", then you've run into a resource limit in your Unix kernel.  On a Mac (at least) you can increase the allowed total number of processes, and the allowed number of processes per user like this:

  sudo sysctl -w kern.maxproc=2500
  sudo sysctl -w kern.maxprocperuid=2000

To execute such commands (under OS-X) at startup, do the following.

cat > /tmp/sysctl.plist << \STOP
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>sysctl</string>
 <key>ProgramArguments</key>
 <array>
 <string>/usr/sbin/sysctl</string>
 <string>kern.tty.ptmx_max=255</string> 
 <string>kern.maxproc=2500</string>
 <string>kern.maxprocperuid=2000</string>
 </array>
 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>
\STOP

sudo cp /tmp/sysctl.plist /Library/LaunchDaemons/
sudo chown root:wheel     /Library/LaunchDaemons/sysctl.plist
sudo launchctl    load    /Library/LaunchDaemons/sysctl.plist



Note that shells provide various limits on resources consumed by that shell or its children.  The three commands below show (or change) limits on process resources.  It's not clear if they are all controlling the same limit, or if some are a per-shell limit and some are a per-user limit.  

  limit
  ulimit -a 
  launchctl limit maxproc 

The per-shell limit may be important, because all windows in a screen session are children of the shell that created the session.  Try putting "limit" or "ulimit" commands in .cshrc file.

The only way I know to count processes is

  'ps' -A | wc -l


=====================================================================
Instructions for installing the "screen" utility via macports.

For large projects we need to install the "screen" source code via macports; it has a higher limit.
  sudo port -s install screen
Run "rehash" in the shell where you need to launch screen.


If the the window limit in the macports version is not large enough, then you'll have to change the parameter MAXWIN in the source code for the "screen" package, and then recompile it. 


First, install the "screen" package from source code:

  sudo port uninstall screen
  sudo port -s -v install screen
  sudo port     uninstall screen


Look for the "screen" package's source code files.  They should be found in /opt/local/var/macports/distfiles/screen/.  If you find only a tarball there, then untar it.  

  cd /opt/local/var/macports/distfiles/screen/
  (if necessary)  sudo tar -xzvf screen-*.tar.gz  

In the screen-X.X.X/ directory created by tar, preserve the published version of config.h.in

   cd screen-*/
   sudo chmod 777 .
   rsync -av config.h.in config.h.in.orig
   mkdir /tmp/patch/
   rsync -av config.h.in config.h.in.orig /tmp/patch/



We need to patch the file /opt/local/var/macports/distfiles/screen/screen-4.8.0/config.h.in
How we do that depends on whether Macports already has a patch for that file.

  cd /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/screen/files/
  grep config.h.in patch*
  

---------------------------------------------------------------------

IF THERE IS ALREADY A PATCH (probably named patch-config.h.in.diff) FOR config.h.in THEN WE MUST REVISE IT TO ADD OUR CHANGE.

  sudo chmod 777 .
  mv patch-config.h.in.diff  patch-config.h.in.diff.original
  rsync -av patch-config.h.in.diff.original  /tmp/patch/
  
  pushd /tmp/patch/
  /usr/bin/patch -p0 < patch-config.h.in.diff.original
  vi config.h.in
  Edit the line "define MAXWIN" to allow more windows.  DO NOT REMOVE THE "#" character from that line!

  diff -rU 3 config.h.in.orig  config.h.in  > /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/screen/files/patch-config.h.in.diff

  Re-install the "screen" package from source code:
    cd ~/work/
    sudo port -s -v install screen |& tee screen.log
    grep patch screen.log



---------------------------------------------------------------------

IF THERE IS NOT A PATCH FOR config.h.in THEN WE MUST CREATE ONE  
  
In the screen-X.X.X/ directory created by tar, preserve config.h.in and edit config.h.in.
   cd screen-*/
   sudo cp config.h.in config.h.in.orig
   sudo vi config.h.in
Edit the line "define MAXWIN" to allow more windows.  DO NOT REMOVE THE "#" character from that line!


Create a patch for config.h.in, in the directory that holds other patches for screen:

   sudo diff -rU 3 config.h.in.orig  config.h.in  > /tmp/patch-config-MAXWIN.diff
   
   sudo rsync -av /tmp/patch-config-MAXWIN.diff /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/screen/files/
   
   cat /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/screen/files/patch-config-MAXWIN.diff


Edit the  portfile for screen to declare the new patch.

  sudo vi /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/sysutils/screen/Portfile
  # Look for an existing "patchfiles" declaration, and add 
  patch-config-MAXWIN.diff
  # If no existing "patchfiles" declaration, then add the line:
  patchfiles patch-config-MAXWIN.diff 


Re-install the "screen" package from source code:
  cd ~/work/
  sudo port -s -v install screen |& tee screen.log
  grep patch screen.log




----
Note that the MacPorts Guide (https://guide.macports.org S4.5.2 and S4.6) and the discussion at http://simx.me/technonova/tips/using_experimental_macports_portfiles.html indicate that we should be able to do this patching work in a "local MacPorts repository", such as /Volumes/cochise1/MACPORTS/, that takes precedence over the published repository.  Presumably such a local patch would survive when new versions of screen are released.  

I believe that the steps shown below achieve that local repository build, but I don't know yet what happens when a new version of "screen" is released by MacPorts.org. 

Create local repository:

  mkdir /Volumes/cochise1/MACPORTS/
  Add to /opt/local/etc/macports/sources.conf (BEFORE the "rsync:" line) the following line:
    file:///Volumes/cochise1/MACPORTS


Copy the "screen" portfile directory (containing the file "Portfile" and directory "files") into your local portfile repository, inside a directory that reflects the port's primary "category", which is shown in the "categories" line of the Portfile.  The category for "screen" is "sysutils".

  mkdir /Volumes/cochise1/MACPORTS/sysutils/

  rsync -a --info=NAME --delete /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/sysutils/screen /Volumes/cochise1/MACPORTS/sysutils/


Index the local repository.

  portindex /Volumes/cochise1/MACPORTS


Re-install the "screen" package from source code:

  sudo port     uninstall screen
  sudo port -s -v install screen | tee screen.log
  grep patch screen.log





#################################################################################
######################   APPENDIX: MODELING STRUCTURED BACKGROUND COMPONENTS    ###################### 
#################################################################################
When the background (e.g. diffuse emission) has strong gradients across the symmetric background regions built by AE, background estimates (and thus Pb values) can be biased.  This problem can be reduced if a model of that structured background can be constructed.

This is inherently an iterative algorithm.
  * Modeling diffuse background requires diffuse images.
  * Diffuse images require point source masking.
  * Maskings require a catalog.
  * A catalog requires aperture backgrounds.
  * Aperture backgrounds are improved by a model of diffuse background.


========================================================
MASK POINT SOURCES
Similar to diffuse_procedure.txt
========================================================
This needs to be run inside the validation screen session, window0.
From <target>/data/extract/point_sources.noindex/, run:

cat > block.txt << \STOP
   
   nice idl -queue |& tee -a ae_masking.${OBS}.log 
     file_delete, "../obs"+getenv("OBS")+'/ae_finished', /ALLOW_NONEXISTENT 
     file_delete, "waiting_for_"+getenv("OBS")         , /ALLOW_NONEXISTENT 
\STOP
  screen -S $SCREEN_NAME                -X readreg A block.txt

cat > block.txt << \STOP
    .run ae 
    .run 
    obsdir = "../obs"+getenv("OBS")+"/" 
    semaphore = wait_for_cpu(LOW_PRIORITY=strmatch(obsdir,'*_BI'))

    ; We cannot re-use models in ae_better_masking.sav because the dimensions of the 
    ; emap may have changed.
    ae_better_masking, getenv("OBS"), EVTFILE_BASENAME="validation.evt", REUSE_MODELS=0,  /SKIP_EXTRACT_BACKGROUNDS, /SKIP_RESIDUALS, PLOT=0, EXTRA_MASKFILE='', GENERIC_RMF_FN="generic.rmf"

    file_move, obsdir+"ae_lock", obsdir+"ae_finished", /OVERWRITE 
    exit, /NO_CONFIRM 
    end 
\STOP
  screen -S $SCREEN_NAME                -X readreg B block.txt


  foreach obs ($OBS_LIST)
    set marker=waiting_for_${obs}; touch ${marker}
    screen -S $SCREEN_NAME -p obs${obs} -X   paste A
    sleep 1
    while ( -e ${marker} )
      echo "Launching IDL ..."
      sleep 1
    end
    printf "\nLaunching ObsID ${obs}\n"
    screen -S $SCREEN_NAME -p obs${obs} -X   paste B
    sleep 2
  end

***Wait for this to finish in all ObsIDs before moving on!***
You have to go check each ObsID to make sure they are all done!  Don't stay in window0 -- you will be confused because it's not doing anything.



=====================================================================
SETUP SYMLINKS TO DATA PRODUCTS FOR DIFFUSE ANALYSIS
=====================================================================
***Do not start this until the point source masking above has finished in all ObsIDs!***

For the following, you can move up a directory in one of your screen windows, or run it in a new shell (as long as $OBS_LIST is defined correctly there).

From <target>/data/extract/, run:

 foreach obs ($OBS_LIST)
 echo ${obs} 
 pushd obs${obs}
    'rm' -f diffuse.emap
    'rm' -f diffuse.evt
    if (`dmkeypar validation.evt GRATING echo=yes` == 'NONE') ln -sv background.emap diffuse.emap
    if (`dmkeypar validation.evt GRATING echo=yes` == 'NONE') ln -sv background.evt diffuse.evt
 popd
 end

 idl -queue 
    .run ae
    .run
    run_command, file_which('setup_AE_for_obsid_data.csh')+' "../pointing_*/obsid_*" AE'
    exit, /NO_CONFIRM
    end
  


========================================================
SMOOTH THE DIFFUSE DATA IN FULL-BAND ONLY
Similar to diffuse_procedure.txt
========================================================
From <target>/data/extract/, run:

  mkdir adaptive_smoothing 
  cd    adaptive_smoothing
  ~/bin/acis_lkt4 .
  ln -s  ../../tangentplane_reference.evt .

  idl -queue |& tee -a build_images.log
    file_delete, /ALLOW_NONEXISTENT, 'build_composite.sav'
    .run
    obs_pattern  = '../obs[0-9]*'
    scene_name = 'fullfield'

    template_row =  {image_spec, name:'', emap_basename:'', energy_filter:''}
    image_spec   = [{image_spec,  'full_band'    , 'full'    ,  '500:7000' }]

    image_spec.emap_basename =   'obs.'+image_spec.emap_basename+'.emap'
    image_spec.energy_filter ='energy='+image_spec.energy_filter

    obs_event_fn     = file_search(obs_pattern+'/validation.evt', COUNT=num_obs)
    if (num_obs EQ 0) then begin
      print, 'ERROR: no validation.evt files found.'
      retall
    endif

    obsdir           = file_dirname(obs_event_fn)
    diffuse_event_fn = obsdir+'/diffuse.evt'
    stowed_event_fn  = obsdir+'/diffuse.bkg.evt'

    good_ind = where(file_test(diffuse_event_fn) AND $
                     file_test(stowed_event_fn), good_count, COMPLEMENT=bad_ind, NCOMPLEMENT=bad_count)

    if (bad_count GT 0) then begin
      print, 'ERROR: The following ObsIDs are missing the files diffuse.evt and/or diffuse.bkg.evt:'
      forprint, obsdir, SUBSET=bad_ind
      print, 'Exit IDL or type .c to ignore those ObsIDs and continue.'
      stop
      obs_event_fn     = obs_event_fn    [good_ind]
      diffuse_event_fn = diffuse_event_fn[good_ind]
      stowed_event_fn  = stowed_event_fn [good_ind]
    endif

    ; Build a scene mask (EMAP_BASENAME supplied) from unmasked exposure maps.
    build_scene, OBS_EVENT_FN=obs_event_fn, EMAP_BASENAME='obs.emap', scene_name, /CREATE_TEMPLATE, /CREATE_IMAGE, IMAGE_FILTER_SPEC='energy=500:7000', IMAGE_NAME='full_band', DESIRED_IMAGE_PIXELS=1000L^2

    file_copy, 'fullfield.mask', 'nominal_fullfield.mask', /OVERWRITE

    extra_maskfile = '../diffuse_sources.noindex/extra_maskfile.reg'

    if ~file_test(extra_maskfile) then $
      extra_maskfile = DIALOG_PICKFILE(TITLE='Select the extra_maskfile.reg that was given to ae_better_masking.', FILTER='*.reg')

    if logical_true(extra_maskfile) && logical_true( file_lines(extra_maskfile) ) then run_command, string(extra_maskfile, $
        F="(%'dmcopy ""nominal_fullfield.mask[exclude sky=region(%s)]"" fullfield.mask clob+')")

    build_scene, OBS_EVENT_FN=diffuse_event_fn, EMAP_BASENAME=image_spec.emap_basename, MASK_BASENAME='background.emap', RESOLUTION=4, scene_name, SUFFIX='.diffuse', /CREATE_IMAGE, /SHOW, IMAGE_FILTER_SPEC=image_spec.energy_filter, IMAGE_NAME=image_spec.name

    build_scene, OBS_EVENT_FN=stowed_event_fn, EMAP_BASENAME='diffuse.bkg.scaling', RESOLUTION=4, scene_name, SUFFIX='.bkg',  /CREATE_IMAGE, /SHOW, IMAGE_FILTER_SPEC=image_spec.energy_filter, IMAGE_NAME=image_spec.name, /SUM_RATES_NOT_COUNTS

    tara_smooth, scene_name, 15, /TOPHAT, /DISCARD_EXISTING, BAND_NAME='full_band',$
                 MASK_FILENAME='nominal_fullfield.mask', RUN_NAME='hole_free'
    exit, /NO_CONFIRM
    end

One ds9 session spawned by the runs above shows the diffuse emap and the diffuse images in each of the energy bands (listed in structure array "image_spec" earlier).  A second ds9 session shows the corresponding stowed images in each energy band.  These visualizations are simply sanity checks, for you to confirm that the data are masked and look reasonable.

{
It is clear that discontinuities in the diffuse model push ABB to bkgd regions that stay on one side of the boundary.
We need a diffuse model that smooths over all the masks, no matter how large they are.
So, above we are building a hole-free model of diffuse emission (full-band, SNR=15).
}





========================================================
BUILD or REBUILD THE DIFFUSE MODEL IMAGE ABB_background_model.img FOR EACH OBSID
========================================================
From screen window #0 (in data/extract/point_sources.noindex/) run:

cat > block.txt << \STOP
  nice idl -queue |& tee -a ae_validation.${OBS}.log 
    file_delete,    "waiting_for_"+getenv("OBS"), /ALLOW_NONEXISTENT
    .run ae
    .run
    par = ae_get_target_parameters()
    obsdir = "../obs"+getenv("OBS")+"/" 

    file_delete, obsdir+"ae_finished", /ALLOW_NONEXISTENT 
    semaphore = wait_for_cpu(LOW_PRIORITY=strmatch(obsdir,'*_BI'))

    tempdir = temporary_directory( 't-rex.', VERBOSE=0, SESSION_NAME=session_name)
    run_command, PARAM_DIR=tempdir

    emapfile               = obsdir + 'obs.emap'
    diffuse_model_filename = '../adaptive_smoothing/full_band/hole_free/sig015/tophat/fullfield.diffuse_filled.flux'

    wait_until_files_exist, 60, diffuse_model_filename

    reprojected_diffuse_model_filename = tempdir + getenv("OBS")+"_diffuse_model.img"
    background_model_filename          =  obsdir +         "ABB_background_model.img" 

    ; The smoothed diffuse image we are using has instrumental background removed and has been
    ; normalized by a 1 keV exposure map, whose units are s cm^2 ct /photon.

    ; The diffuse image is a "photon surface brightness" map with units of photon /cm**2 /s /arcsec**2 

    ; First, we reproject that surface brightness map onto the pixel grid of this ObsID's exposure map.
    ; Since surface brightness is a (2-D) function (not a tally of something), we use method=average.
    ; Regridding does not change the units.
    resolution = 1
    run_command, string(diffuse_model_filename, emapfile, reprojected_diffuse_model_filename, resolution,$
                         F="(%'reproject_image  infile=""%s""  matchfile=%s  outfile=%s  method=average resolution=%d clob+')")

    ; Second, we multiply the reprojected surface brightness map by this ObsID's emap. 
    ; Units are changed from  photon /cm**2 /s /arcsec**2  to  ct /arcsec**2
    term1 = readfits(reprojected_diffuse_model_filename)
    term2 = readfits(emapfile, emap_header)
    if ~array_equal( size(/DIM,term1), size(/DIM,term2) ) then message, 'BUG!'
    count_density_map = term1 * term2

    ; Third, we must scale the count_density_map to a prediction of the *number of counts*
    ; expected in each pixel of this ObsID's emap.
    ; That is a change of units from  ct /arcsec**2  to  ct /ImagePixel^2.
    ; The inverse of this scaling was done in tara_smooth.pro.
    extast, emap_header, img2wcs_astr
    arcsec_per_imgpixel = abs(img2wcs_astr.CDELT[0] * img2wcs_astr.CD[0,0])*3600.0  

    count_map = count_density_map * arcsec_per_imgpixel^2
    writefits, background_model_filename, count_map, emap_header

    cmd = string('Diffuse Model ('+getenv("OBS")+')', background_model_filename, F='(%"ds9 -title \"%s\" -linear %s -zoom to fit >& /dev/null &")')
    run_command, /QUIET, cmd

    print, total(/DOUBLE, count_map), getenv("OBS"), F='(%"\nThe model predicts %d diffuse counts in ObsID %s.")'
    exit, /NO_CONFIRM
  end
\STOP
  screen -S $SCREEN_NAME                -X readreg A block.txt


Then also in window #0:

  foreach obs ($OBS_LIST)
    set marker=waiting_for_${obs}; touch ${marker}
    screen -S $SCREEN_NAME -p obs${obs} -X   paste A
    sleep 1
    while ( -e ${marker} )
      echo "Launching IDL for ObsID ${obs}..."
      sleep 1
    end
    sleep 2
  end

Go through the ObsID windows to make sure everything finishes correctly.  A ds9 session will be generated for each screen, showing the new backgrounds.


========================================================
USE EACH OBSID'S ABB_background_model.img IN FUTURE RUNS OF ae_better_backgrounds TOOL 
========================================================
Create/edit the ASCII file named target_parameters.par and insert the following line:

  BACKGROUND_MODEL_FILENAME = 'ABB_background_model.img'

Proceed with normal pruning passes.  The ABB_background_model.img models should automatically be passed to the ae_better_backgrounds tool.





#################################################################################
######################   APPENDIX: OTHER NOTES    ###################### 
#################################################################################
  
========================================================
(NOT VERY USEFUL!)
Examine the correlation between Pb computed by recon_detect and Pb in validated catalog.

  idl
    .run
    candidate = mrdfits('../../recon_detect/proposed.target.fits',1)
    validated = mrdfits('tables/xray_properties.fits',1)

    candidate.label    = strtrim(candidate.label,2)
    validated.LABEL    = strtrim(validated.LABEL,2)

    num_candidates = n_elements(candidate)
                                                                 
    temp = replicate(create_struct(candidate[0], 'Pb_recon',0.0,  'Pb_ae',!VALUES.F_NAN, 'offset',0.0), num_candidates)
    struct_assign, candidate, temp, /NOZERO
    candidate = temp

    candidate.Pb_recon = candidate.PB
    candidate.Pb_ae    = 0.02 + 0.002*random(num_candidates)
    
    
    for ii=0,num_candidates-1 do begin
      ind = where(/NULL, validated.LABEL EQ candidate[ii].label )
      if isa(ind, /INTEGER) then begin
        candidate[ii].Pb_ae = validated[ind].PROBNOSRC_MIN
        deghour = 24D/360
        gcirc, 1, candidate [ii].ra   *deghour, candidate [ii].dec,$
                  validated[ind].RADEG*deghour, validated[ind].DEDEG, offset
        candidate[ii].offset = offset
      endif
    endfor ; ii
    info, candidate.Pb_recon
    info, candidate.Pb_ae
    
    dataset_1d, id2, candidate.offset, XTIT='offset [arcsec]', BINSIZE=0.02
    dataset_2d, id1, PSYM=3, candidate.Pb_recon, candidate.Pb_ae, XTIT='Pb (recon_detect)',YTIT='Pb (AE)'
    end



