Running the pipeline

The DRP is instantiated using a startup script offers contains several command line options and different execution modes.

Process files, file lists and entire directories

  • To reduce all files in a directory in the order in which they appear and group them correctly according to the logical sequence needed by the pipeline:
reduce_kcwi -f kb*.fits -g
  • To reduce only a subset of the files (for example, a single object out of an entire observing run):
reduce_kcwi -l input_files.lst
  • To reduce a single file:
reduce_kcwi -f kb001.fits

For the “list” and “single file” cases, please note that the preliminary calibrations needed to reduce the file or the file list must be already on disk. For example, if the file you are trying to reduce is a science frame, the wavelength calibration, flat field and bias frames must already exist.

Reducing a single file is a good way to re-reduce a target for which the pipeline didn’t do a good job. An example case would be the sky subtraction: if you realize that the sky subtraction is not correct, and you modify the sky subtraction using an external file, then you can rerun the pipeline just on the trouble file. In this case it is advisable to use the clobber=True option.

The following option can be used to modify the behaviour of the DRP when processing files in a directory:

--groups This options forces the DRP to group the files by image type and then processes them in the correct order so that master calibrations are produced before processing science images, regardless of the order in which the files appear on disk or are specified in the input list. This is also accessible with the -g flag.

Running individual steps

This is not a standard operating mode and is not supported but it is possible. The pipeline has a number of high level procedures that determine the correct order for data reduction.

If you know what you are doing, you can in principle create individual file lists for the steps.

Example:

Create a file containing bias frames and call it bias.lst. You can can run

reduce_kcwi -l bias.lst

Because the files are all and only bias, the pipeline will only proceed as far as generating the master bias.

For an even finer control, each file in the bias list could be run individually, and only when enough bias frames are present and reduced, the DRP will generate a master bias.

Monitor directories

The DRP has the ability of monitoring a specified directory. When files appear, they are ingested and processed. To start the DRP in this mode use:

reduce_kcwi -d /home/mydata -i kb*.fits -m

The -i KB*.fits (or -i kb*.fits if you are using OFNAME files) is the filter used to recognize the correct files. If it is not specified the pipeline will ingest all files appearing in a directory, and might fail if those files are not KCWI frames.

Other command line options

  • -c config_file.cfg This options overrides the standard configuration file that is stored

in the installation directory in kcwidrp/config/kcwi.cfg.

  • -p proctable.proc When the DRP runs, it keeps track of the files processed using

a processing table. Normally that table is called kcwi.proc and is stored in the current directory. This options is used to specify a different file if needed.

  • -t taper_fraction TBD