Development Procedure
The following document describes the procedure to follow for making changes to KCWI_DRP.
Installing for Development
We recommend installing directly from the source code if you plan on working on the pipeline.
git clone git@github.com:Keck-DataReductionPipelines/KCWI_DRP.git cd KCWI_DRP pip install -e .
Note
We are using ssh
to clone from GitHub, this is
required if you plan on making pushes directly to the repository. You may use
HTTPS (i.e. git clone https://github.com/Keck-DataReductionPipelines/KCWI_DRP.git
)
if needed for your setup.
Using the -e
flag with pip
makes the installation editable. This means
that any changes you make to the code will take effect immediately, and you will
not need to re-install the pipeline after each change.
Warning
Installing using setup.py
directly can lead to conflicts if you
later try using pip
. If you need to install using pip
later, it is
highly recommended that you create a new conda environment first. It is possible
to uninstall the kcwidrp
module, but it is not 100% reliable.
Managing Git
There are two persistent branches in the KCWI_DRP repo:
master
(default): This branch contains the current release version of the code, and provides the source used to build the package forconda
andpip
.develop
: This branch contains new features and non-essential fixes. It should be stable at all times. A new release is triggered by this branch being merged intomaster
. Any new code should be added to this branch.
When adding new code, please do your development in a branch off of develop
:
git checkout develop
git pull
git checkout -b new_feature
While developing, it is important to stay up-to-date with any changes in the
develop
branch:
git checkout develop
git pull
git checkout new_feature
git merge --no-ff develop
Alternatively, you can do your own development in your own fork of the repo.
Pull Requests
Once your new feature is ready for merging into the develop branch, you need to issue a pull request. The pull request must contain the following:
A short but thorough explanation of the feature and its purpose
A pass from the automatic TravisCI tests
Evidence of successfully processing a night of KCWI data
Additionally, the code must meet the following:
All code must be documented. This includes updated docstrings for altered code and new ones for new functions or classes
Comments explaining any unusually complicated or unintuitive code blocks, in addition to the documentation above
Remove all print statements. If you need to output information to terminal, use the logging interface
At least two developers must approve of the request for it to be accepted
If your pull request meets all of the above, it will be merged into develop
during the next developer meeting.
Merging develop
into master
At the developer’s discretion, develop
will be merged into master
, which
will signify a new sub-release. The version number should be incremented in
develop
, and a summary of changes should be added to changelog.txt. Then,
issue and accept a pull request from develop
into master
.
Building for Release
These instructions summarize the steps required to build a new release of
kcwidrp for conda or pip. These steps should be followed every time master
is updated.
Pip
In order to upload to pip, you will need access to a PyPI account with owndership status for the kcwidrp project. For access to the KeckDRPs account, ask the DSI Team.
After your pull request is merged into master, download the changes:
git checkout master
git pull
Next, ensure that the dist
directory is empty, or does not exist. If any
previous versions exist in the dist
directory, they will conflict with the upload to PyPI):
# Make sure you have the most recent version of twine installed
pip install twine --upgrade
# Construct the pip distribution
python setup.py sdist bdist_wheel
# Test the upload by uploading to TestPyPI
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Make sure that the test upload worked using the following command, replacing
VERSION
with the version number in setup.py
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple kcwidrp==VERSION
Setting --index-url
tells pip where to find our package (in this case, the
test server). --extra-index-url
tells pip to install all other packages
(i.e. requirements) from the default server.
Ensure that the pipeline is built correctly, requirements were installed, and
that you are able to reduce a test dataset. If all went well, finalize the
upload to PyPI (pip
) with
twine upload dist/*
Conda
Eventually, these steps will be rendered obsolete by the use of conda-forge. In the meantime, the following instructions will build a conda package from the pip package. This should be run whenever a new pip version is created.
conda update conda
conda install conda-build anaconda-client
conda-build conda_build_files
conda build conda_build_files --output
anaconda login
anaconda upload PATH-FROM-OUTPUT