Advanced track bootcamp¶
Bootcamp to cover more advanced topics in scientific Python.
Program¶
We are going to start up using a typical development workflow in Python, using the example of making a package for ordinary least squares fitting.
To make things clear to our future selves, and other collaborators, we will document what we are doing as we go.
Introduction¶
General pattern of work: explore; write tests; write code; have code reviewed and merged; repeat.
Dividing into groups;
Revision on git, github, pull requests.
Editors;
Getting help:
- Python docs at https://docs.python.org;
- IPython tab completion, something?, something??;
- np.lookfor;
- your search engine (sigh).
If you haven’t read this already, read http://practical-neuroimaging.github.io/git_parable.html
Information on installing via pip on Debian / Ubuntu at : https://matthew-brett.github.io/pydagogue/installing_on_debian.html
Exercises¶
- Installing Python 3, pip;
- install Python 3 / pip as necessary.
- Working with forks and remotes; making feature branches; making pull
requests;
- forking the repo; cloning the repo; adding the remotes; make feature
branch
edit-readme
; edit the README, make a pull request.
- forking the repo; cloning the repo; adding the remotes; make feature
branch
- Introduction to Restructured Text.
ReST in Sphinx;
- make feature branch
rst-readme
; editREADME.rst
; installdocutils
; check rendering withrst2html.py
; add; commit. Check rendering on github. Do pull request (PR).
- make feature branch
- Starting from
origin/master
;- get merged changes from github; make new feature branch
add-requirements
starting at the merged state.
- get merged changes from github; make new feature branch
- Introduction to virtualenvs and virtualenvwrapper;
- install virtualenvwrapper; create
ols
virtualenv.
- install virtualenvwrapper; create
- Pip installs, requirements files:
- install numpy; feature branch (FB)
add-requirements
; specify in requirements file, addrequirements.txt
file, commit, do PR.
- install numpy; feature branch (FB)
- The Python path; Python packages;
- create package structure for
ols
package. Show you canimport ols
. Add some explanatory text in init file. Importols
again to show docstring. Change into another directory and try import. Fix. Make FBpackage-structure
; add and commit; do PR.
- create package structure for
- The
.gitignore
file;- add
.gitignore
file to FBadd-gitignore
; Commit, PR.
- add
- The setup.py
file; installing in pip develop mode:
- create
setup.py
forols
, pip install in develop mode. Show that you can now import from anywhere. Make FBadd-setup
. Commit; do PR.
- create
- The explore phase. Reminder about ordinary least squares fitting for model
$Y = X B + E$. See GLM intro;
- in IPython, recreate psychopathy $Y$, clamminess $vec{x}$. Create $X$. Fit model and find coefficients $B$.
- Test first development; pytest; comparing arrays;
typical imports such as
import numpy as np
:- FB
add-test-fit
. Install pytest with pip. Add to requirements. Make a test for a new functionols.model.fit
testing that that you can reconstruct parameters from the GLM intro. Run the test and make sure it fails. Write the functionfit
inols/model.py
. Commit, do PR.
- FB
- Floating point, almost equal
- FB
close-tests
. Test that results are close to expected. Do PR.
- FB
- Numpy and almost equal;
- FB
np-testing
; usenp.testing
to do the almost equal test. Document your testing procedure in the README.
- FB
- Importing from the top level:
- FB:
top-level-imports
. How do you importfit
directly fromols
? As infrom ols import fit
? Do PR.
- FB:
- Remember the README:
- FB
document-testing
. Document the testing procedure so someone else can reproduce it.
- FB
- Check collaboration / replication.
- Start in a new directory; make a new virtualenv, and follow the instructions in the README exactly, to run the tests. Fix any problems; do a PR.
- Testing edge cases; reminder on reshape and introduction to
newaxis
:- what happens if you pass a list into the
fit
function? Or a 1D array? FBextend-tests
; Make test cases. Run the tests. Fix. What happens if the design is rank deficient? Test. Fix if necessary. Do PR.
- what happens if you pass a list into the
- Dealing with errors. Try / except / finally.
- use try / except to test / confirm that error arises from passing design and data of different number of rows.
- More advanced pytests:
- see if you can find a neater way to test for errors being raised using pytest.
- Rethinking the design as objects. The
Model
andResults
objects. Thinking about the design.- write tests for the Model and Results objects. Run tests and fail. Write Model and results objects.
- Loading text files with numpy. Working out which directory your test is
running in.
- I will add a random X and Y to the repository, and save the expected coefficients out as another text file. Make a feature branch; in the tests, load the X and Y, fit the model, test the coefficients against the result I got. Do a PR.
- Testing against other packages:
- Load text files from R. Find model coefficients using
lm
. Save from R session. Make test from saved R results. Clue: R command for writing a simple text file of numeric values iswrite.table(var, row.names=FALSE, col.names=FALSE, file='filename.txt')
.
- Load text files from R. Find model coefficients using
- Decorators and properties.
- write tests for residuals as a property. Fix tests.