This is a long-form version of a talk I was invited to give at PyDiff (the Cardiff Python meetup) last year The audience was an exciting range of people, from professional software engineers, to mathematicians who’d written their first Python a few weeks prior.
Tools for better Python
There are things that will help you write better Python that are not Python-specific, like practice, good posture, documentation and empathy. These will help you be better at almost everything you do.
There are Python-specific things that will help you write better Python. However you may find them less useful in other areas of life…
If you’ve a lot of Python experience, you’re probably familiar with virtualenv, but if you’re not already using it, it should make your life substantially easier.
Virtualenv is for creating isolated Python environments.
Are you using root with
sudo or something to install your Python packages?
Virtualenv means you don’t have to.
Got a project here that uses Django 1.4 and a project there that uses Django 1.8? Two different virtualenvs, job done.
Under the hood, there’s some fantastic magic and hackery. My favourite bit is that it’s just one Python file; the various skeleton files it deposits in new virtualenvs are stored as zlib-compressed base64-encoded strings.
No need for a manual process of “create a virtualenv, now activate it, now install these things”; tox will do that for you.
For example, here’s a very basic
[tox] envlist = py27, py33, py34 [testenv] deps = -rrequirements-test.txt commands = nosetests
With this config I’ve declared three “environments” (represented by different virtualenvs). Thanks to some tox magic, each will have a different version of Python. I’ve specified how to install dependencies, and I’ve said what test commands to run
Now I can run
tox (with no arguments) and it’ll create the three different virtualenvs,
install the packages specified in
requirements-test.txt into each,
nosetests from each.
Or I can run
tox -e py27 and only do that for one environment (
e.g. if I was running tests locally by hand and just wanted partial testing in less time.
There’s a lot of static analysis tools out there:
- …and more!
You point them at your code, they’ll give you feedback on quality and conformance. There’s a whole bunch of them, and many people’s opinions differ on what aspects they care about.
Prospector aims to provide sensible defaults. For example most people want to conform to PEP 8. Except PEP 8 mandates a maximum line length of 79 (72 for docstrings). Most common override for pep8? Line length. You can grab each tool individually and tweak as you like, or just grab prospector because life’s too short
Now, frankly, I find rST hard to write compared to Markdown; all the backticks and the fiddly inline formatting, but what Sphinx is able to layer on top is fantastic interlinking support. For example tables of contents:
.. toctree:: :maxdepth: 2 what_is_laterpay getting_to_the_office slack filing_issues idonethis key_user_facing_urls
which ends up as:
We collect application errors in :doc:`/services/external/sentry`
which ends up as:
You can also link to subsections, to figures and diagrams and more, with internal links being checked and validated as part of the build process. Plus you’ve got the usual things like code highlighting, image inclusion etc.
Coverage is a tool primarily used for measuring how much of a codebase is exercised by a test suite.
You have some code. You have some tests. How do you know that you’re testing it all, or if you’re not, what bits you’ve missed.
From David R. MacIver’s Empirically Derived Testing Principles (which I strongly agree with):
100% coverage is mandatory
though it’s worth noting that:
100% coverage tells you nothing, but less than 100% coverage tells you something
“100% coverage” doesn’t mean you have good tests, but less than 100% means you definitely have untested code.
It’s particularly important for Python because the following code will quite happily run
(despite the notable absence of
cheesecake from Python’s stdlib):
$ cat cheesecake.py if True: pass else: cheesecake $ python cheesecake.py $ echo $? 0
Yes that’s a trivial example,
but imagine the case where the
if clause is checking the result of a network call,
cheesecake is your error-handling code.
Most of the time the network call succeeds,
until that one time it doesn’t,
and you find your error-handling code is utter junk and your program dies.
The sooner you find problems, the better. Tests help you do that, especially with good coverage, especially with 100% coverage.
doctest lets you test your code by running examples embedded in documentation
My description would be: doctest lets you test your documentation by testing example code you embed.
It’s easy for documentation to rot; you change the code, you forget to change the documentation.
For example, here’s a (terrible) implementation of string reversal - takes a string, returns a string. It doesn’t properly handle graphemes and composition etc. but will work well enough. But someone had come along and changed it to return a pair, because, well, reasons.
def reverse(s): """ Takes a string and returns it reversed >>> reverse("hello") "olleh" """ return s, s[::-1]
Thanks to doctest:
$ python -m doctest 05-doctest-source.py ********************************************************************** File "05-doctest-source.py", line 5, in 05-doctest-source.reverse Failed example: reverse("hello") Expected: "olleh" Got: ('hello', 'olleh') ********************************************************************** 1 items had failures: 1 of 1 in 05-doctest-source.reverse ***Test Failed*** 1 failures.
Classic testing often looks like a series of anecdotes - David R. MacIver
“Here’s my function, if I give it A do I get B? If I give it C do I get D?” etc. This is limited to “anecdotes you think of”. Hypothesis lets you write tests of the form “given some data like this, does the output satisfy these properties?”
For example, here’s a buggy quicksort:
def quicksort(xs): if xs == : return xs pivot = xs left = quicksort([x for x in xs if x < pivot]) right = quicksort([x for x in xs if x > pivot]) return left + [pivot] + right
You might write some tests and have them pass and believe it works:
>>> quicksort() ==  True >>> quicksort() ==  True >>> quicksort([1,2,3]) == [1,2,3] True >>> quicksort([3,2,1]) == [1,2,3] True >>> quicksort([3,2,1,4]) == [1,2,3,4] True
However, a very basic Hypothesis property test…
from hypothesis import given from hypothesis.strategies import integers, lists @given(lists(integers())) def test_length_unchanged(l): assert len(quicksort(l)) == len(l) test_length_unchanged()
…will quickly show you that’s not the case:
$ python hypothesis-test.py Falsifying example: test_length_unchanged(l=[0, 0])
(The sort doesn’t cover the case when
x == pivot!)
Things I wouldn’t recommend. If you use them, I wouldn’t suggest that you stop, but if you’re not, I also wouldn’t necessarily recommend you start.
I’d rather see people diagnose their problems by adding more tests and logging.
I’m sure Visual Studio, PyCharm, NetBeans etc. are great,
but if you’re not familiar with one already,
I’d suggest just grabbing a text editor you’re comfortable with.
I’ve seen people do awesome things with MS Word and a
If you have an IDE you use and like, great, but while I wouldn’t want to write Java without something like Eclipse or IntelliJ, for Python I’d go more minimal.
I believe these tools are awesome.
I believe they’ll help you write better Python.
I use these for virtually every Python project I build, and suggest you do too.
Recommendations of your own? Let me know!
Why “The Book of Armaments”?
Shoehorning in Monty Python references can be hard sometimes.