Batteries Included Django

A key benefit to using Python and/or Django for your next project, is the wealth of high quality 3rd party libraries that can help accelerate your development. Python has a rich ecosystem of 3rd party libraries including often offical bindings for an external service. This offers you significant tools to building rich, complex systems based on the available open source solutions. Django's popularity and architecture provides the opportuntiy to find many existing solutions to common problems.

For education and discussion, I thought it would be helpful to enumerate the many libraries we currently use on Subblime to leverage the power of of the Python/Django ecosystem. A what's in our toolbox sort of post.

Requirements.txt

distribute>=0.6.28
mysql-python>=1.2.4
Django>=1.5.4
pillow
Werkzeug
south>=0.8,<2.0
sorl-thumbnail
hamlpy
requests>=1.0
djangorestframework>=2.3.4
djangorestframework-csv
unicodecsv
# pynliner
# # Use our fork that leverages lxml, tinycss and cssselect
https://github.com/subblime/pynliner/tarball/master#egg=pynliner
cssutils

# Django Enhancements
django-extensions>1.2.1	# 1.2.0 included a unicode bug (see https://github.com/django-extensions/django-extensions/pull/366)
django-model-utils
django-templated-email
django-grappelli>=2.4.5
django-floppyforms
django-filter>=0.7
django-braces
django-autoslug

# Datetime
pytz>=2013d
iso8601
python-dateutil>=1.5,<2.0	# 2.1 includes breaking changes

# Deployment
uwsgi
fabric

# Caching
redis
hiredis
django-redis-cache
# johnny-cache
# Patch to skip caching ORDER BY RAND() queries
https://github.com/subblime/johnny-cache/tarball/master#egg=johnny-cache

# Django Compressor
django-compressor>=1.3	# Need recent django-compressor for precompile filter support
lxml
BeautifulSoup<4.0
cssselect
cssmin
tinycss
slimit
django-sekizai

# Celery
django-celery-with-redis	# includes celery and django-celery
flower
django-celery-transactions

# YouTube
gdata
google-api-python-client
oauth2client
django-social-auth>=0.7.22	# v0.7.22 removed social_auth.signals.pre_update which we now support through pipeline stages

# Social
facepy
twitter

# Mixpanel
# v0.6 introduced Celery 3.0 support
mixpanel-celery>=0.6.0

# For Sentry
raven>=3.0

# Lettuce
# Need version of Lettuce that supports proper Background sections including xunit support
lettuce>=0.2.18

# Charting
numpy
pandas>=0.12
sqlalchemy

# Search
django-haystack
pyelasticsearch
celery-haystack

Testing Requirements.txt

factory_boy>=1.3,<2.0	# Guard against potential 2.0 breaking changes
mock
#pytimecop

# Nose
nose<1.3	# Nose 1.3 breaks our test discovery somehow
django-nose
coverage
nose-exclude
nose-progressive>=1.4.1
nosexcover
yanc
xtraceback

# Validaiton
pep8
flake8

Lettuce Requirements.txt

# Make sure you have Chromium installed (or appropriate for your dev platform)
splinter>=0.5    # Need 0.5 for PhantomJS support
selenium>=2.28	# Need 2.28 for PhantomJS support

Basic Includes for Every Project

These are requirements I type before I even start a project. You're just likely to need them with some part of your problem set.

Pillow (Image Manipulation)
sorl-thumbnail (Thumbnail Generation)
South (DB Migrations)
Requests (HTTP client)
Django-Rest-Framework (ReST API endpoint)

DateTime Fixes

pytz (Timezone support)
iso8601 (date formatting and parsing)
dateutil (parsing and translation)

You always end up exporting importing to CSV

unicodecsv (Fixes encoding for CSV import/export)

General Django Helpers

Some utility knife style general enhancements to Django

django-extensions - runserver_plus + shell_plus management commands (Werkzeug Debugger) - Useful generic model fields (CreationDateTimeField, etc)
django-model-utils - Improved Querysets (PassThroughManager, InheritanceManager)
django-braces - CBV Mixins (instead of ugly decorators)

Django CRUD Helpers

django-filter - Don't write your own query string parameterizations until you've tried django-filter
django-floppyforms - Better templates for django forms - import floppyforms as forms

Admin Tools

django-grappelli (nicer skin)
django-extensions (again) - ForeignKeyAutocompleteAdmin makes admin useable for relations (search field instead of dropdown)

Batch Processing

Celery

django-celery
django-celery-transactions - Delays task delivery until transactions commit to eliminate race conditions

Email

django-templated-email - Template support for mail subjects, and HTML + Plain versions
pynliner - inline your CSS for to re-use your css in mail templates

Cleaner Frontend Code

HamlPy (Template loader) - More readable and concise html generation - (Also more version control friendly since it's line based)
Django Compressor - Supports LessCSS and Coffeescript - lxml, BeautifulSoup, cssmin, slimit for minification - NPM Packages: less, jshint, recess, uglify-js, coffee-script
django-sekizai - Adding resources in template fragments on demand

Haml Templates

Regular Django HTML

<p>A bunch of placeholder text about your company.</p>
<div class="actions">
    <div class="action-container pull-left">
        <a class="flat-button block-level call-to-action" href="{% url "join" %}">Join our Site!</a>
    </div>
    <div class="action-container pull-right">
        <a class="flat-button block-level call-to-action" href="{% url "login" %}">Sign In</a>
    </div>
</div>

Updated for Haml

%p
    A bunch of placeholder text about your company.
.actions
    .action-container.pull-left
        %a.flat-button.block-level.call-to-action{'href': '{% url "join" %}'}
            Join our Site!
    .action-container.pull-right
        %a.flat-button.block-level.call-to-action{'href': '{% url "login" %}'}
            Sign In

Another Example

<div class="product-information-container">
    <div class="product-information">
        <h4>
            {% if headline %}{{ headline }}{% endif %}
            {% if url %}
            <p>
                <p>
                    <a href="{{ url }}">{{ url|format_url }}</a>
                </p>
            </p>
            {% endif %}
        </h4>
        {% if description %}
        <p class="description">{{ description }}</p>
    </div>
</div>

.product-information-container
    .product-information
        %h4
            - if headline
                = headline
            - if url
                %p
                    %a{'href': '={url}'}
                        = url|format_url
        - if description
            %p.description<
                = description

Django Sekizai

Sekizai allows you to specify assets (css, js) anywhere in your templates and capture them to include in the base template.

In Your Base Template:

{% render_block "js" %}

or with Django Compressor Processors (say for Coffeescript)

{% render_block "js" postprocessor "compressor.contrib.sekizai.compress" %}

Included Fragment Example

- addtoblock "js"
    %script{'type': 'text/javascript', 'src': '{% static "js/jquery/jquery.timeago.js" %}'}
- endaddtoblock "js"

Included Fragment Example (Coffeescript)

{% addtoblock "js" %}
    <script type="text/coffeescript" src="{% static "js/directives/date.coffee" %}"></script>
{% endaddtoblock "js" %}

Backend Services

Algorithmic/Data Science

NumPy
Pandas - Powerful data transformations (esp. for time series data)

Other Useful Libs

SciPy - Algorithms galore
NLTK (language processing + statistics primitives)
PyMC (Bayesian inference)

Caching

redis-py
hiredis-py (faster bindings for redis)
django-redis-cache (cache backend using redis)
johnny-cache (transparent queryset caching)

APIs and Authentication

django-social-auth (oauth consumers) - now python-social-auth
Google - oauth2client - google-api-python-client - gdata
Facebook - facepy

Tracking/Reporting

mixpanel-celery (background mixpanel tracking)
raven (for Sentry)

Deployment and Testing

Fabric (automation)
uWSGi (WSGi container)
Werkzeug (Lightweight WSGi framework)

Testing

Factory-boy (Fixtures)
mock (Mocking and Stubbing)

Feature Tests

lettuce
selenium (web browser automation)
splinter (selenium wrapper)

Validation

pep8 (coding standards)
flake8 (static analysis based on PyFlakes)

Nose Plugins

nose
django-nose - Nose based Test Runner for Django
coverage - Code Coverage
nose-exclude - Adds --exclude-dir= flags to skip certain dirs
nose-progressive - Progress bar for test runner
nosexcover (worst package name ever) - Code coverage for xunit results (for Jenkins)
yanc - Output Color-izer
xtraceback - Better Test Tracebacks