Kevin Stone

Batteries Included Django

July 16, 2013

A key benefit to using Python and/or Django for your next project, is the wealth of high quality 3rd party libraries that can help accelerate your development. Python has a rich ecosystem of 3rd party libraries including often offical bindings for an external service. This offers you significant tools to building rich, complex systems based on the available open source solutions. Django's popularity and architecture provides the opportuntiy to find many existing solutions to common problems.

For education and discussion, I thought it would be helpful to enumerate the many libraries we currently use on Subblime to leverage the power of of the Python/Django ecosystem. A what's in our toolbox sort of post.

Requirements.txt

distribute>=0.6.28
mysql-python>=1.2.4
Django>=1.5.4
pillow
Werkzeug
south>=0.8,<2.0
sorl-thumbnail
hamlpy
requests>=1.0
djangorestframework>=2.3.4
djangorestframework-csv
unicodecsv
# pynliner
# # Use our fork that leverages lxml, tinycss and cssselect
https://github.com/subblime/pynliner/tarball/master#egg=pynliner
cssutils

# Django Enhancements
django-extensions>1.2.1	# 1.2.0 included a unicode bug (see https://github.com/django-extensions/django-extensions/pull/366)
django-model-utils
django-templated-email
django-grappelli>=2.4.5
django-floppyforms
django-filter>=0.7
django-braces
django-autoslug

# Datetime
pytz>=2013d
iso8601
python-dateutil>=1.5,<2.0	# 2.1 includes breaking changes

# Deployment
uwsgi
fabric

# Caching
redis
hiredis
django-redis-cache
# johnny-cache
# Patch to skip caching ORDER BY RAND() queries
https://github.com/subblime/johnny-cache/tarball/master#egg=johnny-cache

# Django Compressor
django-compressor>=1.3	# Need recent django-compressor for precompile filter support
lxml
BeautifulSoup<4.0
cssselect
cssmin
tinycss
slimit
django-sekizai

# Celery
django-celery-with-redis	# includes celery and django-celery
flower
django-celery-transactions

# YouTube
gdata
google-api-python-client
oauth2client
django-social-auth>=0.7.22	# v0.7.22 removed social_auth.signals.pre_update which we now support through pipeline stages

# Social
facepy
twitter

# Mixpanel
# v0.6 introduced Celery 3.0 support
mixpanel-celery>=0.6.0

# For Sentry
raven>=3.0

# Lettuce
# Need version of Lettuce that supports proper Background sections including xunit support
lettuce>=0.2.18

# Charting
numpy
pandas>=0.12
sqlalchemy

# Search
django-haystack
pyelasticsearch
celery-haystack

Testing Requirements.txt

factory_boy>=1.3,<2.0	# Guard against potential 2.0 breaking changes
mock
#pytimecop

# Nose
nose<1.3	# Nose 1.3 breaks our test discovery somehow
django-nose
coverage
nose-exclude
nose-progressive>=1.4.1
nosexcover
yanc
xtraceback

# Validaiton
pep8
flake8

Lettuce Requirements.txt

# Make sure you have Chromium installed (or appropriate for your dev platform)
splinter>=0.5    # Need 0.5 for PhantomJS support
selenium>=2.28	# Need 2.28 for PhantomJS support

Basic Includes for Every Project

These are requirements I type before I even start a project. You're just likely to need them with some part of your problem set.

DateTime Fixes

  • pytz (Timezone support)
  • iso8601 (date formatting and parsing)
  • dateutil (parsing and translation)

You always end up exporting importing to CSV

  • unicodecsv (Fixes encoding for CSV import/export)

General Django Helpers

Some utility knife style general enhancements to Django

  • django-extensions - runserverplus + shellplus management commands (Werkzeug Debugger) - Useful generic model fields (CreationDateTimeField, etc)
  • django-model-utils - Improved Querysets (PassThroughManager, InheritanceManager)
  • django-braces - CBV Mixins (instead of ugly decorators)

Django CRUD Helpers

  • django-filter - Don't write your own query string parameterizations until you've tried django-filter
  • django-floppyforms - Better templates for django forms - import floppyforms as forms

Admin Tools

Batch Processing

Celery

Email

  • django-templated-email - Template support for mail subjects, and HTML + Plain versions
  • pynliner - inline your CSS for to re-use your css in mail templates

Cleaner Frontend Code

  • HamlPy (Template loader) - More readable and concise html generation - (Also more version control friendly since it's line based)
  • Django Compressor - Supports LessCSS and Coffeescript - lxml, BeautifulSoup, cssmin, slimit for minification - NPM Packages: less, jshint, recess, uglify-js, coffee-script
  • django-sekizai - Adding resources in template fragments on demand

Haml Templates

Regular Django HTML

<p>A bunch of placeholder text about your company.</p>
<div class="actions">
    <div class="action-container pull-left">
        <a class="flat-button block-level call-to-action" href="{% url "join" %}">Join our Site!</a>
    </div>
    <div class="action-container pull-right">
        <a class="flat-button block-level call-to-action" href="{% url "login" %}">Sign In</a>
    </div>
</div>

Updated for Haml

%p
    A bunch of placeholder text about your company.
.actions
    .action-container.pull-left
        %a.flat-button.block-level.call-to-action{'href': '{% url "join" %}'}
            Join our Site!
    .action-container.pull-right
        %a.flat-button.block-level.call-to-action{'href': '{% url "login" %}'}
            Sign In

Another Example

<div class="product-information-container">
    <div class="product-information">
        <h4>
            {% if headline %}{{ headline }}{% endif %}
            {% if url %}
            <p>
                <p>
                    <a href="{{ url }}">{{ url|format_url }}</a>
                </p>
            </p>
            {% endif %}
        </h4>
        {% if description %}
        <p class="description">{{ description }}</p>
    </div>
</div>
.product-information-container
    .product-information
        %h4
            - if headline
                = headline
            - if url
                %p
                    %a{'href': '={url}'}
                        = url|format_url
        - if description
            %p.description<
                = description

Django Sekizai

Sekizai allows you to specify assets (css, js) anywhere in your templates and capture them to include in the base template.

In Your Base Template:

{% render_block "js" %}

or with Django Compressor Processors (say for Coffeescript)

{% render_block "js" postprocessor "compressor.contrib.sekizai.compress" %}

Included Fragment Example

- addtoblock "js"
    %script{'type': 'text/javascript', 'src': '{% static "js/jquery/jquery.timeago.js" %}'}
- endaddtoblock "js"

Included Fragment Example (Coffeescript)

{% addtoblock "js" %}
    <script type="text/coffeescript" src="{% static "js/directives/date.coffee" %}"></script>
{% endaddtoblock "js" %}

Backend Services

Algorithmic/Data Science

  • NumPy
  • Pandas - Powerful data transformations (esp. for time series data)

Other Useful Libs

  • SciPy - Algorithms galore
  • NLTK (language processing + statistics primitives)
  • PyMC (Bayesian inference)

Caching

APIs and Authentication

Tracking/Reporting

Deployment and Testing

Testing

Feature Tests

Validation

  • pep8 (coding standards)
  • flake8 (static analysis based on PyFlakes)

Nose Plugins

© Kevin Stone 2020, Built with Gatsby