Kevin Stone

    Batteries Included Django

    July 16, 2013

    A key benefit to using Python and/or Django for your next project, is the wealth of high quality 3rd party libraries that can help accelerate your development. Python has a rich ecosystem of 3rd party libraries including often offical bindings for an external service. This offers you significant tools to building rich, complex systems based on the available open source solutions. Django's popularity and architecture provides the opportuntiy to find many existing solutions to common problems.

    For education and discussion, I thought it would be helpful to enumerate the many libraries we currently use on Subblime to leverage the power of of the Python/Django ecosystem. A what's in our toolbox sort of post.

    Requirements.txt

    distribute>=0.6.28
    mysql-python>=1.2.4
    Django>=1.5.4
    pillow
    Werkzeug
    south>=0.8,<2.0
    sorl-thumbnail
    hamlpy
    requests>=1.0
    djangorestframework>=2.3.4
    djangorestframework-csv
    unicodecsv
    # pynliner
    # # Use our fork that leverages lxml, tinycss and cssselect
    https://github.com/subblime/pynliner/tarball/master#egg=pynliner
    cssutils
    
    # Django Enhancements
    django-extensions>1.2.1	# 1.2.0 included a unicode bug (see https://github.com/django-extensions/django-extensions/pull/366)
    django-model-utils
    django-templated-email
    django-grappelli>=2.4.5
    django-floppyforms
    django-filter>=0.7
    django-braces
    django-autoslug
    
    # Datetime
    pytz>=2013d
    iso8601
    python-dateutil>=1.5,<2.0	# 2.1 includes breaking changes
    
    # Deployment
    uwsgi
    fabric
    
    # Caching
    redis
    hiredis
    django-redis-cache
    # johnny-cache
    # Patch to skip caching ORDER BY RAND() queries
    https://github.com/subblime/johnny-cache/tarball/master#egg=johnny-cache
    
    # Django Compressor
    django-compressor>=1.3	# Need recent django-compressor for precompile filter support
    lxml
    BeautifulSoup<4.0
    cssselect
    cssmin
    tinycss
    slimit
    django-sekizai
    
    # Celery
    django-celery-with-redis	# includes celery and django-celery
    flower
    django-celery-transactions
    
    # YouTube
    gdata
    google-api-python-client
    oauth2client
    django-social-auth>=0.7.22	# v0.7.22 removed social_auth.signals.pre_update which we now support through pipeline stages
    
    # Social
    facepy
    twitter
    
    # Mixpanel
    # v0.6 introduced Celery 3.0 support
    mixpanel-celery>=0.6.0
    
    # For Sentry
    raven>=3.0
    
    # Lettuce
    # Need version of Lettuce that supports proper Background sections including xunit support
    lettuce>=0.2.18
    
    # Charting
    numpy
    pandas>=0.12
    sqlalchemy
    
    # Search
    django-haystack
    pyelasticsearch
    celery-haystack

    Testing Requirements.txt

    factory_boy>=1.3,<2.0	# Guard against potential 2.0 breaking changes
    mock
    #pytimecop
    
    # Nose
    nose<1.3	# Nose 1.3 breaks our test discovery somehow
    django-nose
    coverage
    nose-exclude
    nose-progressive>=1.4.1
    nosexcover
    yanc
    xtraceback
    
    # Validaiton
    pep8
    flake8

    Lettuce Requirements.txt

    # Make sure you have Chromium installed (or appropriate for your dev platform)
    splinter>=0.5    # Need 0.5 for PhantomJS support
    selenium>=2.28	# Need 2.28 for PhantomJS support

    Basic Includes for Every Project

    These are requirements I type before I even start a project. You're just likely to need them with some part of your problem set.

    DateTime Fixes

    • pytz (Timezone support)
    • iso8601 (date formatting and parsing)
    • dateutil (parsing and translation)

    You always end up exporting importing to CSV

    • unicodecsv (Fixes encoding for CSV import/export)

    General Django Helpers

    Some utility knife style general enhancements to Django

    • django-extensions - runserverplus + shellplus management commands (Werkzeug Debugger) - Useful generic model fields (CreationDateTimeField, etc)
    • django-model-utils - Improved Querysets (PassThroughManager, InheritanceManager)
    • django-braces - CBV Mixins (instead of ugly decorators)

    Django CRUD Helpers

    • django-filter - Don't write your own query string parameterizations until you've tried django-filter
    • django-floppyforms - Better templates for django forms - import floppyforms as forms

    Admin Tools

    Batch Processing

    Celery

    Email

    • django-templated-email - Template support for mail subjects, and HTML + Plain versions
    • pynliner - inline your CSS for to re-use your css in mail templates

    Cleaner Frontend Code

    • HamlPy (Template loader) - More readable and concise html generation - (Also more version control friendly since it's line based)
    • Django Compressor - Supports LessCSS and Coffeescript - lxml, BeautifulSoup, cssmin, slimit for minification - NPM Packages: less, jshint, recess, uglify-js, coffee-script
    • django-sekizai - Adding resources in template fragments on demand

    Haml Templates

    Regular Django HTML

    <p>A bunch of placeholder text about your company.</p>
    <div class="actions">
        <div class="action-container pull-left">
            <a class="flat-button block-level call-to-action" href="{% url "join" %}">Join our Site!</a>
        </div>
        <div class="action-container pull-right">
            <a class="flat-button block-level call-to-action" href="{% url "login" %}">Sign In</a>
        </div>
    </div>

    Updated for Haml

    %p
        A bunch of placeholder text about your company.
    .actions
        .action-container.pull-left
            %a.flat-button.block-level.call-to-action{'href': '{% url "join" %}'}
                Join our Site!
        .action-container.pull-right
            %a.flat-button.block-level.call-to-action{'href': '{% url "login" %}'}
                Sign In

    Another Example

    <div class="product-information-container">
        <div class="product-information">
            <h4>
                {% if headline %}{{ headline }}{% endif %}
                {% if url %}
                <p>
                    <p>
                        <a href="{{ url }}">{{ url|format_url }}</a>
                    </p>
                </p>
                {% endif %}
            </h4>
            {% if description %}
            <p class="description">{{ description }}</p>
        </div>
    </div>
    .product-information-container
        .product-information
            %h4
                - if headline
                    = headline
                - if url
                    %p
                        %a{'href': '={url}'}
                            = url|format_url
            - if description
                %p.description<
                    = description

    Django Sekizai

    Sekizai allows you to specify assets (css, js) anywhere in your templates and capture them to include in the base template.

    In Your Base Template:

    {% render_block "js" %}

    or with Django Compressor Processors (say for Coffeescript)

    {% render_block "js" postprocessor "compressor.contrib.sekizai.compress" %}

    Included Fragment Example

    - addtoblock "js"
        %script{'type': 'text/javascript', 'src': '{% static "js/jquery/jquery.timeago.js" %}'}
    - endaddtoblock "js"

    Included Fragment Example (Coffeescript)

    {% addtoblock "js" %}
        <script type="text/coffeescript" src="{% static "js/directives/date.coffee" %}"></script>
    {% endaddtoblock "js" %}

    Backend Services

    Algorithmic/Data Science

    • NumPy
    • Pandas - Powerful data transformations (esp. for time series data)

    Other Useful Libs

    • SciPy - Algorithms galore
    • NLTK (language processing + statistics primitives)
    • PyMC (Bayesian inference)

    Caching

    APIs and Authentication

    Tracking/Reporting

    Deployment and Testing

    Testing

    Feature Tests

    Validation

    • pep8 (coding standards)
    • flake8 (static analysis based on PyFlakes)

    Nose Plugins