Django Syntax Highlighting with reStructuredText and Pygments

Despite launching my blog only last week using Markdown as my blogging markup language, after reading Eric Holscher's Large Problems in Django, Mostly Solved: Documentation, I was curious if it would be much, if any, work to switch to reStructuredText and still retain the code highlighting support I was getting from Markdown's CodeHilite extension. The actual highlighting comes from Pygments, which is stand alone and should be possible to hook into reStructuredText as well.

Why bother switching? reStructuredText is pretty well loved among the Python community for documentation, and rightfully so. Although ReST is much more extensive than I need for blogging, I see no need to be using two markup languages. Plus, it saves me a module, since I don't have to install markdown.

First, make sure you have Docutils and Pygments installed. I tend to install Python modules with pip:

$ pip install docutils pygments

django.contrib.markup comes with a template filter, |restructuredtext that converts ReST into HTML, and if that works for your purposes, go ahead and use it. I, however, following advice from James Bennett's Practical Django Projects choose to store the rendered content in my database. This way it doesn't have to be parsed every time it is displayed. I peeked into the restructuredtext filter to see how it worked:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
def restructuredtext(value):
    try:
        from docutils.core import publish_parts
    except ImportError:
        if settings.DEBUG:
            raise template.TemplateSyntaxError("Error in {% restructuredtext %} filter: The Python docutils library isn't installed.")
        return force_unicode(value)
    else:
        docutils_settings = getattr(settings, "RESTRUCTUREDTEXT_FILTER_SETTINGS", {})
        parts = publish_parts(source=smart_str(value), writer_name="html4css1", settings_overrides=docutils_settings)
        return mark_safe(force_unicode(parts["fragment"]))
restructuredtext.is_safe = True

So, basically it reads in your RESTRUCTUREDTEXT_FILTER_SETTINGS and renders using docutils.core.publish_parts. Personally, it's a pet peeve of mine when I find a template tag that has built in logic that would be useful outside the context of a template tag, but in this case, the template tag function will work fine if we import it and call it from the model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from django.db import models
from django.contrib.markup.templatetags.markup import restructuredtext

class Entry(models.Model):
    title = models.CharField(max_length=150)
    slug = models.SlugField(unique=True)
    content = models.TextField(editable=False, blank=True)
    content_raw = models.TextField()

    def save(self, *args, **kwargs):
        if self.content_raw:
            self.content = restructuredtext(self.content_raw)
        super(Entry, self).save(*args, **kwargs)

On save, I render whatever was typed into the content_raw to html, then store it in content. content is hidden in the admin (editable=False) and content|safe is used in the templates.

ReST, as documented, now works in my Entry model. However, it knows nothing about code syntax highlighting. The Pygments website has a brief entry on ReST support that merely left me confused. Here's the gist of it: you need to create a "directive" (aka markup tag) to display source code and pygmentize it. Pygments graciously provides the necessary code in the external folder of the download, or you can view it online. I have included it below, taking the liberty to uncomment the "linenos" variant option:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
# -*- coding: utf-8 -*-
"""
    The Pygments reStructuredText directive
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    This fragment is a Docutils_ 0.5 directive that renders source code
    (to HTML only, currently) via Pygments.

    To use it, adjust the options below and copy the code into a module
    that you import on initialization.  The code then automatically
    registers a ``sourcecode`` directive that you can use instead of
    normal code blocks like this::

        .. sourcecode:: python

            My code goes here.

    If you want to have different code styles, e.g. one with line numbers
    and one without, add formatters with their names in the VARIANTS dict
    below.  You can invoke them instead of the DEFAULT one by using a
    directive option::

        .. sourcecode:: python
            :linenos:

            My code goes here.

    Look at the `directive documentation`_ to get all the gory details.

    .. _Docutils: http://docutils.sf.net/
    .. _directive documentation:
       http://docutils.sourceforge.net/docs/howto/rst-directives.html

    :copyright: Copyright 2006-2010 by the Pygments team, see AUTHORS.
    :license: BSD, see LICENSE for details.
"""

# Options
# ~~~~~~~

# Set to True if you want inline CSS styles instead of classes
INLINESTYLES = False

from pygments.formatters import HtmlFormatter

# The default formatter
DEFAULT = HtmlFormatter(noclasses=INLINESTYLES)

# Add name -> formatter pairs for every variant you want to use
VARIANTS = {
    'linenos': HtmlFormatter(noclasses=INLINESTYLES, linenos=True),
}


from docutils import nodes
from docutils.parsers.rst import directives, Directive

from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer

class Pygments(Directive):
    """ Source code syntax hightlighting.
    """
    required_arguments = 1
    optional_arguments = 0
    final_argument_whitespace = True
    option_spec = dict([(key, directives.flag) for key in VARIANTS])
    has_content = True

    def run(self):
        self.assert_has_content()
        try:
            lexer = get_lexer_by_name(self.arguments[0])
        except ValueError:
            # no lexer found - use the text one instead of an exception
            lexer = TextLexer()
        # take an arbitrary option if more than one is given
        formatter = self.options and VARIANTS[self.options.keys()[0]] or DEFAULT
        parsed = highlight(u'\n'.join(self.content), lexer, formatter)
        return [nodes.raw('', parsed, format='html')]

directives.register_directive('sourcecode', Pygments)

But what the heck do you do with this? You need to import it somewhere, anywhere really, where it will get loaded. I saved it as rstdirective.py and played it safe by importing it at the top of my settings.py file. Other people import it in the __init__.py of the app using it.

#settings.py
import rstdirective.py

Now you can outline code by using:

.. sourcecode:: python

    import foo #source code!

Or, with line numbers:

1
2
3
4
.. sourcecode:: python
    :linenos:

    import foo #source code!

The code should be rendered, marked up, and ready for color. Pygments doesn't come with any CSS files, although it does have a tool to create them automatically:

$ pygmentize -f html -S monokai -a .highlight > media/css/pygments.css

Basically: create me CSS under the class .highlight using the theme monokai. You can preview the default Pygments themes here. If you're lazy/confused, richleland has a github repo with the CSS files already generated, you just need to change the class from .codehilite to .highlight. Include the CSS in your template link you would any other CSS file, refresh, and enjoy!

Customizing Django Comments

The django.contrib.comments app, like all good apps, comes with many default templates so you can get them up and running in minutes. Unfortunately, by default they will not match your website, since they couldn't possibly know about your site's template structure. Fortunately, because of the way template loading works in Django, it's easy to extend these templates to match.

You may have noticed these lines in your settings.py:

1
2
3
4
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
)

When Django looks up a template, it will check these modules in order.

django.template.loaders.filesystem.load_template_source will check the directories you specified in the TEMPLATE_DIRS setting. If it doesn't find the associated template, it will move onto django.template.loaders.app_directories.load_template_source, which will look in the templates directories of apps you've listed in INSTALLED_APPS.

Barbara Shaurette recommended overriding any of the templates you wanted to customize by changing each to inherit from your base template, but there's a quicker way that lets you play dumb about how many templates there are (and any updates made to them).

Copy only the base.html template ([DJANGO_DIRECTORY]/contrib/comments/templates/comments/base.html) to your template directory and edit it so that instead of having the page markup included, like this:

templates/comments/base.html

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>{% block title %}{% endblock %}</title>
</head>
<body>
    {% block content %}{% endblock %}
</body>
</html>

... have it inherit from your base template (or whatever template you want):

1
2
3
4
{% extends 'base.html' %}
{% block body %}
    {% block content %}{% endblock %}
{% endblock %}

Using this method, by overriding just the top level template, you've customized all the comment templates, without even having to know what they are.

This method will of course work with any application. I just ran in to it specifically using the comments app.

Returning to the Object with Django Comments

If you are using django.contrib.comments for your project, it may not be immediately obvious how to get back to the page the comment was posted from. The user gets left at a "Thank you for your comment" message.

However, the comment object returns a comment specific URL with its get_absolute_url() method. If you have defined get_absolute_url() on your object, the comment URL will effectively redirect to your object's URL. It even passes along an anchor in the form of #c[comment_id] so you can automatically scroll right to the comment!

Since the posted.html template that renders the thank you message already has access to the comment object, one easy way to get the user back where they came from is to override this template by copying it to your template directory and add in a link back to the object:

templates/comments/posted.html

1
2
3
4
{% block content %}
    <h1>{% trans "Thank you for your comment" %}.</h1>
    <a href="{{ comment.get_absolute_url }}">Return to blog entry</a>
{% endblock %}