# HTML Templating with twisted.web.template

## A Very Quick Introduction To Templating In Python

HTML templating is the process of transforming a template document (one which describes style and structure, but does not itself include any content) into some HTML output which includes information about objects in your application. There are many, many libraries for doing this in Python: to name a few, jinja2, django templates, and clearsilver. You can easily use any of these libraries in your Twisted Web application, either by running them as WSGI applications or by calling your preferred templating system's APIs to produce their output as strings, and then writing those strings to `Request.write`.

Before we begin explaining how to use it, I'd like to stress that you don't need to use Twisted's templating system if you prefer some other way to generate HTML. Use it if it suits your personal style or your application, but feel free to use other things. Twisted includes templating for its own use, because the `twisted.web` server needs to produce HTML in various places, and we didn't want to add another large dependency for that. Twisted is not in any way incompatible with other systems, so that has nothing to do with the fact that we use our own.

## twisted.web.template - Why And How you Might Want to Use It

Twisted includes a templating system, `twisted.web.template`. This can be convenient for Twisted applications that want to produce some basic HTML for a web interface without an additional dependency.

`twisted.web.template` also includes support for `Deferred`s, so you can incrementally render the output of a page based on the results of `Deferred`s that your application has returned. This feature is fairly unique among templating libraries.

In `twisted.web.template`, templates are XHTML files which also contain a special namespace for indicating dynamic portions of the document. For example:

    <html xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
    <body>
      <div t:render="header" />
      <div id="content">
        <p>Content goes here.</p>
      </div>
      <div t:render="footer" />
    </body>
    </html>

template example - listings/template-1.xml
The basic unit of templating is `twisted.web.template.Element`. An Element is given a way of loading a bit of markup like the above example, and knows how to correlate `render` attributes within that markup to Python methods exposed with `twisted.web.template.renderer`:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    from twisted.web.template import Element, renderer, XMLFile
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('template-1.xml'))

        @renderer
        def header(self, request, tag):
            return tag('Header.')

        @renderer
        def footer(self, request, tag):
            return tag('Footer.')

element example - listings/element_1.py
In order to combine the two, we must render the element. For this simple example, we can use the `flattenString` API, which will convert a single template object - such as an `Element` - into a `Deferred` which fires with a single string, the HTML output of the rendering process.

    1
    2
    3
    4
    5
    from twisted.web.template import flattenString
    from element_1 import ExampleElement
    def renderDone(output):
        print output
    flattenString(None, ExampleElement()).addCallback(renderDone)

rendering snippet - listings/render_1.py

This short program cheats a little bit; we know that there are no `Deferred`s in the template which require the reactor to eventually fire; therefore, we can simply add a callback which outputs the result. Also, none of the `renderer` functions require the `request` object, so it's acceptable to pass `None` through here. (The 'request' object here is used only to relay information about the rendering process to each renderer, so you may always use whatever object makes sense for your application. Note, however, that renderers from library code may require an `IRequest`.)

If you run it yourself, you can see that it produces the following output:

    <html>
    <body>
      <div>Header.</div>
      <div id="content">
        <p>Content goes here.</p>
      </div>
      <div>Footer.</div>
    </body>
    </html>

rendering output 1 - listings/output-1.html
The third parameter to a renderer method is a `Tag` object which represents the XML element with the `t:render` attribute in the template. Calling a `Tag` adds children to the element in the DOM, which may be strings, more `Tag`s, or other renderables such as `Element`s. For example, to make the header and footer bold:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    from twisted.web.template import Element, renderer, XMLFile, tags
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('template-1.xml'))

        @renderer
        def header(self, request, tag):
            return tag(tags.b('Header.'))

        @renderer
        def footer(self, request, tag):
            return tag(tags.b('Footer.'))

tag manipulation example - listings/element_2.py
Rendering this in a similar way to the first example would produce:

    <html>
    <body>
      <div><b>Header.</b></div>
      <div id="content">
        <p>Content goes here.</p>
      </div>
      <div><b>Footer.</b></div>
    </body>
    </html>

tag manipulation output - listings/output-2.html
In addition to adding children, call syntax can be used to set attributes on a tag. For example, to change the `id` on the `div` while adding children:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    from twisted.web.template import Element, renderer, XMLFile, tags
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('template-1.xml'))

        @renderer
        def header(self, request, tag):
            return tag(tags.p('Header.'), id='header')

        @renderer
        def footer(self, request, tag):
            return tag(tags.p('Footer.'), id='footer')

attributes example - listings/element_3.py
And this would produce the following page:

    <html>
    <body>
      <div id="header"><p>Header.</p></div>
      <div id="content">
        <p>Content goes here.</p>
      </div>
      <div id="footer"><p>Footer.</p></div>
    </body>
    </html>

attributes output - listings/output-3.html

Calling a tag mutates it, it and returns the tag itself, so you can pass it forward and call it multiple times if you have multiple children or attributes to add to it. `twisted.web.template` also exposes some convenient objects for building more complex markup structures from within renderer methods in the `tags` object. In the examples above, we've only used `tags.p` and `tags.b`, but there should be a `tags.x` for each x which is a valid HTML tag. There may be some omissions, but if you find one, please feel free to file a bug.

### Template Attributes

`t:attr` tags allow you to set HTML attributes (like `href` in an `<a href="...`) on an enclosing element.

### Slots

`t:slot` tags allow you to specify "slots" which you can conveniently fill with multiple pieces of data straight from your Python program. The following example demonstrates both `t:attr` and `t:slot` in action. Here we have a layout which displays a person's profile on your snazzy new Twisted-powered social networking site. We use the `t:attr` tag to drop in the "src" attribute on the profile picture, where the actual value of src attribute gets specified by a `t:slot` tag within the `t:attr` tag. Confused? It should make more sense when you see the code:

    <div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1"
        t:render="person_profile"
        class="profile">
    <img><t:attr name="src"><t:slot name="profile_image_url" /></t:attr></img>
    <p><t:slot name="person_name" /></p>
    </div>

slots and attributes template - listings/slots-attributes-1.xml

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    from twisted.web.template import Element, renderer, XMLFile
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('slots-attributes-1.xml'))

        @renderer
        def person_profile(self, request, tag):
            # Note how convenient it is to pass these attributes in!
            tag.fillSlots(person_name='Luke',
                          profile_image_url='http://example.com/user.png')
            return tag

slots and attributes element - listings/slots_attributes_1.py

    <div class="profile">
    <img src="http://example.com/user.png" />
    <p>Luke</p>
    </div>

slots and attributes output - listings/slots-attributes-output.html

### Iteration

Often, you will have a sequence of things, and want to render each of them, repeating a part of the template for each one. This can be done by cloning `tag` in your renderer:

    <ul xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
        <li t:render="widgets"><t:slot name="widgetName"/></li>
    </ul>

iteration template - listings/iteration-1.xml

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    from twisted.web.template import Element, renderer, XMLFile, flattenString
    from twisted.python.filepath import FilePath

    class WidgetsElement(Element):
        loader = XMLFile(FilePath('iteration-1.xml'))

        widgetData = ['gadget', 'contraption', 'gizmo', 'doohickey']

        @renderer
        def widgets(self, request, tag):
            for widget in self.widgetData:
                yield tag.clone().fillSlots(widgetName=widget)

    def printResult(result):
        print result

    flattenString(None, WidgetsElement()).addCallback(printResult)

iteration element - listings/iteration-1.py

    <ul>
        <li>gadget</li><li>contraption</li><li>gizmo</li><li>doohickey</li>
    </ul>

iteration output - listings/iteration-output-1.xml

This renderer works because a renderer can return anything that can be rendered, not just `tag`. In this case, we define a generator, which returns a thing that is iterable. We also could have returned a `list`. Anything that is iterable will be rendered by `twisted.web.template` rendering each item in it. In this case, each item is a copy of the tag the renderer received, each filled with the name of a widget.

### Sub-views

Another common pattern is to delegate the rendering logic for a small part of the page to a separate `Element`. For example, the widgets from the iteration example above might be more complicated to render. You can define an `Element` subclass which can render a single widget. The renderer method on the container can then yield instances of this new `Element` subclass.

    1
    2
    3
    <ul xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
        <li t:render="widgets"><span t:render="name" /></li>
    </ul>

subview template - listings/subviews-1.xml

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    from twisted.web.template import (
        XMLFile, TagLoader, Element, renderer, flattenString)
    from twisted.python.filepath import FilePath

    class WidgetsElement(Element):
        loader = XMLFile(FilePath('subviews-1.xml'))

        widgetData = ['gadget', 'contraption', 'gizmo', 'doohickey']

        @renderer
        def widgets(self, request, tag):
            for widget in self.widgetData:
                yield WidgetElement(TagLoader(tag), widget)

    class WidgetElement(Element):
        def __init__(self, loader, name):
            Element.__init__(self, loader)
            self._name = name

        @renderer
        def name(self, request, tag):
            return tag(self._name)

    def printResult(result):
        print result

    flattenString(None, WidgetsElement()).addCallback(printResult)

subview element - listings/subviews-1.py

    <ul>
          <li><span>gadget</span></li><li><span>contraption</span></li><li><span>gizmo</span></li><li><span>doohickey</span></li>
    </ul>

subview output - listings/subviews-output-1.xml

`TagLoader` lets the portion of the overall template related to widgets be re-used for `WidgetElement`, which is otherwise a normal `Element` subclass not much different from `WidgetsElement`. Notice that the name renderer on the `span` tag in this template is satisfied from `WidgetElement`, not `WidgetsElement`.

### Transparent

Note how renderers, slots and attributes require you to specify a renderer on some outer HTML element. What if you don't want to be forced to add an element to your DOM just to drop some content into it? Maybe it messes with your layout, and you can't get it to work in IE with that extra `div` tag? Perhaps you need `t:transparent`, which allows you to drop some content in without any surrounding "container" tag. For example:

    <div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
    <!-- layout decision - these things need to be *siblings* -->
    <t:transparent t:render="renderer1" />
    <t:transparent t:render="renderer2" />
    </div>

transparent template - listings/transparent-1.xml

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    from twisted.web.template import Element, renderer, XMLFile
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('transparent-1.xml'))

        @renderer
        def renderer1(self, request, tag):
            return tag("hello")

        @renderer
        def renderer2(self, request, tag):
            return tag("world")

transparent element - listings/transparent_element.py

    <div>
    <!-- layout decision - these things need to be *siblings* -->
    hello
    world
    </div>

transparent rendering output - listings/transparent-output.html

## Quoting

`twisted.web.template` will quote any strings that place into the DOM. This provides protection against XSS attacks, in addition to just generally making it easy to put arbitrary strings onto a web page, without worrying about what they might have in them. This can easily be demonstrated with an element using the same template from our earlier examples. Here's an element that returns some "special" characters in HTML ('<', '>', and '"', which is special in attribute values):

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    from twisted.web.template import Element, renderer, XMLFile
    from twisted.python.filepath import FilePath

    class ExampleElement(Element):
        loader = XMLFile(FilePath('template-1.xml'))

        @renderer
        def header(self, request, tag):
            return tag('<<<Header>>>!')

        @renderer
        def footer(self, request, tag):
            return tag('>>>"Footer!"<<<', id='<"fun">')

renderers returning "special" characters - listings/quoting_element.py
Note that they are all safely quoted in the output, and will appear in a web browser just as you returned them from your Python method:

    <html>
    <body>
      <div>&lt;&lt;&lt;Header&gt;&gt;&gt;!</div>
      <div id="content">
        <p>Content goes here.</p>
      </div>
      <div id="&lt;&quot;fun&quot;&gt;">&gt;&gt;&gt;"Footer!"&lt;&lt;&lt;</div>
    </body>
    </html>

output containing "special" characters - listings/quoting-output.html

## Deferreds

Finally, a simple demonstration of Deferred support, the unique feature of `twisted.web.template`. Simply put, any renderer may return a Deferred which fires with some template content instead of the template content itself. As shown above, `flattenString` will return a Deferred that fires with the full content of the string. But if there's a lot of content, you might not want to wait before starting to send some of it to your HTTP client: for that case, you can use `flatten`. It's difficult to demonstrate this directly in a browser-based application; unless you insert very long delays before firing your Deferreds, it just looks like your browser is instantly displaying everything. Here's an example that just prints out some HTML template, with markers inserted for where certain events happen:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    import sys
    from twisted.web.template import XMLString, Element, renderer, flatten
    from twisted.internet.defer import Deferred

    sample = XMLString(
        """
        <div xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1">
        Before waiting ...
        <span t:render="wait"></span>
        ... after waiting.
        </div>
        """)

    class WaitForIt(Element):
        def __init__(self):
            Element.__init__(self, loader=sample)
            self.deferred = Deferred()

        @renderer
        def wait(self, request, tag):
            return self.deferred.addCallback(
                lambda aValue: tag("A value: " + repr(aValue)))

    def done(ignore):
        print("[[[Deferred fired.]]]")

    print('[[[Rendering the template.]]]')
    it = WaitForIt()
    flatten(None, it, sys.stdout.write).addCallback(done)
    print('[[[In progress... now firing the Deferred.]]]')
    it.deferred.callback("<value>")
    print('[[[All done.]]]')

deferred example - listings/wait_for_it.py
If you run this example, you should get the following output:

    [[[Rendering the template.]]]
    <div>
        Before waiting ...
        [[[In progress... now firing the Deferred.]]]
    <span>A value: '&lt;value&gt;'</span>
        ... after waiting.
        </div>[[[Deferred fired.]]]
    [[[All done.]]]

output from deferred example - listings/waited-for-it.html
This demonstrates that part of the output (everything up to "`[[[In progress...`") is written out immediately as it's rendered. But once it hits the Deferred, `WaitForIt`'s rendering needs to pause until `.callback(...)` is called on that Deferred. You can see that no further output is produced until the message indicating that the Deferred is being fired is complete. By returning Deferreds and using `flatten`, you can avoid buffering large amounts of data.

## A Brief Note on Formats and DOCTYPEs

The goal of `twisted.web.template` is to emit both valid HTML or XHTML. However, in order to get the maximally standards-compliant output format you desire, you have to know which one you want, and take a few simple steps to emit it correctly. Many browsers will probably work with most output if you ignore this section entirely, but the HTML specification recommends that you specify an appropriate DOCTYPE.

As a `DOCTYPE` declaration in your template would describe the template itself, rather than its output, it won't be included in your output. If you wish to annotate your template output with a DOCTYPE, you will have to write it to the browser out of band. One way to do this would be to simply do `request.write('<!DOCTYPE html>\n')` when you are ready to begin emitting your response. The same goes for an XML `DOCTYPE` declaration.

`twisted.web.template` will remove the `xmlns` attributes used to declare the `http://twistedmatrix.com/ns/twisted.web.template/0.1` namespace, but it will not modify other namespace declaration attributes. Therefore if you wish to serialize in HTML format, you should not use other namespaces; if you wish to serialize to XML, feel free to insert any namespace declarations that are appropriate, and they will appear in your output.

Note: This relaxed approach is correct in many cases. However, in certain contexts - especially <script> and <style> tags - quoting rules differ in significant ways between HTML and XML, and between different browsers' parsers in HTML. If you want to generate dynamic content inside a script or stylesheet, the best option is to load the resource externally so you don't have to worry about quoting rules. The second best option is to strictly configure your content-types and DOCTYPE declarations for XML, whose quoting rules are simple and compatible with the approach that `twisted.web.template` takes. And, please remember: regardless of how you put it there, any user input placed inside a <script> or <style> tag is a potential security issue.

## A Bit of History

Those of you who used Divmod Nevow may notice some similarities. `twisted.web.template` is in fact derived from the latest version of Nevow, but includes only the latest components from Nevow's rendering pipeline, and does not have any of the legacy compatibility layers that Nevow grew over time. This should make using `twisted.web.template` a similar experience for many long-time users of Twisted who have previously used Nevow for its twisted-friendly templating, but more straightforward for new users.

Index

Version: 13.0.0