Profiling Django templates with Django-Debug-Toolbar

My last post about the speed of Django's templating language caused a bit of a stir and it was clear that people didn't really have a clue how long the templates were taking to render in their applications.

Enter Template-timings

Template-timings is a panel for Django-debug-toolbar (which everyone should be using, right?) that lets you see a breakdown of the rendering process. It currently gives you timing information on each block and template rendered in a convenient panel. It works on {% blocks %}, {% extends %} and {% include %}'ed templates.

Download

Check out the GitHub Repo for installation instructions

How does this work?

Its quite simple. This function replaces Template.render and BlockNode.render. It simply records how long the real render function takes.

Django's template rendering code is somewhat complex, and this is the best solution I could find. Its not perfect - there doesn't seem to be a way to tell what block belongs to what template for example, but it works.

Just how slow are Django templates?

Edit 2: I made a Django debug toolbar panel that profiles your Django templates and all their components. You can find it here: https://pypi.python.org/pypi/django-debug-toolbar-template-timings

Edit: It appears that Django does have some form of template caching. The graphs have been updated to include this.

I've always kind of known that Django templates are kind of slow compared to Jinja2 but I have never been bothered enough to test them head to head, until now. Note: This isn't an attack on Django, and if you haven't tried it already its well worth a go. Speed of execution is not everything, and template rendering is only one component, optimizing network and database time is often more fruitful.

Jinja2 or Django?

I have heard good things about Jinja2 with regards to performance, mostly because Jinja2 compiles templates down to Python bytecode during rendering, so the template only needs to be parsed once. Django does no such thing (as far as I know), and thus I would assume it would be slower.

Test Setup

I created two basic applications: howslow_django and howslow_flask. These applications time how long it takes to render 4 templates:

It tests the speed by timing how long it takes to render each template 100 times, then rendering it once while profiling (using the cProfile module). I believe these templates are enough to test the efficiency of each templating library, but the tests are synthetic - your mileage may vary. The stackoverflow homepage weighs in at over 200kb, this is included to see if any of the libraries make excessive copies of the template internally and to see if this effects the performance.

Hardware/Software

I ran the tests on my laptop and a Linux VPS that hosts this blog. My laptop runs Windows 8 and has 6gb of RAM and an i7 running at 2.4ghz, while the blog has 1.5gb of RAM and 2 virtual cores (running on an i7 920). The tests were run on cPython 2.7 with Django 1.5 and Flask 0.9 (the latest, at the time of writing). Debug was set to False in both applications.

Results

The results were surprising - Jinja2 completely destroyed Django in every test, and my VPS outperformed my development laptop. Django also used nearly 2,400 function calls to render a fairly basic template, which seemed a bit excessive.

Linux

Windows

Function calls

For Django the profile printout for the tutorial page can be found here: https://gist.github.com/orf/5153108

Conclusions

Django templates are slow compared to Jinja2 templates. This may be to do with Jinja2 compiling the templates or perhaps due to the excessive number of function calls Django makes while rendering templates. However, in my experience Django templates are fast enough. A lot of the time will be spent waiting for the database or other IO, so blazing fast templates are not absolutely necessary.

Finding fried chicken with C#, Mono and ServiceStack

I think I just solved a very first world problem - Where to find the nearest takeaway that sells fried chicken.

When you are stumbling out of Spiders at 1am in the morning and craving some SFC goodness the last thing you want to do is wander the backstreets of hull in the cold looking for an open takeaway. Clearly an app was needed that could display a list of open nearby takeaways that sells fried chicken with the fastest route to it. This was originally a joke project of mine but it became pretty involving, and I learned a lot that might be useful to anyone interested in C# web services, parsing HTML, reverse engineering web applications and running .NET on Linux with Mono.

Architecture

A lot of screen scraping was required so both a server and a phone app were necessary. The phone app should be as simple as possible: it would use the location service to find the users' co-ordinates then send them to the server, which would be responsible for finding chicken from a variety of different sources and returning the reduced results.

There are two main reasons for needing a separate server to handle the searching logic: the sources being searched can be changed or fixed without needing to issue an update on the phone marketplace. The searching itself is pretty network intensive, requiring many network calls which is not suitable on a phone network that is normally high-latency and low bandwidth.

The server would have to be modular (each source of restaurant information should be in a separate class) and when a client makes a search each of the modules are called in parallel and return information. The reduced results are then serialized and sent to the client.

Information Sources

I identified 3 main sources of information on local chicken establishments:

KFC

The KFC website was the simplest to implement: their site has a "Find your local KFC" section where you can enter your location and it will show a map with local KFC's and your distance from them. I opened up Chromes' development console (F12) to see how it was getting this information. After navigating to Network->XHR and refreshing the page I discovered it was making an ajax request to find the locations:

The users browser sends a request to a URL like this: http://www.kfc.co.uk/our-restaurants/search?latitude=53.7456709&longitude=-0.3367412999999715&radius=10 which returns a JSON array with information on local KFC's.

I used the extremely useful Json.Net library to parse the JSON response from this API. This library is brilliant when combined with C#'s dynamic keyword, for example the snippet below parses a response from the KFC website then writes all the found storenames and phone numbers to the console:

dynamic converted_data = JsonConvert.DeserializeObject<List<dynamic>>(kfc_response_data);
foreach (var entry in converted_data)
{
    Console.WriteLine(entry.storeName);
    Console.WriteLine(entry.phoneNumber);
}

Just-Eat

Just-Eat was possibly the hardest to implement and took me quite a while to crack. It only accepts postcodes, this means the webservice needs to translate the latitude and longitude that the phone application sends it into a postcode: this is called Reverse Geocoding. It is impossible to implement yourself with any degree of accuracy - thankfully there are a few services that offer this functionality, namely Yahoos' GeoFinder and Googles' Geocoding API. Word on the street as far as I could find is that Googles' API is superior quality, but must be associated with a "Google maps view". Yahoo however has a 'good enough' API with generous free limits. Bing also offer a Geolocation service - which I implemented - but I found that sometimes the service would not return any results until you made the request again. Screw it.

I coded a mostly working (if not a bit temperamental) module that scraped the HTML from the just-eat website using the HTMLAgilityPack library. HtmlAgilityPack has quite a nice API which revolves around using XPath queries to navigate through the document tree. The snippet below iterates through any li tag which has a class that contains the string 'cat':

var page = new HtmlDocument();
page.Load("http://www.mywebsite.com/afile.html");
foreach (var node in page.DocumentNode.SelectNodes(".//li[contains(@class,'cat')"))
{
    Console.WriteLine("li tag found: ID = {0}", node.Attributes["id"]);
}

Extracting information from websites by parsing their HTML is prone to breaking: in the example above if the web owner changed all his li tags to div's then the code would break and have to be fixed.

Soon after I got the screen scraping code mostly working I did notice that they had an iPhone app that didn't use their website. This means that the app must be getting its data from somewhere else, and it must be from some form of API. So I installed it on my girlfriends iPhone and configured its proxy settings to point to my laptop so I can inspect any network traffic the app sends. After opening the app and running the excellent Burp suite on my computer I saw the following requests were being made:

So it looked like Just-Eat did have an API after all. After having a poke around I found that they helpfully have a Web Services Defintion Language file which Visual Studio can use to generate client code. This code interacts with their API. Importing the WSDL file into VS2012 was easy: Simply right click a project in Visual Studio, select "Add Service Reference" and input "http://api.just-eat.com/MenuApi.svc?wsdl=wsdl0" into the Address field. Visual Studio discovers all of the methods available and the parameters and generates classes that match them. You can view the code that uses the API here, it really is much nicer to look at and much more maintainable than the HTML parsing version.

Just-Eat also doesn't give any geographical co-ordinates of its restaurants, only an address. Therefore to work out the distance from the user to the restaurants the address has to be turned into a latitude and longitude. I used Yahoo's Geocoding service to do this.

HungryHouse

HungryHouse was certainly much less effort to implement than Just-Eat. It offers a keyword search, so the server searches for "Fried chicken" and returns all open restaurants. Also the latitude and longitude of the restaurants is displayed on the page in the form of a map centered around a point, which means on average less Geocoding requests need to be made.

Mono

Deploying to mono was mostly painless, although I had to switch to an Ubuntu server rather than my preferred CentOS because it has a proper mono package that can just be installed by 'apt-get install mono-complete'. Mono comes with several useful tools that helped me test my code: xbuild and xsp4. XBuild is run in the root of your solution and it builds it, reading the .sln file, which meant I could just commit from my windows machine and it would build on Linux without any effort, which is nice. I did run into some dependency issues: nuget can be run on mono to download the dependencies automatically, but for some reason it didn't want to locate them after they had downloaded so they had to be manually added to the /bin file to work. Xsp4 is mono's development webserver which is run in the root of your ASP.net site to start serving, and while it can't handle production traffic it's handy while debugging.

One weird thing with mono is the web service reference I added for the Just-Eat source didn't seem to work. I had to remove the reference and use code that the linux 'wsdl' (a tool that comes with mono for generating C# code from WSDL files) program generated.

ChickenChecker should be hitting the marketplace soon. Watch this space.

HtmlToWord

You can find the code here on github and the package here on PyPi

I have written and continue to maintain a reporting system for a group of pentesters. During/after the tests the results and details are inputted into a web application using a WYSIWYG editor called Redactor (which is pretty awesome!) and the system generates a word document based upon this input which is then sent to the client. There doesn't seem to be a reliable way of inputting HTML into a Word document via COM (apart from simulating pasting HTML, which is too hacky and offers too little control) so I ended up writing this little library to do it for me, and I think it could be useful to someone else.

HtmlToWord is a Python library that takes HTML input (like that outputted from a WYSIWYG editor) and converts it to a stream of instructions that will render the HTML onto a Word document. It supports most common HTML tags (full list here) but doesn't support any form of line styles (yet?).

Example:

parser = HtmlToWord.Parser()
Html = '''<h3>This is a title</h3>
          <p><img src='http://placehold.it/150x150' alt='I go below the image as a caption'></p>
          <p><i>This is <b>some</b> text</i> in a <a href="http://google.com">paragraph</a></p>
          <ul>
              <li>Boo! I am a <b>list</b></li>
          </ul>'''

parser.ParseAndRender(Html, word, document.ActiveWindow.Selection)

This code will create a new Word document and fill it like so:

Its pretty neat I think - I can't be the only one with this kind of issue so I hope this library helps someone.

Adding mobile support to Simple

Last week I finally got round to adding support for mobile devices to Simple (the software that powers this blog). I thought I would write a quick post about getting a mobile version of your site up and running using Bootstrap from Twitter without changing much code at all.

When including the responsive version of Bootstrap in a page it exposes a few classes you can use to modify the displayed content of the page depending on if the device that is viewing the page is a computer, tablet or phone: visible-phone, visible-tablet, visible-desktop (and hidden-* counterparts). These classes work by using CSS Media Queries to detect the display size of the screen - you can have a play with the default bootstrap values here by re-sizing your browser window.

Integrating these with Simple was ridiculously easy: Add a hidden-phone class to the sidebar and other small elements and a visible-phone to a smaller header which appears above the post rather than as a sidebar. One thing that did stump me for a while is you have to add this tag inside the head of the page:

<meta name="viewport" content="width=device-width, initial-scale=1.0">

Once you have added that tag to the page head then making a mobile version of the site is as simple as modifying a few element classes. It should be noted that this is not the best way to make a mobile version of your site: the entire page is still downloaded to the phone and processed even if only a small portion of it is being displayed. This is a bit of a waste - building a proper mobile version of your site is preferable but this solution works great for Simple, who's pages are pretty small and almost all of the content is displayed on the mobile version.

I'm loving Bootstrap more every time I use it - its awesome and packed full of lovely features, and getting a mobile version of Simple working in under half an hour was pretty neat. Go visit this page on your phone for a demo.