django experiments projects rust security

I scanned every package on PyPi and found 57 live AWS keys

After inadvertently finding that InfoSys leaked an AWS key on PyPi I wanted to know how many other live AWS keys may be present on Python package index. After scanning every release published to PyPi I found 57 valid access keys from organisations like: Amazon themselves šŸ˜… Intel Stanford , Portland...

AWS may have broken your Cloudfront API for nearly a month

tl;dr : If you rely on the x-forwarded-for header with Cloudfront and have enabled Origin Shield, between October the 10th 2022 and November the 2nd 2022 the value of this header may have been incorrect for a percentage of requests. If your API relies on knowing the clients IP address in any way it ...

Infosys leaked FullAdminAccess AWS keys on PyPi for over a year

You can check out their website for a lot of buzwords , but itā€™s clear from all the stock photos that they take security Very Seriously Indeed ā„¢ļø. However, from what Iā€™ve found recently, it seems that Infosys use the following Comprehensive Management-Endorsed Proficiently Driven Cybersecurity Strat...

Visualizing how S3 deletes 1 billion objects with Athena and Rust

A few weeks ago I had the chance to delete 1 petabyte of data spread across 1 billion objects from S3. Well, actually 940 million, but close enough to the click-baitable 1 billion. I thought it would be interesting challenge to try and visualize the execution of these deletions and possibly gain som...

CVE-2022-0329 and the problems with automated vulnerability management

Update: Github have responded and said they will stop sending notifications about this CVE. Yesterday Github started notifying tens of thousands of people about a critical remote code execution vulnerability in a package named loguru. Their reviewed advisory is here. It references CVE-2022-0329 whic...

Benchmarking Rustpython With Criterion

Iā€™ve been very excited about RustPython since I saw the project being presented at FOSDEM 2019. Last week I finally got to contributing something to the project ! For context, RustPython benchmarks specific Python files against CPython, the ā€œreference implementationā€ for Python (read: the only real ...

Managing Rust bloat with Github Actions

Cargo and crates.io are an amazing part of the Rust ecosystem and one of the things that makes Rust so pleasant to work with. However, itā€™s quite easy for your cargo dependencies to pile up without you noticing. For example you might blindly add the (rather awesome) reqwest crate, not realizing that...

Distributing malware with Google images, service workers and vegan Twitter bots

What started with a simple image search for ā€œSkippers hatā€ quickly turned into investigating a clever malware distribution network utilizing service workers, which is somehow linked to a group of hardcore vegan twitter bots. The hat at the entrance of the rabbit hole was innocent enough: However, wh...

Django Docker Box is now an official Django project!

When I was working on adding queryset.explain() to Django I got annoyed by how complex it was to set up a local Django environment with multiple databases and versions. The traditional way of handling this was to use django-box which utilizes Vagrant to spin up a VM and install different types of da...

Testing my dotfiles with Github actions

I went through a phase that involved reinstalling MacOS way, way too often. At a certain point I got fed up with installing things over and over again and decided to version my dotfiles. And so https://github.com/orf/dotfiles was born. This reolved around a bootstrap.sh script that clones your dotfi...

I hope I hate this code one day

I remember the first program I built fully on my own: a music downloader. There was a site that would basically index random music files found on Google and present you with a way to search them. I guess they would periodically search google for things like intitle:index.of mp3 (which you can still ...

Europython 2019 Talk - Rewriting the Django autoreloader

Over two years ago I set out to try and improve Djangoā€™s autoreloader implementation. My PR was merged and the feature released in Django 2.2, and I thought I should at least mention it on my blog. The role of an autoreloader is conceptually pretty simple: When a developer changes a file your softwa...

Redesigning my blog

I like Hugo a lot. Itā€™s really fast and really simple. However itā€™s very much still evolving, and the theme I was using had not been updated in nearly a year. This meant that it broke with the new release of Hugo, and I was generally getting a bit sick of the design. Now Iā€™m not a designer by any me...

Creating an index of Apple Watch/MacOS/iOS features per-country

tl;dr: check out https://applewatchfeatures.com/ , https://iosfeatures.com/ and https://macosfeatures.com/ I recently brought an Apple Watch. Itā€™s a pretty fantastic product! One thing that quite annoyed me while evaluating if I should buy it is that features can vary quite drastically per country, ...

XCat 1.0 released or: XPath injection issues are severely underrated

Iā€™ve just released xcat 1.0 and itā€™s demonstration application after like 5 years of on-off development. Feels good! The genesis of xcat was when my boss, Sid, walked up to me out of the blue and asked if I wanted to go on an all expenses paid trip to Amsterdam. Who the hell wouldnā€™t say yes to that...

Invited to join the Django software foundation

A few days ago I was invited to become a member of the Django software foundation due to my contributions to Django. Awesome! Now I get to hang about in the super-secret mailing list and discuss django-related illuminati business. Removal of core developers The hot news in Django-land is the proposa...

Using bulk update in Django 2.2

My work on adding bulk_update() to Django has been merged and will be released in Django 2.2! Like my filtered aggregates feature it relies heavily on the wonderfully versatile CASE statement to achieve some pretty good speedups for certain use-cases. The development documentation gives a pretty cle...

Queryset.explain() released in Django 2.1

While working on any large-ish Django project you are bound to come across a slow query thatā€™s perhaps missing an index or doing something else expensive. My workflow for diagnosing this was to get the query that is being executed ( str(queryset.query) ) and paste it into a database shell, prefixing...

Filtered aggregates lands in Django 2.0!

Big Django projects often suffer from a few problems regarding database modelling and relations. Django provides incredibly easy to use tools to model your domain along with an awesomely powerful ORM to query on them. Often back office reporting software written in Django makes extensive use of the ...

How not to make an april fools joke

Picture the scene. Youā€™ve recently installed the awesome Lineage OS on your android device and are enjoying it so far. You install an update, it all goes smoothly, but upon restarting you are presented with this notification: What You press on the notification and youā€™re taken to the Lineage changel...

Suggestions added to Django manage.py

My recently merged PR for ticket #28398 adds very simple ā€˜did you meanā€™ suggestions to Djangoā€™s manage.py command, which is the primary way of interacting with Django from the terminal. So in Django 2.1 this is what you will expect to see if you misspell a management command:...

Goodby Simple, hello Gatsby

Many years ago, inspired by a blogging platform called Svbtle , I built a clone called Simple. It was pretty simple - just a minimal Flask app backed by a Sqlite database, but had some cool features. You could log in with Mozillaā€™s Persona (RIP), edit posts using an in-browser editor and drag ā€˜nā€™ dr...

Counting Pizza with Python

Iā€™m a full time nerd, even when Iā€™m ordering pizza online I canā€™t stop myself from investigating how the websites Iā€™m ordering from work. My latest investigation was Dominoes where I found a neat way to count the number of orders that they process throughout the day. This post is supposed to highlig...

Syntax highlighting and CSS support added to wordinserter

I recently added syntax highlighting and support for CSS stylesheets to wordinserter , and the implementation was satisfying enough that I thought I would blog about it. Wordinserter is a library I maintain that lets you insert HTML documents/snippets into Word documents: Itā€™s primary use case is wh...

Segfaulting Python with afl-fuzz

American Fuzzy Lop is both a really cool tool for fuzzing programs and an adorable breed of bunny. In this post Iā€™m going to show you how to get the the tool (rather than the rabbit) up and running and find some crashes in the cPython interpreter. Fuzzing? Explaining in detail what fuzzing is would ...

Scraping websites with Cyborg

I often find myself creating one-off scripts to scrape data off websites for various reasons. My go-to approach for this is to hack something together with Requests and BeautifulSoup , but this was getting tiring. Enter Cyborg , my library that makes writing web scrapers quick and easy. Cyborg is an...

HtmlToWord is now WordInserter

Iā€™ve released a redesign of my HtmlToWord library, specifically it now supports Markdown and multiple different ways to interact with Word. Itā€™s now also been renamed to WordInserter to reflect this. Originally HtmlToWord was designed to take HTML input, process it and then insert a representation o...

HP Support Solutions Framework Security Issue

After discovering the flaw in Dellā€™s System Detect software I looked into other similar software for issues. This post details two issues I found with the HP Product Detection software and explores the protections HP put in place. Iā€™m also going to explain how they could be easily bypassed to allow ...

Dell System Detect RCE vulnerability

I recently discovered a serious flaw with Dell System Detect that allowed an attacker to trigger the program to download and execute an arbitrary file without any user interaction. Below is a summary of the issue and the steps taken to bypass the protections Dell put in place. Timeline: The issue wa...

Simple 2

Iā€™ve just about finished the next version of Simple , the markdown based blog that powers this site. When I first made Simple it was because I disliked WordPress, which seemed a bit too bloated. Then I saw Svbtle and I really liked the minimalist design (mostly the posting interface ) and decided to...

Exploiting XPath injection vulnerabilities with XCat

I just released XCat 0.7 , the companion tool to this paper. XCat is a command line tool to automate the exploitation of Blind XPath Injection Vulnerabilities , utilizing some pretty cool techniques. The most interesting technique is that xcat can automate out of band attacks to massively speed up e...

A test RSS feed service

The coursework set for my Distributed Systems involves reading new items from RSS feeds (such as the BBC News feed or the UK traffic incident feed). To help me build the system I developed a simple service that serves up RSS feeds that are regularly automatically updated with nonsense items, and it ...

2 years of blogging

When I first came to University lots of people (like Rob Miles ) were trying to get undergraduates to start blogging. On the 6th of March 2012 I registered this domain and started blogging, getting myself added to the awesome Hull Compsci blogs syndicate. That was two years ago and a lot has changed...

Opera is really nice

I really like the Opera browser. A couple of months ago I got a bit tired of using Google Chrome, it was just a bit sluggish sometimes and I fancied a change. So I switched to Firefox, which I enjoyed using for a month or so until it too became irksome - it used a hell of a lot of memory and was als...

Submitting a patch to Pythonā€™s lxml library

While working on a system for work I ran into a bug with Pythonā€™s lxml library and decided to fix it. I thought I would document how easy the process was, hopefully to encourage others to contribute to open source projects. Lxml is a ā€œpythonic binding for the libxml2 and libxslt librariesā€ which put...

University Presentation

So I did a presentation on Information Security at University today. I think it went rather well, however I couldnā€™t show a couple of the demonstrations due to some SkyDrive files only being available online. That sucked because those were my best demonstrations, but overall I was happy. A few peopl...

Hack things, acquire clothing.

I discovered an XSS flaw in a website a month ago and reported it to the owners. As a thank you they sent me a hat, a rather large american sized t-shirt and a pair of ā€œDeFeetā€ socks (guaranteed to stay cooler and drier than any other brand). I didnā€™t expect them to ship something like that overseas...

Restricting Thrift clients to specific IP addresses with Twisted

Apache Thrift is pretty awesome - you can build Twisted bindings for your Thrift interface file that work fantastically. There is one thing that took me a while to figure out: I want to restrict clients connecting to the service to a specific set of IP addresses stored in a database. There were thre...

Adding tail-call optimization to Python

Tail-call optimization is a trick many languages and compilers use to avoid creating excess stack frames when dealing with recursive code like this:...

My Uni's timetable system sucks, so I built a better one.

tl;dr The timetable system sucks, so I made one that works Getting your timetable sorted at Uni has never been fun. In years 1 and 2 of my study the department posted a timetable for each year showing all modules and students were expected to remove the classes they did not take, which while not the...

Purchasing a Ā£30,000 numberplate for the price of a bus ticket

Regtransfers.co.uk is a website that allows you to purchase customized numberplates for your car or motorbike. They boast a large number of famous clients and short numberplates are often on sale for upwards of Ā£20,000 (the plate ABC 4 is up for Ā£30,000). While playing with their site I discovered a...

Breaking out of secured Python environments

A week or so ago I was browsing /r/Python and I saw a link to a website called rise4fun.com , which is a Microsoft Research project that contains a lot of cool demos and tools that you can run in your browser. The demo I was linked to was a restricted Python shell that could be used to experiment wi...

Inspecting .NET applications with ILSpy

Every once in a while I come across an application that is so comically insecure that I feel the urge to blog about it. The application in question is a .NET application to manage care homes and provide a Medical Administration Record for residents. Staff login to the app using a username and passwo...

Automatically inline Python function calls

Edit: Code is here on GitHub Calling functions in Python can be expensive. Consider this example: there are two statements that are being timed, the first one calls a function that returns an integer while the second one calls a function that returns the result of a second function call which return...

SSDs are awesome, buy one.

I recently brought a Samsung 840 Series Pro 256GB 2.5 inch SATA Solid State Drive and its easily the best PC hardware purchase I have ever made. Before I purchased it I was thinking about replacing my laptop as it was getting pretty sluggish, despite having decent specs even 2 years after I purchase...

Displaying a processes output on a web page with Websockets and Python

A few days ago a colleague of mine asked me how you would pipe the standard output of a process into a browser. I hacked around for a few hours and came up with a websockets based solution (using Twisted and Autobahn.ws ) that you can see below ( Your browser needs to support WebSockets, sorry IE9 a...

Profiling Django templates with Django-Debug-Toolbar

My last post about the speed of Djangoā€™s templating language caused a bit of a stir and it was clear that people didnā€™t really have a clue how long the templates were taking to render in their applications. Enter Template-timings Template-timings is a panel for Django-debug-toolbar (which everyone s...

More holes than swiss cheese

tl;dr Never trust user input. Note: Before I published this I contacted the owner of the site and he has since replaced it with a new and more secure version When I browse the net I often see the hallmarks of a security issue with websites that I visit, and little alarm bells go off in my head. I wa...

Just how slow are Django templates?

Edit 2: I made a Django debug toolbar panel that profiles your Django templates and all their components. You can find it here: https://pypi.python.org/pypi/django-debug-toolbar-template-timings Edit: It appears that Django does have some form of template caching. The graphs have been updated to inc...

Finding fried chicken with C-Sharp, Mono and ServiceStack

I think I just solved a very first world problem - Where to find the nearest takeaway that sells fried chicken. When you are stumbling out of Spiders at 1am in the morning and craving some SFC goodness the last thing you want to do is wander the backstreets of hull in the cold looking for an open ta...

HtmlToWord

You can find the code here on github and the package here on PyPi I have written and continue to maintain a reporting system for a group of pentesters. During/after the tests the results and details are inputted into a web application using a WYSIWYG editor called Redactor (which is pretty awesome!)...

Adding mobile support to Simple

Last week I finally got round to adding support for mobile devices to Simple (the software that powers this blog). I thought I would write a quick post about getting a mobile version of your site up and running using Bootstrap from Twitter without changing much code at all. When including the respon...

Parsing Wikipedia database dumps with C-sharp and Postgres (6 degrees of Wikipedia)

tl;dr C# and Postgres are pretty damn fast. View code on github Recently I began working on a little experiment after I saw that Wikipedia offers XML dumps of their entire database for people to use. I wanted to create a website where users could enter two article titles and it would compute the sho...

Transplanting/Replacing Django child instances without deleting the parent

Django has a very neat feature called Multi Table Inheritance which allows you to create a ā€˜parentā€™ model with common fields and a variety of ā€˜childā€™ ones with specific fields. For example:...

Using Python metaclasses to make awesome Django model field choices

Edit: This code is now on PyPi: https://pypi.python.org/pypi/django-choice-object tl;dr Metaclasses are awesome When using Djangoā€™s Model or Form frameworks you can define a fixed set of choices for fields which are list of tuples containing a value and some text to associate with that value. The do...

In favour of ORM's

I recently read this post entitled ā€œ(Some) ORM Haters Do Get Itā€ and I wanted to make a few points in favour of ORMā€™s. The author of the post argues that ORMā€™s are bad because of impedance mismatch. I donā€™t know enough about that subject to comment on it but I have been using ORMā€™s long enough to th...

Making a film recommendation site by cheating

View the site here: https://movies.tomforb.es or the code here To distract me from my ever approaching 1st year exams I decided to create a site that recommends people films to watch based upon their previous viewing habits. I gave it to my girlfriend to use and evaluate and she gave positive feedba...

Creating a URL shortening service with Django

View it live here or get the code here The first URL shortening site I saw was several years ago and was called TinyURL. Soon after Twitter gained popularity a whole slew of them popped up ( bitly , tiny.cc , is.gd ) to cater for the masses constrained by Twitters 140 character limit, but a lot shut...

Using a custom SQLAlchemy Users model with Django

I really dislike Djangoā€™s ORM. For my job I have written (and continue to maintain) a large internal project that uses Djangoā€™s ORM, templating language and MVC framework to serve requests, and I made the unfortunate mistake of sticking with Djangoā€™s ORM instead of using the much more powerful SQLAl...

Draconian internet filters

My universities student network is pretty restricted. I just finished coding a few changes to Simple and realised I couldnā€™t push any changes to GitHub due to port restrictions. It appears that they block almost all ports bar 80 and 445 via TCP, which is fine for most users but is quite annoying for...

Simple.

I like things to be simple. So I wrote my own blog software to replace the rather un-simple WordPress. Its not that WordPress its hard to use or install, far from it, Its just got a lot of bloatware in my opinion, so I replaced it with Simple. Simple uses MarkDown to format posts, an aims to be as s...