Categories
Programming Security web-design

Open Redirects

In this post I’ll discuss an issue I tackled a short while ago – open redirects. But first, the story of how I got to it. Feel free to skip ahead to the technical discussion.

Background

Our analytics for plnnr.com – the website for trip planning wasn’t working as well as we wanted. We’re using Google Analytics, and it’s hard generating the specific report we want, and when we did get it, it seemed to show inaccurate numbers. To partially alleviate the issue, I was required to add tracking pixels for facebook & adwords, so we can better track conversions.
For us, an “internal” conversion is when a user clicks on a link to a booking url (for a hotel, or any other “bookable” attraction).
After reviewing it, I decided that the best course of action would be to create an intermediate page, on which the tracking pixels would appear. Such a page would receive as a parameter the url to redirect to, and will contain the appropriate tracking pixels.

Description of problem

Let’s say we build the url for the page like so:

/redirect/?url=X

This page will load the appropriate tracking pixels, and then redirect to the given url (the X).
The problems are:
1. We are potentially exposing ourselves to Cross Site Scripting (XSS), if we don’t filter the redirect url correctly. A malicious website could create links to our page that will run scripts in our context.

2. A malicious webmaster could steal search engine authority. Let’s say he has two domains: a.com and b.com, of which he cares about b.com. He creates links on a.com to:

ourdomain.com/redirect/?url=b.com

A search engine crawls his page, sees the links to his domain, and gives ourdomain.com authority to b.com. Not nice.

3. A malicious website could create links to ourdomain.com that redirect to some malware site, this way harming the reputation of ourdomain.com, or creating better phishing links for ourdomain.com.

Possible solutions

Before we handle the open-redirect issues it’s important to block cross site scripting attacks. Since the attack might be possible by inject code into the url string, this is doable by correctly filtering the urls, and by using existing solutions for XSS.

As for the open redirect:

1. Non solution: cookies. We can pass the url we want in a cookie. Since cookies may only be set by our domain, other websites would not be able to set the redirect url. This doesn’t work well if you want more than one redirect link, or with multiple pages open, etc.

2. Checking the referrer (“referer”), and allowing redirects to come only from our domain. This will break for all users who use a browser that hides referrer information, for example, those using zone-alarm. Google also suggests that if the referrer information is available, block if it’s external. That way we are permissive for clients that hide it.

3. Whitelisting redirect urls. This solutions actually comes in two flavors – one is keeping a list of all possible urls, and then checking urls against it. The other is keeping a list of allowed specific url parts, for example, domains. While keeping track of all allowed urls may be impractical, keeping track of allowed domains is quite doable. Downside is that you have to update that piece of the code as well each time you want to allow another domain.

4. Signing urls. Let the server keep a secret, and generate a (sha1) hash for each url of “url + secret”. In the redirect page, require the hash, and if it doesn’t match the desired hash, don’t redirect to that url. This solution is quite elegant, but it means that the client code (the javascript) can’t generate redirect URLs. In my case this incurs a design change, a bandwidth cost, and a general complication of the design.

5. Robots.txt. Use the robots.txt file to prevent search engines from indexing the redirect page, thereby mitigating at least risk number 2.

6. Generating a token for the entire session, much like CSRF protection. The session token is added to all links, and is later checked by the redirect page (on the server side). This is especially easy to implement if you already have an existing anti-csrf mechanism in place.

7. A combination of the above.

Discussion and my thoughts

It seems to me, that blocking real users is unacceptable. Therefor, only filtering according to referrer information is unacceptable if you block users with no referrer information.
At first I started to implement the url signing mechanism, but then I saw the cost associated with it, and reassessed the risks. Given that cross-site-scripting is solved, the biggest risk is stealing search-engine-authority. Right now I don’t consider the last risk (harming our reputation) as important enough, but this will become more acute in the future.

Handling this in a robots.txt file is very easy, and that was the solution I chose. I will probably add more defense mechanisms in the future. When I do add another defense mechanism, it seems that using permissive referrer filtering, and the existing anti-csrf code will be the easiest to implement. A whitelist of domains might also be acceptable in the future.

If you think I missed a possible risk, or a possible solution, or you have differing opinions regarding my assessments, I’ll be happy to hear about it.

My thanks go to Rafel, who discussed this issue with me.

Further reading

* http://www.owasp.org/index.php/Open_redirect
* http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=171297
* Open Redirects and Phishing Vectors/

Categories
Javascript Optimization Programming Projects startup web-design

Javascript Element Creator

Some time ago I was working on optimizing the client side code of my website, plnnr.com, an online trip planner.
This website does automatic trip planning, and the problem was that recalculating trips was slow. After profiling, I found out that most of the time wasn’t actually taken up by the algorithm, but by the UI. Rendering the trip to html was the costly part. The process was like so:

Client-side Javascript code generates new trip prefs -> application calculates new trip -> Client-side Javascript gets the new trip, and creates new html.

It’s important to note that the app is “ajax based”, so the actual trip html was generated by the Javascript code, and not the server. At the time I was using Mochikit to generate the new html. Mochikit has a pretty nifty API for generating html, but it’s quite a bit on the slow side. Basically, this API is a wrapper around createElement.

Well, first I did a little test, and found out that generating html with cloneNode and innerHTML is much faster than createElement. Still, there was a problem – I needed to generate many similar elements – similar but not identical. Consider entries on a trip itinerary – they all look the same, yet each one has a different name, a different time string, and a different onclick event.

What I needed was a Javascript based html template library. My requirements:
1. Speed. Html had to be generated quickly.
2. Expressiveness. It had to be able to create pretty arbitrary html with a given context. For example, an anchor element (<a> tag) with a given href property, and a given text content.
3. References to inner elements: Many elements inside the generated html need various events attached to them, or various code processing. This should be easy to achieve.
4. The library has to allow the template html to be written as html, and not only as javascript strings.

So, I sat down with Benny, and we wrote the Javascript Element Creator, which we are now releasing under the BSD license. I originally wrote it to work with Mochikit and the Sizzle library, and Benny changed his version to worked with jquery.

After adding the code to my project, I got two things: first, everything worked much, much faster. Second, it was much easier changing the generated html when it was generated according to a template, and not directly in code.

Instructions

1. Write your html somewhere visible to the javascript code. Add the “template” class to the upper node, and the id will be the name of the template. For example:

<body>
    <div id="some_div" class="template">
    </div>
</body>

2. Similarly to other template engines, add double brackets to signify where text should be inserted:

<body>
    <div id="some_div" class="template">
        <a href="[[link_url]]">[[link_text]]</a>
    </div>
</body>

3. Create a creator object. It will “collect” your template, and will make it available to your code.

var creator = new ElementCreator();

4. Generate your DOM object, and add it to the document;

var obj = creator.generate("some_div",
                           {link_url: '/url/',
                            link_text: 'hello world'});
appendChildNodes(foo, obj);

The code

We decided to publish for now only the jquery version. I might publish the mochikit version as well at a later date. Since Benny wrote the jquery version, he also wrote the tests for that version.

All in all, the final code is pretty short, and could probably be even shorter. Still, it’s good enough, and gave me a very real performance boost.

Here is the code, have fun with it.

Categories
web-design

PNG Minification

Following Ned Batchelder’s advice I’ve used pngout to minify the png’s on my startup’s web page.

It took them down from 641,281 bytes to 338,705. This is quite a nice return for the effort of a download and a single command line:

for %i in (*.png) do pngout “%i”

Categories
Javascript Programming startup web-design

Mochikit Drag&Drop Corner Case

I found myself working again on the UI for my startup. As my Javascript library, I use Mochikit. One of the reasons for that is that it’s the Turbogears builtin, and I came to like it. The other is that it’s really easy to create DOM objects with it.

In any case, Mochikit has really easy support for good looking drag&drop. However, as usual, my requirements were strange enough to fall upon the following corner case:
I wanted to add a “tool tip popup” for some text, where I would display pertinent information to said text. To make the tool tip popup thingy work, I used the following css “on mouse over” visibility trick:
[css]
.tooltip {
display: none;
}

.parent_object_class:hover .tooltip {
display: block;
}
[/css]

This works beautifully, and with a little bit of positioning, and maybe an event here and there, you can make it appear where you want.

Cue the drag&drop. I wanted to add some drag&drop based slider to that tool tip. Since I wanted to limit the “draggability” of the slider’s selector, I used the snap argument for Mochikit’s Draggable object so that if you move the mouse too far, the dragged selector stays at the limit of a predefined area.
This was all very well, and both of the tricks described worked pretty fine separately, until I tried to put them together.
When dragging and leaving the allowed area for the drag, because of the snap argument, the dragged object stays back, and mouse is no longer over a child element of the original tooltip and tooltipped text. This means that the css trick no longer applies, and the tooltip loses visibility. This would have been fine if the drag ended there. However, the drag was not ended, and at each move of the mouse, the coordinates would grow more. Since I use the drag coordinates to compute the result of the drag, I got some pretty strange results.

To work around this behavior, I used Draggable’s starteffect and endeffect optional arguments to make sure the tooltip remained visible, thus avoiding this issue.

Still, there were many other issues with all this drag&drop going around, and I decided to go for a simpler design, and not put in more time on this.
Issue sealed with a Keep It Simple Stupid.

Categories
CSS Javascript startup web-design

Website development and not supporting Internet Explorer 6

My partner and I started to work on a website a few months ago. We have a working prototype, and we are always improving it. My work is mostly concentrated on a smart Python backend, and on a Javascript front-end, while a thin controller acts as a messenger between the two.
Lately, I’ve worked on improving the UI. As expected, I rely heavily on CSS. I generate a lot of html elements using Mochikit and format them with CSS classes. While obviously better than the old alternatives, I still don’t like CSS. Maybe it’s because I don’t understand it deeply enough, but for me, there is still a lot of voodoo involved. An example I found, which luckily I didn’t run into yet, is collapsing margins.

Still, even with all its voodoo, CSS is bearable. At least until you get to IE. My latest run in with it was a scrolling bug, and I ran into many other issues. However, as much as I complain, I’m probably getting it easy, as when we started work, we decided not to support IE 6, at least until required.
Our reasoning was:

  • Developing for IE6 both independently and consistently with other browsers has a high cost attached to it.
  • IE6’s use rates are declining, and will decline even more by the time we launch (See these statistics for example).
  • Our first versions were mostly required as a prototype to prove our technology to potential investors.
  • As a two-men team, and a one man programming team, we are very low on development resources.

Given my latest bout of UI programming, this choice made me just a little bit happier.

Categories
Programming Philosophy Security web-design

Breaking Rapidshare's Annoying Captcha the Easy Way

Like many others, I got stuck in front of Rapidshare’s captcha. After more than five attempts at reading different letters with kittens and other critters hidden behind them, I was thinking of giving up. Especially because each time I failed I had to wait a half a minute again. However, in one instance I went *back* via my browser, and tried solving the same captcha again. Turns out this works, and I got the file.

I know I could probably have solved it in a smarter fashion, but it wasn’t worth the effort.

My lesson:

When someone writes crappy software, their software is probably crappy in more than one way.

This is not the first time I’ve seen this happen.

Categories
Programming Python web-design

LStrings and ASCII-Plotter at UtilityMill.com

It seems Gregory is fond of ascii-art, as he took my scripts, and put them online at utilitymill.com.

UtilityMill.com is a cool new website, which allows you to run scripts as small web applications. What Gregory actually did, with just a few key-strokes, was turning my two scripts into web applications. While these scripts aren’t the most useful web applications in the world, I could think of a few others that are.

Gregory also thought of another use for the UtilityMill. Consider my post ‘Rhyme and Reason with Python’. Let’s say you want to try out my script as well, but you don’t want to go through the hassle of downloading it and all the required dependencies, and then figure out how to run it. If I set it up as a small web application there, I could have any reader be able to see the code and its output without any effort. I’ll be sure to try something like that in my next code-oriented post.

Categories
Security web-design

Troubles with Wild Themes

Some time ago, I wrote that I was planning on using a new theme for this blog. To do this, I first looked for possible candidates on themes.wordpress.net, and then started to adapt the one I liked. However, while working on the theme, I noticed hidden links in the code of the theme.

These links were hidden by using “font-size: 1px”. Hidden links are there to increase the search engine placement of the creator and his affiliates. In this case the creators were ‘wildconcepts.net‘. You can check their stats in technorati, and see they have about 250 blogs linking to them, mostly via regular credits.

Upon further examination I found two more themes by these guys, with the same hidden SEO links.
Afterwards, I checked some 20 blogs that had their themes, about half of which still had the SEO (Search Engine Optimization) links in them. Other linked urls were kianah.com (a blog) and ads-ph.com (an ad exchange service).
I reported this to themes.wordpress.net, and it seems I can no longer find the themes via their search engines. However, the themes are still available for download.

The SEO links, while being an annoyance and an underhanded thing to do, are not the main issue here. The big problem I had with this theme was, that if someone has no compunction about putting SEO links, he\they might put a backdoor there as well. I’m not saying that these guys did it, but I know that if I was a bad guy looking to make money – I’d do it.
This is a very easy way to infect servers. Just prepare a few good-looking themes for wordpress, phpbb or any other standard web-application, sit back, and watch the botnets grow. You don’t only get to infect the server, you can try to infect any client that connects to an infected server. Instead of researching a new vulnerability, just use social engineering, like you do with end users surfing the web.

You can publish your themes, and evidently, they won’t go through too much scrutiny. With an appropriate Google Alert in place – you can also be informed whenever someone new installs your theme.

This is a trust issue – and it seems that you shouldn’t trust WordPress’ theme DB. While this may seem obvious to you, it wasn’t to me at first, and I bet it isn’t obvious to many others starting their small blogs and looking for a good theme.

Categories
Game Development Personal Programming Python web-design

Misc. Projects: a preview for the new (academical) year

So now it’s been too long since I’ve written a post, and that’s mostly because I haven’t been programming lately, as I’ve been busy with buying a car\moving\studying. This is going to change now that I’m finally settling in.

There are however few directions I’m going to go about, and I’d like to mention them here. If you are interested in reading more about any of them, just drop me a line in the comments, and that direction will get a bit more attention.

The first direction (and the most serious one) is a website I’m building. I’ll still not talk about the website itself, at least until a beta version is done, but I will say I’m doing it with TurboGears. I’ve always looked for a good Python framework for writing websites, and for now, TurboGears seems to do the trick, although I’m just starting with it. I am going to write a little bit about my experience with TurboGears, and my opinions. Stay tuned.

Another subject that I thought of writing about is a small tank game I wrote with pygame. This game actually works, although I didn’t do the finish. I intended to make it multi player (over the ‘net) but then found out that it is really not that simple. So I intend to work it out sometime in the future, and write about it. For now, I will say a few words about the game. It started when a friend of mine said that he wanted some simple tank game for one-on-one games, with some certain behaviors (such as bullets ricocheting off walls). I heard him talking about it, and told him, “hey, that’s really simple, I can do that”, so I went home, and during five days, one hour a day, I worked it out. That was an interesting experience in itself. Maybe I’ll write a little about that as well.

Other small projects I’ve been working on include doing some more IMDB mining (which I started to talk about), and maybe expanding that to allmusic.com as well. One thing I find missing in Amarok, is choosing music according to musical relations between singers and bands. Maybe I’ll even write a plugin :)

Some other ideas I had include 3d l-strings. I actually implemented those, but to my disappointment, they didn’t turn out as aesthetically pleasing as I’d hoped. Still, it was a curious result (with PyOpenGL doing the drawing). I wanted to implement more simple fractals. I’ll probably write more of those when I break from writing more serious projects. Other small ideas I had include more ascii-art, for example, ascii-art drawing of some vector image format seems very interesting, or maybe a 3d engine (maybe with cell-shading) which renders the result in ascii-art, and then writing an FPS with it… The options are endless!

All of this of course if I have the time when I’m not working on one of the other current projects, which include distorm, and my studies. Among the courses I’m taking there is a seminar on graph algorithms, a course in compilation and a course in cryptography.

This is going to be a very interesting year indeed!

Categories
Origami web-design

Origami section added

I wanted to add this section for quite some time now. There are a few foldings I created, which I wanted to put online. Now the first page is ready.

I also wanted to create the web-page myself, so I had to learn a bit of JavaScript, html, css, and other cuss-words. Indeed, JavaScript seems to me a bad language. There is no printf equivalent! I tried looking for it on the web, and all I could find were partial implementations by people around the world. Did I say these implementations were partial?

Nevermind. So here is the new origami page, soon to be filled with many wondrous creatures.

Here is a preview of the first fold waiting there: