Dave Taylor answers free tech support questions about a wide variety of business and technical topics, including blogging, iphone help, ipod help, AdSense, MySpace, Sony PSP help, Mp3 players, Windows XP, Windows Vista, Linux, SEO, Mac OS X, Facebook, Twitter and LinkedIn.

How can I unwrap or decode Web URL redirects?

I'm frustrated as heck! I clicked on a URL and next thing I knew I was on a completly different site that was quite, um, inappropriate for work. I know how to write a redirect in my own script, but I am wondering: how can I unwrap, decode, slow down or trace a sequence of redirects so I can see what's going on? If it helps, I have a Linux system I can use for the task.

Dave's Answer:

That does help, actually. In fact, the easiest way I know to understand a sequence of redirects is from the Linux (or Mac OS X) command line, using the splendid, powerful curl utility.

If you dig into it -- type "man curl" on the command line -- you'll find that it has about a zillion different things it can do that encompass not only Web site interaction and form hacking, but also FTP and more. Here's the description from the author:

"curl is a tool to transfer data from or to a server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE). The command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user authentication, ftp upload, HTTP post, SSL connections, cookies, file transfer resume and more. As you will see below, the amount of features will make your head spin!"

How can you not love that? :-)

To trace a series of redirects what you want to see are the http header sections. You can get that with the -I or --head flag. Add -L or --location for location related information (redirects) and you're good to go.

Here, again, is a quote from the man page: "(HTTP/HTTPS) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code) this option will make curl redo the request on the new place. If used together with -i/--include or -I/--head, headers from all requested pages will be shown."

So that's what we'll do. To test it, I'll use a redirect of my own, which you can first experience by clicking on http://dtlikes.com/stomp. Now, here's what happens when I ask "curl" to unwrap it:

$ curl -I -L http://dtlikes.com/stomp
HTTP/1.1 301 Moved Permanently
Date: Wed, 03 Sep 2008 18:45:46 GMT
Server: Apache
Location: http://dtlikes.com/stomp/
Connection: close
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Wed, 03 Sep 2008 18:45:47 GMT
Server: Apache
Location: https://member.stompernet.net/?r=26&i=68
Connection: close
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 302 Found
Date: Wed, 03 Sep 2008 18:45:56 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.2.6
Set-Cookie: PHPSESSID=hrjve2g3si1279hc5am49154r7; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: aff_id=114; expires=Sat, 03-Sep-2011 18:45:56 GMT; path=/; domain=.stompernet.net
Set-Cookie: tron_hash=7b2934d24a; expires=Sat, 03-Sep-2011 18:45:56 GMT; path=/; domain=.stompernet.net
Set-Cookie: aff_id=26; expires=Sat, 03-Sep-2011 18:45:56 GMT; path=/; domain=.stompernet.net
location: http://stse2.stompernet.net
Content-Type: text/html; charset=UTF-8

HTTP/1.1 200 OK
Date: Wed, 03 Sep 2008 18:46:04 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 03 Sep 2008 16:55:47 GMT
ETag: "f8851a-3cee-b5354ac0"
Accept-Ranges: bytes
Content-Length: 15598
Content-Type: text/html; charset=UTF-8

Phew! That's a lot of data, but when you look through it you can see that the original URL bounces you to an affiliate link, https://member.stompernet.net/?r=26&i=68, which then bounces you one more time, to "http://stse2.stompernet.net".

I hope that'll help you unwind what's going on with the other site you encountered!

Help others find this article at Del.icio.us, Digg, Netscape, Reddit, and Stumble Upon

Categorized: CGI Scripts and Web Site Programming , Unix and Linux Help (Article 8487)
Tagged: 301 redirect, 302 redirect, Linux, stompernet, url redirection, web server hacking
Previous: How can I learn how to use LinkedIn without leaving my house?
Next: Report from the floor of the CEDIA trade show

Subscribe!

Never miss another useful Q&A article again! Subscribe to AskDaveTaylor with Google Reader.

Comments

Rather amazingly, there are no comments on this article yet.

I have a lot to say, but ...

I have a lot to say, and questions of my own for that matter, but most of all I'd like to say thank you for all your efforts on this Web site by buying you a chai!

I do have a comment, now that you mention it!