Posted in Blogosphere, Linux, productivity, Technology and Software, tricks

cURL – Swiss Army Knife for Web Sites

Lately, I have been using cURL a lot to diagnose & get more information about web pages download times, header information, automation of such tasks etc. Therefore I thought some of the very useful but not so obvious usage of cURL.

first of all, if you want to download a copy of a page to a local file, you can use

curl http://www.example.com -o local_copy.htm

If you don’t want to neccessarily download the file but only want to see response headers, you can do

curl -I http://www.example.com

If the web site is returning a redirect, 30x HTTP response code, you can ask curl to follow the redirect for example

curl -IL http://www.example.com

You can specify multiple URLs to curl as part of pattern for instance:

curl -I http://{www,play,sawaal}.ibibo.com

Output:

[1/3]: http://www.ibibo.com –> <stdout>
–_curl_–http://www.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:05 GMT
Server: Apache
Vary: Accept-Encoding
ServerID: Myibibo-34
Content-Type: text/html

[2/3]: http://play.ibibo.com –> <stdout>
–_curl_–http://play.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: ibibo-WS
Cache-Control: no-transform
Vary: User-Agent,Accept,Accept-Encoding
ServerID: 25
Content-Type: text/html

[3/3]: http://sawaal.ibibo.com –> <stdout>
–_curl_–http://sawaal.ibibo.com
HTTP/1.1 200 OK
Date: Thu, 30 Jun 2011 18:19:08 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Set-Cookie: ASP.NET_SessionId=tu5c1y555oywv4fvxj1vj245; path=/; HttpOnly
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 112910

You can also specify advanced regular expressions when using the URLs for instance:

ftp://ftp.numericals.com/file[1-100].txt  ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)  ftp://ftp.letters.com/file[a-z].txt

Nested sequences are not supported, but you can use several ones next to each other:

http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order.

You can specify a step counter for the ranges to get every Nth number or letter:

http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt

More later in another post.

Posted in Blogosphere

Price Discrimination in India

Likethatonly has a great article on Price discrimination in India. He articulates very well some of the abnomalies in pricing for instance:

Convenience charges – It is quite true that vendors save cost with customers doing the booking online instead of doing physical transaction, In India, most of the time, as of now, you end up paying higher if you do the booking online. This is the case for train bookings, movie tickets etc.

Higher pricing for driving lessons for women – as the perceived value to women drivers is higher and as, very right mentioned in the article, they would likely be coming from rich family – Many driving schools have higher pricing for driving lessons for women.

Other examples such as “Peak rate mandis” & different prices for foreigners  for tourist attractions are also very interesting.

The last one “language” is also very interesting that shows price discrimination based upon the language.