The complex world of Domains
Domain names are everywhere. They are now an integrated part of our society, every business should have one and without them, you're not online. So how can the world of domains be complex? I go to my registrar, I buy a domain and hey presto I can have a website.
But my question is - how much do we really know about the world of domains? (As a techy you would hope I know a fair bit!) Lets pose a question. Can you give me a complete list of all the domain names currently in the world?
Sure! First place - Google. You'll discover that domain names use something called Zone Files. Another quick Google. You'll soon discover that its not actually that easy to get hold of them! (You can freely register for ones such as com,net,us,org etc). So the challenge is on - how can one compile a list of all active domain names in the world? Second place - list providers. There are a number of different providers who provide anywhere betweeen 230 and 255m domains. Not a bad find you think. Well firstly consider the cost: anywhere from $80 to $499. But next, consider the scale. According to Verisign and their quarterly domains brief, there are currently 339 million domain names in operation. Ah - so 255m doesn't sound too great (aprox 75%).
Instantly we discover that the world of domains is a bit more of a mystery than we first thought! But lets say you decided to take the (free) .com list as your source and start tracking down other domain names. You'll soon discover thousands of Chinese (cn) and German (de) domain names match up (in a few days I discovered about 600k worth of domains across a number of TLDs (top level domains)). But then you discover (because you know from your own) that domain names expire. Now that long list you've discovered needs to be maintained! But how??
You have 2 options. Firstly you could do a whois lookup. Whois provides free information about who owns a domain name. You could work out when they expire and then periodicially check. Great idea until you find out that many providers (such as de and many European registrars) don't provide expiry dates. Then you'll discover that for those that do, impose limits on how many domains you can check (the UK allow 1000 per day with a total list of 11m domains).
The other option is monitoring to see if the domain name is active. Sure it can be done, but the minute you've checked it your data becomes old. In fact you must continously monitor the domains otherwise your list will simply be invalid.
Why am I telling you all this? Firstly because for the last few years it has been an interesting project of mine and secondly to reflect on why we actually know very little about the whole world wide web. No one knows all the domains in the world (probably except Google) and when you then throw in subdomains the picture is enormous. Should you want to maintain a database of domain names don't - unless you have a lot of money and multiple (hundreds) of servers to manage them (I estimate you would need 30 servers just to maintain the UK domains).
If however you require a list of all the domain names AND have a lot of money to spend on the project, then please get in touch!