Microsoft interview question

Graph theory problem (building a simple web crawler)