How would I go about making a php script that does the following: You give various urls to check. Spider has to check all links on that domain & page besides a few links which are specified not to be searched. The spider checks for certain texts eg http://www.fixeddomain.com/********/*****.*** and saves them all.
How would I go round doing something like this? Thanks :)