The only way I can immediately identify to optimize it is to hard code the ignore list opposed to reading an array every time, but that advantage will probably be negligible at best. It takes PHP a long time to do these things, that might just be a scripting language thing.
This is why I hate shared hosting, you do not have the ability to make programs for the server to run that execute more tedious tasks like the one you have here. I wrote a VB application to do something fairly similar and it could do very fast (an entire hard drive in <2min).
edit: Try appending your output to a single string then outputting it all at the end. You might be surprised how long it takes it to continuously output text.