How do I get the url to stay the same when using a redirect script? - Page 2

Brook · 04-03-2009, 01:22 PM

Ah right thanks that makes sense now :lol:

However I have one more question :)

Quote:

So, they add a link from some crappy PR0 page on their site to point to:
YourDomain.com/redir.php?site=domainXYZ.com/crappyPR0.htm

The SEs will follow that link to your redirect from their site and end up right back at their site because of your redirect. Now, it looks to the SEs like your pretty PR5 page is linking to the crappy PR0 page on their lowlife site.

You say they will link to the link-back to their page, however, while I understand 'they' can fake the header info - surely search engines can't? So even if a search engine clicked on that link (from the spammers site) the HTTP REFERER I have in the code would see that the search engine didn't come from my domain and then the script would just die - hence it not working as the spammers intend it too?

Have I got that bit right? :lol:

WebSavvy · 04-03-2009, 02:51 PM

The only thing I can suggest is try it, and see. If you're able to get it to do what you want and without possibility of abuse, then that's wonderful. However, if it doesn't work in the way as expected, always know there are other options available.

What I'd suggest doing is setting up a redirect script with HTTP REFERER, and sends 301 header (codes are a few posts back) and then ask someone here to put up a test page on their site and link to your redirect script using their page and see what info you get from it.

Example:

Member XYZ creates a page on his/her domain called redirect-test.htm
On that page they put a link to: your-domain.com/redir.php?site=XYZ.com/redirect-test.htm

You can visit the page on their site and click the link, so can they.
Then you'd have to check your log files / AWStats (or whatever you use) to see where the access to the script came from. It'd show you one of two sources : 1- That it came from an external website (or) 2- That it came from your own website.

It's worth giving it a shot. Please do update here once you've tested because I am interested to know too.

Thanks. :)

Brook · 04-03-2009, 03:18 PM

I can tell you already that if they put a link on their site, to my redirect script with one of their own urls in it it will simply go to my page but it will be blank, and no redirect will occur (as the php code will make sure the script dies as the HTTP referrer was not true).

But that's not taking into account them spoofing the HTTP referrer - but could they do that for a link? (ie fool the search engines into spoofing the HTTP referrer too?) I wouldn't have thought so?

Re stats - as per the test above, it will simply show an external site went to that page (but the crucial thing is that page didn't do any redirecting - again this is not taking into account spoofing).

Am I on the right track? Do you still want me to check the stats?

WebSavvy · 04-03-2009, 03:38 PM

Nah, you don't still need to check the stats.

As far as them spoofing the header for a link, yes -- they do do it.

Look at the example I gave with the online proxy site. I'll explain a little more detail of it.

The guy that owned the proxy set his proxy to point at my site. I use full urls inside every link of my site (e.g., mydomain.com/folder/page.htm) and his proxy spoofed HTTP headers.

My URL was actually stripped out of the code and replaced with his url. So my internal links looked like hisdomain.com/folder/page.htm

In the browser bar his url was showing at the top with my content in it.

It showed my own pages as the http referer because he spoofed the headers.

Every single page of my site (50,000 pages) were indexed in google under his crap. It took weeks to get it all sorted. It caused a lot of grief for my site in the SERPs during that time, also.

Your site may not be a target because I don't know how big it is, or what the PR is. But when a blackhat sees a very large site that they can do this to, they aim for it. This same guy even did this to George Bush's website.

I code things differently now, as a result of that experience. What I mean is, I use host lookup and if the IP/host is from a domain I don't want to access my site it punts them and they get shown a page that tells them they don't have access to this domain -- no matter what page they look at on the site.

This stops the fake googlebots too. There's a few rogue scrapers out there that use the user-agent googlebot but they're not googlebot.

So, with host lookup it grabs the IP and dns and determines if it's really google or not. If it is, it's allowed to crawl the site. If it isn't, it punts them -- same message as the other ones get.

However, the ones running automated scripts to hammer the server get a different response. My system sends the request right back to them (like a redirect) heh heh.

Brook · 04-03-2009, 06:55 PM

How can they spoof a header for a link tho? Surely that would get them banned in Google? I mean google would know the difference between the site: real-domain-name.com and a site who's domain name is something-else.com - and then be able to tell if something-else.com is trying to pretend it is real-domain-name.com. I mean it's bound to check and log domain registry details for each domain it indexes (even if as a one off) and if one is spoofing, it gets banned?

WebSavvy · 04-03-2009, 07:06 PM

There are a great number of blackhats and spammers with sites pulling all sorts of things. Sometimes things can go on for quite some time before anyone over at Google notices.

I'm really good friends with a high ranking member of the spam team, and there's things I found out about that he didn't even know. I do let him know about things (if they're really bad) when I trip over them.

With just one link to gain from a redirect, your site might not ever get onto the radar of a blackhat or spammer. Sites that would offer them MANY links are more attractive and worthwhile to them Vs a site that offers them ONE link.

It all comes down to what works for you, Brook. If setting things up the way you've spoken of, works and does what you want it to, then by all means go with it.

I just wanted to inform you about other scenarios that can happen rather than to have you learn the hard way. I wish I'd have known about a lot of these things when I first started, but I learned the hard way. Once someone scrapes your hard work that's taken you years to build, it really opens your eyes (so to speak).

At the very least, I hope some of what I've said has been helpful. :)

WebSavvy · 04-03-2009, 07:28 PM

Here, these links provide more information, and explain in a better way than I am able to.

This one is from the Official Google Webmaster Central Blog:
Open redirect URLs: Is your site being abused?

Also there's this one:
spammers abusing open redirects -- Thomas Rayner Associates

Also this one:
7 Ways to Prevent Site Redirect Abuse by Spammers

Brook · 04-03-2009, 08:46 PM

Everything you have said has been very helpful Websavvy!

That google link is very interesting and explains the situation pretty well, and the tips on how to help prevent abuse are also helpful. Two of the tips are already used in the simple script I came up with, the HTTP referrer basically renders the script useless for spammers who want to hide their url behind a 'respected' url on my site - in other words if they try to send the link to people (spam) via my domain's redirect script, they'd be wasting their time as the script won't work should any of the recipients click on the link (HTTP referrer kicking in).

The HTTP referrer also helps against the other abuse (the one you were on about) where spammers generate links to your pages redirect script redirecting people to their porn/spam site - they usually do this so your domain (being a respected domain) is 'used' to mask their content/spam (so people think they are going to a safe site, yet it redirects to their porn) - by abusing your redirect script, but as I'm using HTTP referrer - it renders their trick useless because even if they did get my redirect script (with their url) indexed, there'd be no point because if people click on the search results (which my page would be in) they won't lead anywhere as the script dies because the HTTP referrer is false :D

The downside is not everyone's browser sends the HTTP referrer (so the script would die for them) - but as my site is aimed at web savvy people they will prob be using modern browsers.

The other thing google recommend is putting your script file in your robotos.txt file which I have done too.

They say to keep an eye on search results for your domain, and if you see your redirect scripts being indexed they're probably getting abused. But as I have disallowed it this should not be a problem.

Thanks again for all your help - you've made me aware of this problem that I didn't even know existed!

To summarise, I think it's safe to use the simpler script, so long as, you don't mind it not working for those people who are using browsers that don't send HTTP referrer (which I would have thought would be a minimal number - I've had no complaints yet but will keep an eye on it and post back if we get anyone complain the links are not working for them).

WebSavvy · 04-03-2009, 09:21 PM

Great! :) I'll be looking forward to any updates on your progress with it. They also mention the hash bit -- I have no idea how that would be implemented, but plan to do some research on it.

As far as some browsers not sending a HTTP REFERER, you know Opera has an option that allows you to turn that off, right? It's a pretty widely used browser too. Probably about equal with FF usage.

I came across some research about a week ago (I didn't bookmark it but I can probably find it again) that gave some interesting statistics on browser usage. I'll have a hunt around later on tonight after my son goes to bed, and I'll see if I can find it for you.

It may shed a little light on which browsers may be a potential issue but I'm sure there's probably a work-around that can be done. For example, you can use PHP to get the browser type the visitor is using.

I do this myself via php on my site because it allows me to serve CSS files based on browser type without any issues. I use dynamic css because I wrote a user preferences script (for web accessibility) that let's my visitors change the default theme of the site, fonts used, font colors, font sizes, and let's them define their own access keys.

Well at any rate, at least we managed to get enough info together so your site won't be so easily abused.

Brook · 04-03-2009, 10:30 PM

I use Opera - it's the best browser out there :D What's the best browser (for web surfing)?

Tbh, I'm not too bothered by people not having http referer on - they are just outgoing links anyway, and I'd rather they stay on my site LOL!

Perhaps another couple of ways to get around it (if needed) would be cookies or sessions.

I'll let you know if any of my users run into any problems, and thanks again for your help - it was nice working this out with ya, hope you stick around the site, there's a nice bunch of people here :)

WebSavvy · 04-03-2009, 11:01 PM

Yeah, I'm not planning on going anywhere. This seems like a really nice forum, which is why I joined it. Everyone seems really helpful and nice, and they all act in a professional manner. I really like that a lot. :)

OK, back off to work for me now. I'm in the middle of about 50 things at once. Thank goodness women were born to multi-task. :D

Brook · 04-04-2009, 02:26 AM

That's good - nice to have you here :)

I may even need you to test something for me at some point - my next project 'may' attract some blind users, so might need you to take it for a spin and let me know what you think!

The only thing is I have no idea how many blind people use the web, do you have any links that may help me there? Also would you mind if I ask how your browser the web? (Can you view anything on the screen at all?) Please don't feel like you have to answer, I'll totally understand.

WebSavvy · 04-04-2009, 03:55 AM

Sure, I don't mind helping you with it.

There are a lot of blind users online. Just as there are also many disabled persons online (e.g., persons with physical limitations, cognitive disorders, or motor skill difficulties). People with disabilities use various types of assistive technology to access the web. This technology functions through their browser.

Just not long ago Joel (a really GREAT guy) from WebAIM, did a survery among blind users to find out more about how the blind use web pages. It really helped to bridge a gap in understanding on a number of fronts.

For example, lots of sighted people that are into web accessibility use skip links on their site. However, they always label them as "Skip Navigation", and I never liked that because my question is ... where are you skipping me to?

On my own site I have skip links and it's marked as "Skip to [this]" ... OR ... "Skip to [that]"
so that way a blind person at least has some idea where this skip link will take them or move them past. Turns out from Joel's survey, all the rest of the blind folk think the same way I do. That was kind of a hoot. Something as simple as that, made perfect sense to me -- but a sighted person couldn't deduce that even though they're involved in the movement to bring about the new standards for web accessibility.

Yes, I can see somewhat. I am not totally blind. I can view distance of less than the length of my arm. I was born this way. I was 2 and a half years old when they figured out I couldn't see. My Father was playing catch with me, and the ball kept hitting me in the face. So, he thought I just couldn't catch that well. Then he thought, hmmm ... maybe she has bad eyesight. So, he held up 3 fingers and asked me to tell him how many he was holding up. I had to ask him which arm was he holding up because I couldn't tell.

They took me into get an eye exam right after that. The Dr. said I was lucky to be able to see leafs on trees, and that by the age of 3 if I had not been put in glasses -- I would have gone completely blind. I was born with an eye disease called progressive high (degenerative) myopia. Most people that have it end up completely blind at some point in their life.

I am able to see colors and shapes, just not very well. Everything I see is a blur of color with several thick fuzzy outlines around it that make it very difficult to distinguish edges that define shapes and colors just smear into one.

I can resize text on a page and read it if my eyes aren't bothering me too bad. I have windows eyes and it blows graphics up real well so I work with that sometimes. I also use a screen reader and it reads the text of a page to me. It's very easy to use and is also very useful to hear what a page sounds like so you can know what a blind person hears when they listen to it.

Brook · 04-04-2009, 04:03 PM

Thanks for the info WS. So many of us take things like being able to see for granted. My niece is also blind, she just turned 7 last week - I feel so sad when I see her she's such a lovely little girl, life can be so cruel can't it?

You should be proud of yourself - you seem to have achieved so much! A lot more than most able-bodied people! ...I wish I was as good a coder as you!

I will definitely get your views on my next project, as I think there may be some blind people interested in it and I want to make sure it's as accessible as possible for them.

A lot of the time we are restricted by the platforms we use (such as vbulletin), but this site I want to build myself (*panics!*) as hopefully it isn't going to be too difficult as it's basically a site of members profiles (with friends lists etc), along with a bit of a mini-blogging. I have pretty much decided on using cakePHP for it - wish me luck! (I'll need it as I am just a php nube right now!).

Thanks again for your help, I will have a good look at the site you posted.

Also I feel compelled to mention this (I can't help myself!) but a few years ago I wasn't feeling so great, and ended up doing a 'detox' - well I call it a detox but it is basically a way of life/health regime, where I had to cut out things like sugar, yeast, lactose etc - my dietician who was advising me on it said that no matter what 'illness' people suffer from this 'detox' would nearly always help, as it helps rebuild and repair your immune system, and so can help with a whole host of problems, esp when conventional medicine fail us. She herself was 'blinded' on some days as she suffered with candida related problems herself that effected her eyesight as well. I know it sounds far-fetched, but I am a firm believer in the importance of diet and how it can effect us - specifically how all the 'crap' in modern food can have a negative impact on us (I know as I am living proof - some things I just can't eat because they make me feel like crap). I'm just mentioning it in case you feel it is worth a try, and while I'm not saying it's a cure, it may help sustain (or even repair a little bit) the level of eyesight you have now.

If you fancy having a go, I've typed it up here: The detox.

WebSavvy · 04-04-2009, 05:06 PM

Diet won't improve my eye disease. I actually do have a very good diet, and always have. I eat very little meats and when I do it's chicken or fish. I love veggies a lot.

I've never had to go on a diet (to lose weight) in my entire life. Even at my age, I am still quite small. I'm 5ft 3in, and weigh 108 lbs. I had my first and only child when I was 35, and 3 days after having him I was back down to 115lbs.

I've always been a health nut. In college I played on our tennis team (wearing really THICK glasses LOL). I also ran cross country track and up until my son was born, I still ran 8.5 miles every night. I was on the women's weight lifting team in college too. I'm not buffed out, just toned. But I can deadlift 250lbs.

I understand how you feel toward your niece, but really the best thing you and your family can do for her is to treat her no differently than you would a sighted child. This will allow her to develop the sense that she is not held back from anything due to her disabilities.

As an example, my cousin is blind and has been since birth. They had to remove his eyes because they were too disease ridden. He has glass eyes. My aunt and uncle treated him the same as they did their other (sighted) children. He grew up with the sense that there wasn't anything he couldn't do and that his only limitations were ones he imposed on himself.

He went to college and earned a degree. He works in Mercy Hospital in Detroit as an X-ray Technologist. Yes, a blind man develops X-rays. Who'd have thunk it?

I too went to college because English was my passion, as was teaching. I taught for many years and retired from that prior to the birth of my son.

My one inspiration that I would always instill in my students was, a person with great foresight is known as a seer ... a visionary. One does not have to have eyesight to be a visionary. They hold the power within themselves to achieve great things, they just have to know it, and believe it, and then they will no doubt achieve it.

Thank you for the link to the detox diet though. I certainly will have a look at it.

Brook · 04-04-2009, 05:25 PM

Sounds like you are sorted

if your diet does however include processed foods, sugar, yeast or dairy... it may still be worth giving a go. But you sound pretty healthy from what you said!

My health went downhill after a bereavement - hence I needed to 'detox' to get healthy again.

My niece has neurological problems too, so unfortunately she will never be able to lead anything like a normal life, hence why I feel so sad about it. She's such a lovely little girl tho, you can't help but hug her.

Anyway, let's talk about something else before we end up depressing ourselves lol - I am bound to be posting lots more threads asking for help, so hopefully catch you in one of them.

Thanks again for your help in this thread - it was nice chatting with ya

WebSavvy · 04-04-2009, 05:27 PM

Sure, Brook. :)

You're also welcome to PM me anytime, as well. It's been lovely getting to speak with you.

Brook · 04-04-2009, 05:31 PM

Ditto - feel free to PM me anytime too