Mirroring

[TheQuirk]

I know there's a command (because I used it once) that downloads a whole webpage. I need to do this so I could mirror a website, and it has thousands of files (small ones, at that - ~5kb each).

[Stryker]

If they are allowing you to mirror it, can't you just download the source and put it in your web directory?

[Master of Reality]

wget -r website.com (i think)
make sure that you change /etc/wgetrc to allow wget to dl deeper than 5 directories.
read the man page for wget too

[TheQuirk]

quote:Originally posted by Stryker:
If they are allowing you to mirror it, can't you just download the source and put it in your web directory?

It's more convinient for the computer to do it for you.

Thanks Master of Reality.

[Stryker]

quote:Originally posted by TheQuirk:


It's more convinient for the computer to do it for you.

Thanks Master of Reality.

i know, but when you do wget you do not get the source code for any cgi, php, or any scripting involved.

[Master of Reality]

why wouldnt you?

[flap]

quote:Originally posted by Stryker:
I know, but when you do wget you do not get the source code for any cgi, php, or any scripting involved.

well there's typically no way to get that using http; you won't have read permissions on it. there's no way to get server side scripts unless the webmaster has specifically made them available somehow.

[Master of Reality]

I thought the scripts were 755 permissions which would make them Executable and readable?

[voidmain]

I use rsync over ssh to mirror web sites. rsync will only update files that have changed. Obviously the first time you run it it will have to pull the entire web site. You will need ssh access to both servers in question and those servers will need to have rsync installed if they do not already.

[flap]

quote:Originally posted by The Master of Reality / Bob:
I thought the scripts were 755 permissions which would make them Executable and readable?

For security reasons you don't want users of your website seeing the raw content of your scripts. The point is if you request a script from a web server it will parse that script and give you rendered html. How would you propose to get, for example, the ultimatebb.cgi script from this webserver?

[ October 14, 2002: Message edited by: flap ]

[voidmain]

quote:Originally posted by flap:


For security reasons you don't want users of your website seeing the raw content of your scripts. The point is if you request a script from a web server it will parse that script and give you rendered html. How would you propose to get, for example, the ultimatebb.cgi script from this webserver?

Not to mention all the other cool things you could do, like get the database passwords for the credit card databases.  

[KernelPanic]

I like

Code: [Select]

[1] Tux in no way edorses the infringing of copyright in any way. The usage of the above command is for getting the contents of your own website when you have no SSH access.

[ October 15, 2002: Message edited by: Tux ]

[Master of Reality]

quote:Originally posted by Tux:
I like

Code: [Select]

[1] Tux in no way edorses the infringing of copyright in any way. The usage of the above command is for getting the contents of your own website when you have no SSH access.

[ October 15, 2002: Message edited by: Tux ][/b]

you especially dont condone its use to grab the entire contents of a  porno site  

[TheQuirk]

Actually, I'm doing it to mirror textfile.com...