Stop Microsoft

Operating Systems => Linux and UNIX => Topic started by: TheQuirk on 14 October 2002, 02:59

Title: Mirroring
Post by: TheQuirk on 14 October 2002, 02:59

I know there's a command (because I used it once) that downloads a whole webpage. I need to do this so I could mirror a website, and it has thousands of files (small ones, at that - ~5kb each).

Title: Mirroring
Post by: Stryker on 14 October 2002, 06:30

If they are allowing you to mirror it, can't you just download the source and put it in your web directory?

Title: Mirroring
Post by: Master of Reality on 14 October 2002, 06:42

wget -r website.com (i think)
make sure that you change /etc/wgetrc to allow wget to dl deeper than 5 directories.
read the man page for wget too

Title: Mirroring
Post by: TheQuirk on 14 October 2002, 06:56

quote:

---

Originally posted by Stryker:
If they are allowing you to mirror it, can't you just download the source and put it in your web directory?

---

It's more convinient for the computer to do it for you.

Thanks Master of Reality.

Title: Mirroring
Post by: Stryker on 15 October 2002, 02:24

quote:

---

Originally posted by TheQuirk:


It's more convinient for the computer to do it for you.

Thanks Master of Reality.

---

i know, but when you do wget you do not get the source code for any cgi, php, or any scripting involved.

Title: Mirroring
Post by: Master of Reality on 15 October 2002, 02:58

why wouldnt you?

Title: Mirroring
Post by: flap on 15 October 2002, 03:10

quote:

---

Originally posted by Stryker:
I know, but when you do wget you do not get the source code for any cgi, php, or any scripting involved.

---

well there's typically no way to get that using http; you won't have read permissions on it. there's no way to get server side scripts unless the webmaster has specifically made them available somehow.

Title: Mirroring
Post by: Master of Reality on 15 October 2002, 03:16

I thought the scripts were 755 permissions which would make them Executable and readable?

Title: Mirroring
Post by: voidmain on 15 October 2002, 03:23

I use rsync over ssh to mirror web sites. rsync will only update files that have changed. Obviously the first time you run it it will have to pull the entire web site. You will need ssh access to both servers in question and those servers will need to have rsync installed if they do not already.

Title: Mirroring
Post by: flap on 15 October 2002, 03:31

quote:

---

Originally posted by The Master of Reality / Bob:
I thought the scripts were 755 permissions which would make them Executable and readable?

---

For security reasons you don't want users of your website seeing the raw content of your scripts. The point is if you request a script from a web server it will parse that script and give you rendered html. How would you propose to get, for example, the ultimatebb.cgi script from this webserver?

[ October 14, 2002: Message edited by: flap ]

Title: Mirroring
Post by: voidmain on 15 October 2002, 04:06

quote:

---

Originally posted by flap:


For security reasons you don't want users of your website seeing the raw content of your scripts. The point is if you request a script from a web server it will parse that script and give you rendered html. How would you propose to get, for example, the ultimatebb.cgi script from this webserver?

---

Not to mention all the other cool things you could do, like get the database passwords for the credit card databases.  (http://smile.gif)

Title: Mirroring
Post by: KernelPanic on 15 October 2002, 21:54

I like

Code: [Select]

[1] Tux in no way edorses the infringing of copyright in any way. The usage of the above command is for getting the contents of your own website when you have no SSH access.

[ October 15, 2002: Message edited by: Tux ]

Title: Mirroring
Post by: Master of Reality on 16 October 2002, 00:58

quote:

---

Originally posted by Tux:
I like

Code: [Select]

[1] Tux in no way edorses the infringing of copyright in any way. The usage of the above command is for getting the contents of your own website when you have no SSH access.

[ October 15, 2002: Message edited by: Tux ][/b]

---

you especially dont condone its use to grab the entire contents of a  porno site  (http://smile.gif)

Title: Mirroring
Post by: TheQuirk on 16 October 2002, 07:56

Actually, I'm doing it to mirror textfile.com...