How to download a site with wget and simultaneous connections
Apart from some other downloader tools like
aria2c, is still pretty functional.
Having to download the whole directory you may want to run:
wget -r [URL]
or with mirror option:
wget -m [URL]
But every file will be waiting for its turn. Actually the whole process will be hopelessly slow.
I recommend to invoke wget like this:
wget -r -np -nc [URL] &
Here’re the options explained:
Turn on recursive retrieving. The default maximum depth is 5.
Do not ever ascend to the parent directory when retrieving recursively. This is a useful option, since it guarantees that only the
files below a certain hierarchy will be downloaded.
If a file is downloaded more than once in the same directory, Wget’s behavior depends on a few options, including -nc. In certain
cases, the local file will be clobbered, or overwritten, upon repeated download. In other cases it will be preserved.
When running Wget without -N, -nc, -r, or -p, downloading the same file in the same directory will result in the original copy of file
being preserved and the second copy being named file.1. If that file is downloaded yet again, the third copy will be named file.2, and
so on. (This is also the behavior with -nd, even if -r or -p are in effect.) When -nc is specified, this behavior is suppressed, and
Wget will refuse to download newer copies of file. Therefore, „”no-clobber”” is actually a misnomer in this mode—it’s not clobbering
that’s prevented (as the numeric suffixes were already preventing clobbering), but rather the multiple version saving that’s prevented.
When running Wget with -r or -p, but without -N, -nd, or -nc, re-downloading a file will result in the new copy simply overwriting the
old. Adding -nc will prevent this behavior, instead causing the original version to be preserved and any newer copies on the server to
When running Wget with -N, with or without -r or -p, the decision as to whether or not to download a newer copy of a file depends on
the local and remote timestamp and size of the file. -nc may not be specified at the same time as -N.
A combination with -O/–output-document is only accepted if the given output file does not exist.
Note that when -nc is specified, files with the suffixes .html or .htm will be loaded from the local disk and parsed as if they had
been retrieved from the Web.
-nc you can also use
-N – meaning is a little bit different but will also get the job done.
Wget is quite verbose by default, so let’s direct the output to /dev/null:
wget -r -np -nc [URL] >/dev/null 2>&1&
The obove command should be run from the terminal X-times. Depending of the number of files and your connection.