https://heidelsec.de/en/tags/web-scraping/Recent content in Web-Scraping on HeidelSechttps://heidelsec.de/images/og.jpghttps://heidelsec.de/images/og.jpgHugo -- 0.167.0en2026 Imprint - Privacy PolicyFri, 08 Nov 2024 15:30:00 +0100https://heidelsec.de/en/posts/2024-11-08-wget-mirror-website/Fri, 08 Nov 2024 15:30:00 +0100https://heidelsec.de/en/posts/2024-11-08-wget-mirror-website/<h2 id="saving-a-website-for-offline-use">Saving a Website for Offline Use</h2> <p>Using <code>wget</code>, you can download an entire website for offline access with this command:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">wget -rkpNc -e <span class="nv">robots</span><span class="o">=</span>off http://www.example.com/ </span></span></code></pre></div><h3 id="parameters-explained">Parameters Explained</h3> <ul> <li><strong><code>-r</code></strong>: Download pages recursively.</li> <li><strong><code>-k</code></strong>: Adjust links for local viewing.</li> <li><strong><code>-p</code></strong>: Fetch all resources (images, scripts).</li> <li><strong><code>-N</code></strong>: Only download new or modified files.</li> <li><strong><code>-e robots=off</code></strong>: Ignore <code>robots.txt</code> restrictions.</li> <li><strong><code>-c</code></strong>: Continue the download if interrupted.</li> </ul> <p>With this command, <code>wget</code> reliably saves the entire site for offline use.</p>