Crawling and Copying Web Pages. #8


 


There are many reasons one may want a copy of a Web page or whole site.


I tend to just use Chrome browser's share > print and pdf printer function to save web pages I hope to be useful information later on. And try  to keep data well sorted by creating and nesting file names.

But on a paged site this is not Idea. And while I do use Archive.org and Wayback machine. 


Having a offline playable copy of a website is often ideal.


There seems to be two main categories of (free) types of web crawlers besides online and downloaded.

And I most always prefer downloaded ones.

Type One is just a complete as site allows working HTML copy of site you can view offline.

For this I have used www.httrack.com for many years.

I have in Windows 10 and today added Adroid apk to the Tablet.

While it would be Hacking nice to have a copier that disobeyed website crawler rules. I have not had one for many years now.

Recommendations welcomed.

The Second type of crawler seems to deal with SEO tags and other useful to site owner or someone wanting to improve their site's findability.


I will look into these later on and update this blog.

  





Comments

Post a Comment

Popular posts from this blog

Beginners Steps-2. OSINT Open Source Intelligence, #11

Image
 OSINT is not just a large nut to crack but a large grove of all sorts of nut bearing trees you may want or need to crack. It also massive subcategories, SOCMINT is a major one.  Beginners Steps series focuses on one small part of OSINT-VAPT at a time. Today is Social Media Intelligence SOCMINT. As always the www has a massive amount of pages dealing with this. Remember, "goolge dorking" is you friend, ok and spies on you too. But while reviewing previous saved dork pages, I felt this one was perfect for "Beginners Steps-2" As SOCMINT is a easy greenfield for everyone. It is also a great place to tryout all the tips, hints and hacks you read. While if you wish picking yourself or other well known targets, that you can really see how accurate the intel you collect is.  Oh and yes a great time to start formating what you are looking for and all the results you obtain. Into a useful share or saleable format. PS think of using a "Markdown Editor"for this. So w...
Read more

Beginners Steps-1. OSINT Open Source Intelligence, #10

Image
OSINT is not just a large nut to crack but a large grove of all sorts of nut bearing trees you may want or need to crack.  Beginners Steps series will focus on one small part of OSINT-VAPT at a time. With a link or few to what I feel is a good explanation of each step. While almost childish to the Pro Investigators or Hackers.  I will try and share what I feel is the most useful and easiest tools and targets to learn for beginners and with the minimum of resources to use.  As I recently read, it not how many tools you have but being able to use a tool like a Master Craftsman. Beginners Steps-1 How to get amazing, focused and detailed results from the Google search line. Using Google's own web crawling and indexing tools or syntax. Called google dorks or google hacking. This powerful tool set will help you find information most often hidden to a normal search on anything or anybody that is anywhere in the www. So without more words, start with the link below for a good exp...
Read more