AI Tip: Web site scraping

Mar 19, 2024 | General

Blog

Have a lot of web content to train?

With Vizaport, and other AI RAG platforms, there are many options for training your customized AI model. Different types of documents, from PDFs to Word docs to spreadsheets and more. But web pages often have content that is already approved for public viewing, and often times summarized for a customer. Therefore, it is common for clients that train an AI to first think about pointing it to their web site, as the first training model.  

In the next image, you’ll find an example of Vizaport’s AI training. Simply drag-and-drop documents of all types. Or give it a URL. 

In our AI tips and tricks, we talk about how web page content may not result in the best accuracy, compared to some other structured data or unstructured data formats. Yet, we know that clients will continue to use the URL extract function. We get it. It’s easy. It’s already there.  

AI Training

What’s the best way to load web site data?

First, let’s review a few different options to get web site data loaded into an AI training model. Here’s a few options, along with the pros and cons of each:

  1. Single web page extract: For sites with limited pages, a URL can be entered into Vizaport and a single page extracted. You control the pages. You delete unnecessary content/pages. It’s helpful… but not for web sites with a lot of content.
  2. Saving pages as individual documents: If you have access to tools that efficiently download and extract web pages, you can save them in a variety of formats and then upload them. It’s helpful to see each document to edit, but can be difficult to maintain for sites with hundreds of pages.
  3. Web scraping into a single document: Perhaps we shouldn’t have used the term “web scraping“, as sometimes this can have a bad connotation, but it’s your web site. You control it.  If you have tools that can download content and put it into a single document, this makes it easy to load and build connectors over time to sync. WordPress has a great one for those that are on WordPress sites as we document below in a video.  

Video Example – Plugin to scrape large web sites

Here’s an example video to walk through the process of legally scraping a web site (that you presumably own) to get its data into a single file that can be loaded and maintained in an AI model like Vizaport. the results are fairly accurate for a broad web site download without customizing data. You can try it yourself if you want to test the results at this web site in the video.

And here’s the link to the plug-in referenced in the video for those that are WordPress users.

More tips and tricks

Recent Articles

A Simple Guide to Efficient Fiber Costing and Quoting

A Simple Guide to Efficient Fiber Costing and Quoting

May 28, 2025 | , ,

Speed wins with fiber – not just internet speed itself, but also the turnaround time between quotation request and response. It many cases, the first reasonably priced quote wins, leaving companies with slow costing and quoting processes in the dust. Unfortunately,...