scraping - Gary Sieling

Tag: scraping

Web scraping, harvesting, data extraction is the process of gathering data from websites. Various tools exist to download, clean, and process data, from built-in UNIX tools to custom software and programming language APIs. Often processing is required – either to extract structured or unstructured data. Sites that generate pages but do not provide an API often are easier to parse, as they are more consistent. Unstructured data is common in data extracted from PDFs, and in some cases manual intervention is even required, where you are forced to use a system like Amazon Mechanical Turk to have humans batch process data.

Below you will find some of the most popular pieces I’ve written, often proof of concept tools to test new technologies and explore database systems. Much data is readily available given some patience, from legal documents such as court cases, financial filings, weather monitoring, advertisement text, search results, auction data, and so on.

There are a lot of examples that get you parts of the text on a page, but most of them don’t seem to be able to get nested text.

The easiest way to do this is with jsDom, which is also the heaviest one:

let jsdom = require('jsdom');

let file = 'D:/projects/image-annotation/data/talks/pages/talk200.html';
jsdom.env(
  file,
  ["http://code.jquery.com/jquery.js"],
  (err, window) => {
    console.log(
      window.$(".transcript-text").text()
    );
  }
);

This will get you the entire text contents within a section of the page, which is ideal if you’re doing scraping.