The following script will pull data values from a wikiart page (an excellent index of paintings) –
import glob
indir = 'D:\\projects\\art\\'
for filename in glob.glob(indir + "*.html"):
print filename
spans = soup.select("span[itemprop]")
ahrefs = soup.select("a[itemprop]")
file = open(filename, 'r')
soup = BeautifulSoup(file, 'html.parser')
spans = soup.select("span[itemprop]")
arefs = soup.select("a[itemprop]")
[(v["itemprop"], v.text) for v in spans + arefs]
Since most of the data values are labelled, you get a data structure like so:
[(u’birthDate’, u’31 March 1902′), (u’dearthDate’, u’24 August 1976′), (u’nation’, u’French’), (u’nation’, u’Russian’), (u’art movement’, u’Art Informel’), (u’art movement’, u’Tachisme’), (u’painting school’, u’\xc9cole de Paris’), (u’genre’, u’abstract’)]
D:\projects\art\wikiart\www.wikiart.org\en\andre-pierre-arnal.html
D:\projects\art\wikiart\www.wikiart.org\en\andre-pierre-arnal.html
[(u’birthDate’, u’16 December 1939′), (u’nation’, u’French’), (u’art movement’, u’Art Informel’), (u’art movement’, u’Contemporary’), (u’painting school’, u’Supports/Surfaces’), (u’genre’, u’abstract’)]
D:\projects\art\wikiart\www.wikiart.org\en\andrea-del-castagno.html
D:\projects\art\wikiart\www.wikiart.org\en\andrea-del-castagno.html
[(u’birthDate’, u’c.1421′), (u’dearthDate’, u’19 August 1457′), (u’nation’, u’Italian’), (u’art movement’, u’Early Renaissance’), (u’painting school’, u’Florentine School’)]