Web Scraping with Ruby

I was learning a little on how to use Ruby to do some web scraping today, following this article which is admittedly out of date: http://danwin.com/2010/04/coding-for-journalists-go-from-a-know-nothing-to-web-scraper-in-an-hour-hopefully/

I had to make a few edits along the way to fix some bugs and thought I would share. Look for "UPDATE" below.

# --- Get a list of all US Presidents' names, loop through to check for any with last names >6 characters and see if they have died. Calculate the average age for those Presidents ---

# open the required libraries
require 'rubygems'
require 'nokogiri'
require 'open-uri'

# Using nokogiri, fetch Wikipedia's list of presidents page

# UPDATE: Had to use https
list_of_presidents = Nokogiri::HTML(open('https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'))

# Using another nokogiri method, grab the second column (td) from every row (tr), and from those, grab the first hyperlink (a) which should also contain the President's name

# UPDATE: Had to modify XPath to match Wikipedia's current layout
an_array_of_links = list_of_presidents.xpath("//tr/td[2]/b/a[1]")

# UPDATE: I changed the output a little, so used a few different variables
long_name_count = 0
dead_prez_count = 0
alive_prez_count = 0
total_age = 0

an_array_of_links.each do |link_to_test|
    # This above statement can be read as: for each element in an_array_of_links, do
    # the following code (until the end line)
    # And as you go through each element, the variable use to reference the element will be named "link_to_test"
    
    # break up the name string by spaces, then get the last element in the resulting array to get the last name
    last_name = link_to_test.content.split(' ')[-1]
    
    if last_name.length > 6
        long_name_count += 1
        
        the_link_to_the_presidents_page = link_to_test["href"]
        # The value of href is going to be something like "/wiki/George_Washington".
        # That's an address relative to the Wikipedia site
 # so we need to prepend "https://en.wikipedia.org" to have a valid address...

 # UPDATE: Had to use https
the_link_to_the_presidents_page = "https://en.wikipedia.org"+the_link_to_the_presidents_page

 # now let's fetch that page
 the_presidents_page = Nokogiri::HTML(open(the_link_to_the_presidents_page))

        # check if they died

 # UPDATE: Had to change this to remove .content for cases where there is no death_date, then change the the 'if' immediately below
death_date = the_presidents_page.xpath("//th[contains(text(), 'Died')]/following-sibling::*")[0]

        if death_date && death_date.content && death_date.content[0]
            # check what their age was

            # UPDATE: Had to change this to add .content and add * to the regex to get all digits of the President's age
age_at_death = death_date.content.match(/aged.+?([0-9]*)/)[1]
            if age_at_death
                # we only get here if there was a "Died" table cell AND a text pattern similar to: "aged XX"
                puts "Age of #{link_to_test.content} is: #{age_at_death}"

                # UPDATE: Had to change to age_at_death.to_i to use the full age, not just a single digit
total_age += age_at_death.to_i  # technically, age_at_death is a String. to_i will make it a Number so we can safely add it to total_age
      dead_prez_count += 1
            end
        else
            puts "#{link_to_test.content} is still alive"
            alive_prez_count += 1
        end
    end
end

puts "Total Presidents: #{an_array_of_links.count}"
puts "...with Surname >6 characters: #{long_name_count}, (#{an_array_of_links.count - long_name_count} had short names)"
puts "...that have died: #{dead_prez_count}, (#{alive_prez_count} are still alive)"
puts "Of dead presidents, their total age is #{total_age} and average age is #{total_age / dead_prez_count}"

# OK, we're at the end of the each loop. Go back to the top