Wednesday, October 07, 2015

Web Scraping with Ruby

I was learning a little on how to use Ruby to do some web scraping today, following this article which is admittedly out of date: http://danwin.com/2010/04/coding-for-journalists-go-from-a-know-nothing-to-web-scraper-in-an-hour-hopefully/

I had to make a few edits along the way to fix some bugs and thought I would share. Look for "UPDATE" below.

# --- Get a list of all US Presidents' names, loop through to check for any with last names >6 characters and see if they have died. Calculate the average age for those Presidents ---

# open the required libraries
require 'rubygems'
require 'nokogiri'
require 'open-uri'

# Using nokogiri, fetch Wikipedia's list of presidents page
# UPDATE: Had to use https
list_of_presidents = Nokogiri::HTML(open('https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'))

# Using another nokogiri method, grab the second column (td) from every row (tr), and from those, grab the first hyperlink (a) which should also contain the President's name
# UPDATE: Had to modify XPath to match Wikipedia's current layout
an_array_of_links = list_of_presidents.xpath("//tr/td[2]/b/a[1]")
# UPDATE: I changed the output a little, so used a few different variables
long_name_count = 0 dead_prez_count = 0 alive_prez_count = 0 total_age = 0 an_array_of_links.each do |link_to_test| # This above statement can be read as: for each element in an_array_of_links, do # the following code (until the end line) # And as you go through each element, the variable use to reference the element will be named "link_to_test" # break up the name string by spaces, then get the last element in the resulting array to get the last name last_name = link_to_test.content.split(' ')[-1] if last_name.length > 6 long_name_count += 1 the_link_to_the_presidents_page = link_to_test["href"] # The value of href is going to be something like "/wiki/George_Washington". # That's an address relative to the Wikipedia site # so we need to prepend "https://en.wikipedia.org" to have a valid address...
 # UPDATE: Had to use https
the_link_to_the_presidents_page = "https://en.wikipedia.org"+the_link_to_the_presidents_page # now let's fetch that page the_presidents_page = Nokogiri::HTML(open(the_link_to_the_presidents_page)) # check if they died
 # UPDATE: Had to change this to remove .content for cases where there is no death_date, then change the the 'if' immediately below
death_date = the_presidents_page.xpath("//th[contains(text(), 'Died')]/following-sibling::*")[0]
        if death_date && death_date.content && death_date.content[0]
            # check what their age was
            # UPDATE: Had to change this to add .content and add * to the regex to get all digits of the President's age
age_at_death = death_date.content.match(/aged.+?([0-9]*)/)[1] if age_at_death # we only get here if there was a "Died" table cell AND a text pattern similar to: "aged XX" puts "Age of #{link_to_test.content} is: #{age_at_death}"
                # UPDATE: Had to change to age_at_death.to_i to use the full age, not just a single digit
total_age += age_at_death.to_i # technically, age_at_death is a String. to_i will make it a Number so we can safely add it to total_age dead_prez_count += 1 end else puts "#{link_to_test.content} is still alive" alive_prez_count += 1 end end end puts "Total Presidents: #{an_array_of_links.count}" puts "...with Surname >6 characters: #{long_name_count}, (#{an_array_of_links.count - long_name_count} had short names)" puts "...that have died: #{dead_prez_count}, (#{alive_prez_count} are still alive)" puts "Of dead presidents, their total age is #{total_age} and average age is #{total_age / dead_prez_count}" # OK, we're at the end of the each loop. Go back to the top

Tuesday, April 21, 2015

VR for Beginners: Cardboard, Unity and iOS

Over three years since my last post, so I better make this a useful one!

I've been playing around with various bits of tech recently and wanted to capture some of the finer points as a reminder for myself later and for anyone else out there that is interested. This one will be about VR.

VR & AR appear to be coming back in a big way and could be important "platforms" to consider for all kinds of development in the near future. Since things are taking off but still pretty new, there are a lot of options to look into in terms of devices and SDK's, not a lot of standards, etc. But what if you just want to get started working on some VR content to see how it works, learn some lessons, etc.? What is the cheapest and easiest way to do that? I decided the best option for me was to try using Google Cardboard, weighing up these points:

Pros:
  1. Cardboard kits are cheap. In fact, I got mine for free at GDC 2015.
  2. Google provide SDK's to build Cardboard apps for Android, including a Unity engine version.
  3. There is now a free version of Unity available.
  4. Mobile devices are commonplace and are already an established platform for games.
  5. There isn't much VR content out there right now for mobile devices.

Cons:
  1. Google's SDK's only work with Android devices.
  2. I have an iPhone right now.
  3. I don't want to spend time writing a game engine, but I don't know Unity.
  4. Cardboard isn't as good as more powerful devices, such as the Oculus Rift, so I would need to limit the scope of my demos and learnings, but my time is limited anyway.
So, what was I going to try and do? I wanted to use my existing setup as much as possible and add any extra stuff I needed for free so I could rapidly learn some new skills through experimentation. I wanted to end up with a demo I could have on my phone to show people and maybe put it up on the App Store. What was the list of ingredients I was starting with?
  1. Google Cardboard kit - Already had this for free, luckily.
  2. Unity engine 5.0.1f1 - I was just starting to experiment with this at work anyway. Not a tool I was familiar with, but it had to be easier to learn rather than spend time building my own engine. Note that v5 is pretty new though.
  3. Mac with XCode 6.3 - Just making sure I had the latest version installed.
  4. iPhone 5S - My current mobile phone.
What were the blockers?
  1. Learning Unity.
  2. Finding a Cardboard SDK for Unity (v5) that supported iOS devices.
  3. Starting to build content.
1. Learning Unity
I started out just doing a simple 2D Flappy Birds tutorial to get some familiarity with the toolset, which only takes about 30 mins, plus some extra time to try and do a couple of follow on improvements. It was a little out of date because it was based on Unity 4, but no big deal. I was doing this to help me at work anyway so I would understand what the tools were like when speaking to other engineers that use it in our games.
Then I started looking at a 3D tutorial on the Unity site, Roll-a-ball.

2. Finding a Cardboard SDK for Unity that supported iOS devices
As mentioned, the Google SDK's only support Android devices, but I have an iPhone 5S. So what alternatives did I dig up?
  1. Durovis Dive is another mobile headset adapter; essentially a more premium version of a cardboard device. They have an SDK that supports both iOS and Android handsets from Unity. Unfortunately this is for Unity 4 and I couldn't get it working quickly with my setup. I may get things running in the Unity emulator, but try to export to XCode and things wouldn't build.
  2. There are other devices out there that have SDK's too, but I couldn't be sure they would work with my setup.
  3. While looking through the Durovis forums I noticed some GitHub projects to improve their SDK. They didn't quite fit what I was looking for, since they were still based on the Durovis SDK that wasn't working for me, so I kept looking.
  4. There have been other attempts at porting Cardboard to iOS, but they don't necessarily include Unity support.
  5. Eventually I came across three promising GitHub projects:
    1. CardboardVR-iOS from equinox2k : A port of the Cardboard SDK to iOS, but without the Unity version.
    2. CardboardSDK-Unity from JScott : An updated version of the Cardboard SDK for Unity that adds more input controls, (e.g. being able to hold the magnet trigger down.) But, only for Android.
    3. CardboardSDK-iOS from rsanchezsaez : A port of the Cardboard SDK to iOS, including the Unity version. Doesn't have 100% parity, since there isn't input for Unity at this point, but a great start.
I ended up getting the demos included with CardboardSDK-iOS working on my iPhone last night. It is based on a slightly older version of Unity, so I had to rebuild the XCode project, but the instructions in the ReadMe helped, along with this issue. It is also based on a slightly older version of the Cardboard SDK, but is good enough for me to get started.

Now I'm wondering if there is a way to combine this with CardboardSDK-Unity to get the best of both. I'm not really familiar with Git yet, (I've used other source control systems,) so I'll need to figure that out to see if I can contribute back in some way.

3. Starting to build content
Now to start building stuff! First I'm going to finish up the Roll-a-ball tutorial and then add a VR camera to it. If I get that working, I can share here. I think the main challenge is going to be figuring out how input should work, but that is the point in trying to learn with these experiments! Then I've got an idea for a new, more in depth project that should work within the limitations of a cardboard device, (player in a stationary position, able to look around them with limited input,) but more on that as I get to it.

After that, let's see what happens. Maybe it means getting a more powerful device such as Rift or Hive. Maybe it means looking into what Razer are doing with their open platform.

Anyway, I hope that helps someone out there!