In this example we'll read data about prevalance of baby names in the US from a bunch of CSV files and try to analyze to data to figure out interesting trends from it.
require 'daru'
require 'gnuplotrb'
The data is contained in multiple CSV files on a per year basis.
Here's what a raw CSV file looks like for the year 1951. The first column is the name, second the sex and the third the number of births that took place with that name.
Find the data here
CSV.read("data/yob1951.txt").first(10)
All the CSV files are loaded into a DataFrame, a new column 'year' added to them for identification, and then concatenated to produce one large DataFrame.
data_frame = Daru::DataFrame.from_csv("data/yob1951.txt",
headers: ['name', 'sex', 'births'])
data_frame['year'] = [1951] * data_frame.size
(1952..2014).each do |year|
temp = Daru::DataFrame.from_csv("data/yob#{year}.txt",
headers: ['name', 'sex', 'births'])
temp['year'] = [year] * temp.size
data_frame = data_frame.concat(temp)
end
data_frame.vectors = Daru::Index.new(['births','name', 'sex','year'])
data_frame
Pivot the DataFrame on the year as the row and sex as the column, using the 'births' column for aggregation.
This tells us the number of male and female births per year.
pivoted = data_frame.pivot_table(
index: ['year'], vectors: ['sex'], agg: :sum, values: 'births')
The row and column names are rather inconvienient so we rename them into something better.
pivoted.index = Daru::DateTimeIndex.date_range(:start => '1951', :periods => pivoted.size, freq: 'YEAR')
pivoted.vectors = Daru::Index.new(['F', 'M'])
pivoted
The number of male births vs. female births can then be plotted against each other using the GnuplotRB gem.
GnuplotRB::Plot.new(
[pivoted['F'], with: 'lines', title: 'F'],
[pivoted['M'], with: 'lines', title: 'M'], title: 'Total births by sex and year')
To further prod into the data, lets perform an SQL style GROUP BY operation on the DataFrame on the 'year' column so that the DataFrame is divided into groups according to year.
groups_by_year = data_frame.group_by(['year'])
nil
This code interates over all the groups created by year and selects the rows that contain a particular name in them from.
In this manner we collect rows from every group on a per year basis and create a DataFrame for each name which tells the number of births of a name during a particular year.
For this example we'll choose the names James, Robert, Jessica and Sophia.
pieces = []
['James', 'Robert', 'Jessica', 'Sophia'].each do |name|
rows = []
groups_by_year.each_group do |group|
rows << group.row[group['name'].index_of(name)]
end
pieces << Daru::DataFrame.rows(rows)
end
Perform some basic preprocessing/cleaning on the DataFrame.
pieces.each do |df|
df['year'].map! {|e| DateTime.new(e) }
df.set_index('year')
df.rename df['name'][0]
df.delete_vector 'name'
end
plots = []
pieces.each do |df|
plot = GnuplotRB::Plot.new([
df['births'], with: 'lines', title: df.name])
plot.format_x = '%Y'
plots << plot
end
GnuplotRB::Multiplot.new(*plots).tap do |mp|
mp.layout = [2,2]
mp.format_x = '%Y'
mp.xtics = 'nomirror rotate by -45'
mp.title = 'Prevalence of certain names according to year'
mp.xlabel = 'Year'
mp.ylabel = 'Occurences'
mp.xrange = '"1945-01-01":"2016-01-01"'
end