R Fundamentals I

Subsetting data

Learning Objectives

  • To be able to subset vectors, factors, matrices, lists, and data frames
  • To be able to extract individual and multiple elements:
    • by index,
    • by name,
    • using comparison operations
  • To be able to skip and remove elements from various data structures.

R has many powerful subset operators and mastering them will allow you to easily perform complex operations on any kind of dataset.

There are six different ways we can subset any kind of object, and three different subsetting operators for the different data structures.

Let’s start with the workhorse of R: atomic vectors.

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5

So now that we’ve created a dummy vector to play with, how do we get at its contents?

Accessing elements using their indices

To extract elements of a vector we can give their corresponding index, starting from one:

  a 
5.4 

  d 
4.8

The square brackets operator is just like any other function. For atomic vectors (and matrices), it means “get me the nth element”.

We can ask for multiple elements at once:

  a   c 
5.4 7.1

Or slices of the vector:

  a   b   c   d 
5.4 6.2 7.1 4.8

the : operator just creates a sequence of numbers from the left element to the right. I.e. x[1:4] is equivalent to x[c(1,2,3,4)].

We can ask for the same element multiple times:

  a   a   c 
5.4 5.4 7.1

If we ask for a number outside of the vector, R will return missing values:

<NA> 
  NA

This is a vector of length one containing an NA, whose name is also NA.

If we ask for the 0th element, we get an empty vector:

named numeric(0)

Skipping and removing elements

If we use a negative number as the index of a vector, R will return every element except for the one specified:

  a   c   d   e 
5.4 7.1 4.8 7.5

We can skip multiple elements:

  b   c   d 
6.2 7.1 4.8

To remove elements from a vector, we need to assign the results back into the variable:

  a   b   c   e 
5.4 6.2 7.1 7.5

Challenge 1

Given the following code:

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5

Come up with at least 3 different commands that will produce the following output:

  b   c   d 
6.2 7.1 4.8

Subsetting through logical operations

We can subset through logical operations:

  a   b 
5.4 6.2

Since comparison operators evaluate to logical vectors, we can also use them to succinctly subset vectors:

  c   e 
7.1 7.5

Any function that returns a logical vector can be used for subsetting:

    a     b     c     e       
FALSE FALSE FALSE FALSE  TRUE

Similarly other functions for identifying special values, e.g.is.nan (Nan values), is.infinite (Inf values), is.finite (values that are not NA, NaN, Inf).

Subsetting by name

We can extract elements by using their name, instead of index:

  a   c 
5.4 7.1

This is usually a much more reliable way to subset objects: the position of various elements can often change when chaining together subsetting operations, but the names will always remain the same!

Unfortunately we can’t skip or remove elements so easily.

To skip (or remove) a single named element we have to find the index of the corresponding column, say “a”:

[1]  TRUE FALSE FALSE FALSE

The condition operator is applied to every name of the vector x. Only the first name is “a” so that element is TRUE.

which then converts this to an index:

[1] 1

Only the first element is TRUE, so which returns 1. Now that we have indices the skipping works because we have a negative index!

  b   c   e 
6.2 7.1 7.5

Skipping multiple named indices is similar, but uses a different comparison operator:

  b   e 
6.2 7.5

The %in% goes through each element of its left argument, in this case the names of x, and asks, “Does this element occur in the second argument?”.

So why can’t we use == like before? That’s an excellent question.

Let’s take a look at just the comparison component:

[1]  TRUE FALSE FALSE FALSE

Obviously “c” is in the names of x, so why didn’t this work? == works slightly differently than %in%. It will compare each element of its left argument to the corresponding element of its right argument.

Here’s a mock illustration:

When one vector is shorter than the other, it gets recycled:

In this case R simply repeats c("a", "c") twice. If the longer vector length isn’t a multiple of the shorter vector length, then R will also print out a warning message:

Warning in names(x) == c("a", "c", "e"): longer object length is not a multiple
of shorter object length

[1]  TRUE FALSE FALSE FALSE

This difference between == and %in% is important to remember, because it can introduce hard to find and subtle bugs!

Matrix subsetting

Matrices are also subsetted using the [ function. In this case it takes two arguments: the first applying to the rows, the second to its columns:

            [,1]       [,2]
[1,]  1.12493092 -0.8356286
[2,] -0.04493361  1.5952808

You can leave the first or second arguments blank to retrieve all the rows or columns respectively:

            [,1]        [,2]
[1,] -0.62124058  0.82122120
[2,] -2.21469989  0.59390132
[3,]  1.12493092  0.91897737
[4,] -0.04493361  0.78213630
[5,] -0.01619026  0.07456498
[6,]  0.94383621 -1.98935170

If we only access one row or column, R will automatically convert the result to a vector:

[1] -0.8356286  0.5757814  1.1249309  0.9189774

If you want to keep the output as a matrix, you need to specify a third argument; drop = FALSE:

           [,1]      [,2]     [,3]      [,4]
[1,] -0.8356286 0.5757814 1.124931 0.9189774

Unlike vectors, if we try to access a row or column outside of the matrix, R will throw an error:

Error in m[, c(3, 6)]: subscript out of bounds

It is useful to note that matrices are laid out in column-major format by default. That is the elements of the vector are arranged column-wise:

     [,1] [,2] [,3]
[1,]    1    3    5
[2,]    2    4    6

If you wish to populate the matrix by row, use byrow = TRUE:

     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6

Matrices can also be subsetted using their rownames and column names instead of their row and column indices.

[1] 2 3 2

     [,1] [,2] [,3]
[1,]    7    9   11
[2,]    8   10   12

List subsetting

Now we’ll introduce some new subsetting operators. There are three functions used to subset lists. [, as we’ve seen for atomic vectors and matrices, as well as [[ and $.

Using [ will always return a list. If you want to subset a list, but not extract an element, then you will likely use [.

$a
[1] "PSRC"

This returns a list with one element.

We can subset elements of a list exactly the same was as atomic vectors using [. Comparison operations however won’t work as they’re not recursive, they will try to condition on the data structures in each element of the list, not the individual elements within those data structures.

$a
[1] "PSRC"

$b
 [1]  1  2  3  4  5  6  7  8  9 10

To extract individual elements of a list, you need to use the double-square bracket function: [[.

[1] "PSRC"

Notice that now the result is a vector, not a list.

You can’t extract more than one element at once:

Error in xlist[[1:2]]: subscript out of bounds

Nor use it to skip elements:

Error in xlist[[-1]]: invalid negative subscript in get1index <real>

But you can use names to both subset and extract elements:

[1] "PSRC"

The $ function is a shorthand way for extracting elements by name:

  city_id hh2016 hh2020 hh2030 hh2040 hh2050     city_name county_id county
1       1   2705   2735   2836   2939   3037 Normandy Park        33   King
2       2  24886  26527  32059  37708  43071        Auburn        33   King
3       3  45021  45724  48094  50515  52813    King-Rural        33   King
4       4  10135  11122  14449  17846  21072        SeaTac        33   King
5       5  22527  23240  25643  28097  30427     Shoreline        33   King
6       6  16769  17481  19881  22332  24658    Renton PAA        33   King

Subsetting data frames

Data frames are lists underneath the hood, so similar rules apply. However they are also two dimensional objects:

[ with one argument will act the same was as for lists, where each list element corresponds to a column. The resulting object will be a data frame:

  city_id
1       1
2       2
3       3
4       4
5       5
6       6

Similarly, [[ will act to extract a single column:

[1] 1 2 3 4 5 6

And $ provides a convenient shorthand to extract columns by name:

[1] "Normandy Park" "Auburn"        "King-Rural"    "SeaTac"       
[5] "Shoreline"     "Renton PAA"

With two arguments, [ behaves the same way as for matrices:

  city_id hh2016 hh2020 hh2030 hh2040 hh2050     city_name county_id county
1       1   2705   2735   2836   2939   3037 Normandy Park        33   King
2       2  24886  26527  32059  37708  43071        Auburn        33   King
3       3  45021  45724  48094  50515  52813    King-Rural        33   King

If we subset a single row, the result will be a data frame (because the elements are mixed types):

  city_id hh2016 hh2020 hh2030 hh2040 hh2050  city_name county_id county
3       3  45021  45724  48094  50515  52813 King-Rural        33   King

But for a single column the result will be a vector (this can be changed with the third argument, drop = FALSE).

Another way of subsetting data frames is using the subset command:

  city_id hh2016 hh2020 hh2030 hh2040 hh2050 city_name county_id county
9       9 329066 344980 398615 453388 505387   Seattle        33   King

[1] 47  9

   city_id hh2016 hh2020 hh2030 hh2040 hh2050    city_name county_id county
29      29  59710  60461  62995  65582  68039 Pierce-Rural        53 Pierce
53      53  13452  14098  16277  18503  20615           UU        53 Pierce
58      59  82851  88758 108668 129001 148304       Tacoma        53 Pierce
73      74   2736   2789   2969   3152   3326   Steilacoom        53 Pierce
74      75   3885   3918   4029   4142   4250         JBLM        53 Pierce
75      76   3530   4131   6156   8224  10188       DuPont        53 Pierce

Challenge 2

Fix each of the following common data frame subsetting errors:

  1. Extract observations for cities in county 33
  1. Extract all columns except one through five
  1. Extract the rows where the number of households in 2016 is larger than 50,000
  1. Extract the first row, and the seventh and eighth columns (city_name and county_id).
  1. Advanced: extract rows that contain information for counties 61 and 35

Challenge 3

  1. Why does hh[1:20] return an error? How does it differ from hh[1:20, ]?

  2. Create a new data.frame called hh_small that only contains rows 1 through 9 and 19 through 23.

Challenge solutions

Solution to challenge 1

Given the following code:

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5

Come up with at least 3 different commands that will produce the following output:

  b   c   d 
6.2 7.1 4.8

Solution to challenge 2

Fix each of the following common data frame subsetting errors:

  1. Extract observations for cities in county 33
  1. Extract all columns except 1 through 5
  1. Extract the rows where the number of households in 2016 is larger than 50,000
  1. Extract the first row, and the seventh and eighth columns (city_name and county_id).
  1. Advanced: extract rows that contain information for counties 61 and 35

Solution to challenge 3

  1. Why does hh[1:20] return an error? How does it differ from hh[1:20, ]?

Answer: hh is a data.frame so needs to be subsetted on two dimensions. hh[1:20, ] subsets the data to give the first 20 rows and all columns.

  1. Create a new data.frame called hh_small that only contains rows 1 through 9 and 19 through 23.