Apply functions in R
R's apply functions are the efficient way to perform repeated
operations on vectors of your data. There are many variants of the "apply"
statement as well as wrappers for special cases of these
functions easier. These notes will cover the apply statements and
wrappers that are most commonly used in data analysis. Much of what we discuss
here carries over to other apply statements as well.
apply
The first function we'll look at is apply. This function applies a given
function to each element of a data structure with defined dimension
(i.e. matrices, arrays, data frames). The elements that apply operates on are
the rows/columns of matrices and data frames (or the sub-matrices of arrays). In
other words, if you want to apply some function to the rows or columns of your
data, you should use the apply function. When you call apply, you must
supply the function with the object you want to operate on (X), the dimension(s)
you would like to apply the function over (MARGIN), and the function you would
like to perform (FUN). There are a few things to note about these arguments
before proceeding to our examples.
-
Xmust have positive dimension. You can easily check this by running dim(X). Note that vectors and lists do not satisfy this requirement. -
For the
MARGINargument, 1 indicates rows and 2 indicates columns. If you are using higher dimensional arrays, integers that correspond to any of the other dimensions are accepatable as well. -
you may specify additional arguments to your function following the
FUNargument.
apply will return the vector, matrix or list that results from applying FUN
to each row or column as defined by MARGIN. Whenever possible, apply will
try to simplify the results of its return into a matrix or vector. If FUN
returns objects with different lengths when applied to the various rows/columns,
the returned object will be a list. Lets look at a few examples.
sample.matrix <- matrix(sample(1:15, 15), ncol = 5)
sample.matrix
## [,1] [,2] [,3] [,4] [,5]
## [1,] 15 8 12 10 2
## [2,] 7 11 5 6 14
## [3,] 3 4 13 1 9
# basic apply statements that calculate the mean of rows and columns
row.means <- apply(sample.matrix, 1, mean)
col.means <- apply(sample.matrix, 2, mean)
row.means
## [1] 9.4 8.6 6.0
col.means
## [1] 8.333 7.667 10.000 5.667 8.333
# faster way to accomplish the same thing
row.means <- rowMeans(sample.matrix)
col.means <- colMeans(sample.matrix)
row.means
## [1] 9.4 8.6 6.0
col.means
## [1] 8.333 7.667 10.000 5.667 8.333
A few of the functions that are frequently used by statisticians (sum
and mean) have slightly faster implementations of an apply statment used to
do the same thing. However, using an apply statement allows us to pass in
additional arguments.
# you can pass in additional arguments after defining the function. Any
# arguments not passed in will use the function's default value
row.means.trimmed <- apply(sample.matrix, 1, mean, trim = 0.35)
col.means.trimmed <- apply(sample.matrix, 2, mean, trim = 0.35)
row.means.trimmed
## [1] 10.000 8.000 5.333
col.means.trimmed
## [1] 7 8 12 6 9
You can also define your own functions for the FUN argument. User defined
functions will take the row or column as their argument depending on the value
you specified for MARGIN.
overall.mean <- mean(sample.matrix)
row.value <- apply(sample.matrix, 1, function(row) {
value <- (mean(row) - overall.mean)/sd(row)
return(value)
})
col.value <- apply(sample.matrix, 2, function(col) {
value <- (mean(col) - overall.mean)/sd(col)
return(value)
})
row.value
## [1] 0.2870 0.1587 -0.4082
col.value
## [1] 0.05455 -0.09492 0.45883 -0.51745 0.05530
As we mentioned, when our functions return objects of varying lengths apply
will return a list.
param.matrix <- matrix(c(5, 10, 0, 5), ncol = 2)
param.matrix
## [,1] [,2]
## [1,] 5 0
## [2,] 10 5
# generate a list of length with 5 samples of mean 0 normal variables and 10
# samples of mean 5 variables
apply(param.matrix, 1, function(row) rnorm(row[1], row[2]))
## [[1]]
## [1] -0.2734 0.2565 1.0476 2.1080 0.1782
##
## [[2]]
## [1] 5.059 3.139 6.402 6.015 4.699 5.704 4.677 5.341 5.737 4.789
sapply (lapply)
The next functions that we will look at are similar to apply in that they
are used to loop over a data structure and apply some function to the elements
of that data structure. Unlike apply, they operate on data structures with
undefined dimension (i.e. lists and vectors). These functions are sapply and
lapply. Actually, sapply is just a wrapper for lapply that tries to
simplify the returned value into a vector or matrix where possible. We use
sapply in our examples with the knowledge that if we wanted to return a list
we could use lapply instead. When you call sapply (or lapply) you need to
supply it with the data structure that will get looped over (X) and a function
to apply to each element of that data structure (FUN). There are a few things
to note
- Running
sapplyon a data frame will performFUNon each of the vectors in that comprise it (remember data frames are just lists of vectors of equal length). - Running
sapplyon a matrix or array will coerce the object into a list and then applyFUNto the individual elements.
sample.list <- list(samp1 = rnorm(10, 0), samp2 = rnorm(10, 5), samp3 = rnorm(10,
10))
# The objects we call sapply on don't have defined dimension so we don't
# need a MARGIN argument. Other than that the function calls are similar
sapply(sample.list, mean)
## samp1 samp2 samp3
## -0.4709 4.4661 10.5416
sapply(sample.list, mean, trim = 0.35)
## samp1 samp2 samp3
## -0.3927 4.4029 10.6850
sapply(sample.list, function(list.element) {
value <- mean(list.element)/sd(list.element)
})
## samp1 samp2 samp3
## -0.3283 9.3778 9.5501
When we call sapply (or lapply) on a data frame it acts on the vectors that
make up the data frame.
# built in R dataset on regional levels of different socioeconomic
# indicators in Switzerland
head(swiss)
## Fertility Agriculture Examination Education Catholic
## Courtelary 80.2 17.0 15 12 9.96
## Delemont 83.1 45.1 6 9 84.84
## Franches-Mnt 92.5 39.7 5 5 93.40
## Moutier 85.8 36.5 12 7 33.77
## Neuveville 76.9 43.5 17 15 5.16
## Porrentruy 76.1 35.3 9 7 90.57
## Infant.Mortality
## Courtelary 22.2
## Delemont 22.2
## Franches-Mnt 20.2
## Moutier 20.3
## Neuveville 20.6
## Porrentruy 26.6
swiss.subset <- swiss[, c("Fertility", "Agriculture", "Education", "Infant.Mortality")]
# lets look the mean and standard deviation of each of the variables
sapply(swiss.subset, mean)
## Fertility Agriculture Education Infant.Mortality
## 70.14 50.66 10.98 19.94
sapply(swiss.subset, sd)
## Fertility Agriculture Education Infant.Mortality
## 12.492 22.711 9.615 2.913
# we can even plot histograms of the data by variable
par(mfrow = c(2, 2)) # this is a nice function to put multiple plots on one page
sapply(swiss.subset, hist)
Unfortunately, the names for the histograms aren't particularly helpful. You could try to adjust this by supplying additional arguments to the hist function
var.names <- names(swiss.subset)
var.names
## [1] "Fertility" "Agriculture" "Education"
## [4] "Infant.Mortality"
par(mfrow = c(2, 2))
sapply(swiss.subset, hist, xlab = "observed value", main = var.names)
Instead of labeling each plotwith the correct variable name, R prints every
string in var.names for each plot. Not exactly what we were looking for. This
is because the functions we apply to the elements of our data structure are
constant. In other words, the only inputs to FUN that change are the elements
of our data structure. Thankfully, R provides the function mapply to help us
get around this.
mapply
The mapply function works much like sapply or lapply. With mapply
though, our functions are not limited to a single varying argument. The call for
mapply is a little different from the other apply functions. We first need to
specify our function then we provide the data structure that the function
operates on. Our histogram example would look something like this:
var.names <- names(swiss.subset)
par(mfrow = c(2, 2))
mapply(function(variable, var.name) hist(variable, main = var.name, xlab = "observed value"),
swiss.subset, var.names)
Notice that the order we provide the different variables at the end of our
mapply statement needs to be the same as the order they are called in the
function. Further, mapply is not restricted to only two variables. As long as
we make sure the orders correspond we may use as many variables as we like.
Apply wrappers: by and replicate
Some apply function tasks are so common amongst R users that the language
includes wrappers for them. Chief among these are by and replicate. by
statements are useful when trying to run functions on various subsets of your
data. Consider the "mtcars" dataset that is built into R and suppose we wanted
to calculate the average mpg of a car by number of cylinders in the engine.
head(mtcars)
## mpg cyl disp hp drat wt qsec vs am gear carb
## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
## Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
# the arguments for a by statment are data, subset factor, function in that
# order
by(mtcars$mpg, mtcars$cyl, mean)
## mtcars$cyl: 4
## [1] 26.66
## --------------------------------------------------------
## mtcars$cyl: 6
## [1] 19.74
## --------------------------------------------------------
## mtcars$cyl: 8
## [1] 15.1
# we can use sapply statement to look at the average of several variables by
# number of cylinders
sapply(mtcars[, c("mpg", "hp", "wt")], function(var) by(var, mtcars$cyl, mean))
## mpg hp wt
## 4 26.66 82.64 2.286
## 6 19.74 122.29 3.117
## 8 15.10 209.21 3.999
replicate statements are useful for simulations and fairly simple to
implement. These statements produce a specified number of replicates of a given
statement. For instance, if we wanted to produce 10 simulations of sampling 25
random uniform variables
replicate(10, runif(25))
## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
## [1,] 0.5748392 0.28749 0.00349 0.44652 0.13699 0.3699 0.51512 0.69841
## [2,] 0.0010590 0.93278 0.47812 0.87892 0.35814 0.3688 0.77206 0.40563
## [3,] 0.0001907 0.89475 0.87006 0.78308 0.70703 0.3261 0.42233 0.14024
## [4,] 0.3188565 0.47157 0.86951 0.13302 0.91671 0.2523 0.86594 0.86434
## [5,] 0.9853210 0.81585 0.45488 0.40766 0.90607 0.6450 0.47121 0.12542
## [6,] 0.7024356 0.90947 0.46666 0.53455 0.69911 0.2381 0.62024 0.94814
## [7,] 0.2249691 0.66061 0.70574 0.47489 0.73907 0.2238 0.55454 0.61841
## [8,] 0.1353921 0.20309 0.27294 0.05341 0.42293 0.9872 0.47972 0.49680
## [9,] 0.4470449 0.44555 0.24052 0.05491 0.08967 0.6046 0.17333 0.27853
## [10,] 0.9385163 0.67798 0.95618 0.99885 0.45088 0.1313 0.97006 0.07001
## [11,] 0.0982774 0.50816 0.56374 0.94642 0.49484 0.7884 0.77314 0.63851
## [12,] 0.2711856 0.86542 0.54552 0.20941 0.12294 0.7534 0.63229 0.27112
## [13,] 0.4084631 0.37047 0.48257 0.78127 0.80352 0.3549 0.08938 0.35646
## [14,] 0.3299767 0.99052 0.89554 0.68312 0.74825 0.3454 0.71481 0.37773
## [15,] 0.9147518 0.37893 0.17761 0.18310 0.35805 0.6820 0.26136 0.86034
## [16,] 0.4389611 0.11486 0.20609 0.37435 0.74195 0.0718 0.76355 0.30815
## [17,] 0.5641979 0.12247 0.86493 0.22845 0.53164 0.7486 0.23534 0.70235
## [18,] 0.9340997 0.11293 0.68794 0.76306 0.68938 0.4741 0.77426 0.29306
## [19,] 0.7925815 0.65003 0.09922 0.08144 0.46391 0.3626 0.91598 0.03533
## [20,] 0.5656450 0.46962 0.49972 0.04007 0.04784 0.8274 0.50144 0.48898
## [21,] 0.1160394 0.30752 0.92019 0.57434 0.68835 0.1342 0.18251 0.36334
## [22,] 0.0960271 0.66276 0.14329 0.08054 0.98979 0.4877 0.93420 0.96727
## [23,] 0.0618901 0.21379 0.27903 0.32398 0.71879 0.8374 0.35423 0.78674
## [24,] 0.3905563 0.66687 0.95751 0.20393 0.94135 0.7348 0.14536 0.76835
## [25,] 0.8456175 0.08976 0.15553 0.13075 0.74546 0.6280 0.23254 0.62025
## [,9] [,10]
## [1,] 0.30124 0.68865
## [2,] 0.67112 0.72139
## [3,] 0.56038 0.99757
## [4,] 0.23748 0.59152
## [5,] 0.07987 0.50159
## [6,] 0.63697 0.91020
## [7,] 0.71425 0.58123
## [8,] 0.07934 0.18153
## [9,] 0.91799 0.85539
## [10,] 0.77144 0.37545
## [11,] 0.43682 0.64109
## [12,] 0.55022 0.43939
## [13,] 0.74331 0.58395
## [14,] 0.01913 0.72981
## [15,] 0.20714 0.95380
## [16,] 0.24238 0.55779
## [17,] 0.16606 0.59220
## [18,] 0.13465 0.03029
## [19,] 0.74797 0.40250
## [20,] 0.11388 0.11504
## [21,] 0.04791 0.58220
## [22,] 0.69066 0.83256
## [23,] 0.00699 0.73928
## [24,] 0.49927 0.18991
## [25,] 0.91362 0.01659