Upload
elz0rr0
View
234
Download
0
Embed Size (px)
Citation preview
7/28/2019 Foreach Iterators - Lewis
1/25
foreach + iterators
Bryan Lewis Steve Weston
Revolution ComputingNew Haven, CT USA
Rmetrics 2009
http://find/7/28/2019 Foreach Iterators - Lewis
2/25
Outline
iterators
foreach
Experimenting with existing packages
http://find/7/28/2019 Foreach Iterators - Lewis
3/25
iterators
An S3 class with tools for iterating over various R data structures:
Conceptually like while loops
Defined by a nextElem function
Like iterators in Java and other languages
http://find/7/28/2019 Foreach Iterators - Lewis
4/25
Simple Examples
it
7/28/2019 Foreach Iterators - Lewis
5/25
Another example
iquery
7/28/2019 Foreach Iterators - Lewis
6/25
foreach
New looping methods for R
An abstract interface to parallel computing Python/Haskell-like list comprehensions
http://find/http://goback/7/28/2019 Foreach Iterators - Lewis
7/25
Foreach Syntax
foreach(iterator,...) %dopar% {statements
}
http://find/7/28/2019 Foreach Iterators - Lewis
8/25
Example
> foreach (j=1:4) %dopar% { j }
[[1]]
[1] 1
[[2]][1] 2
[[3]]
[1] 3
[[4]]
[1] 4
http://find/7/28/2019 Foreach Iterators - Lewis
9/25
Examples
> foreach (j=1:4,.combine=c) %dopar% { j }
[ 1 ] 1 2 3 4
> foreach (j=icount(4),.combine=+) %dopar% { j }
[1] 10
Note the difference withsum (1:4).
http://find/7/28/2019 Foreach Iterators - Lewis
10/25
Another Example
> library (randomForest)
> x y rf
7/28/2019 Foreach Iterators - Lewis
11/25
The %dopar% operator
%dopar% is a registration API for parallel back-ends:
doSEQ (the default backend)
doMC (multicore package)
doNWS doSNOW
http://find/7/28/2019 Foreach Iterators - Lewis
12/25
The %dopar% operator
%dopar% is a registration API for parallel back-ends:
doSEQ (the default backend)
doMC (multicore package)
doNWS doSNOW
doRHIPE?
doRMPI?
. . .
http://find/http://goback/7/28/2019 Foreach Iterators - Lewis
13/25
Foreach tries to parse R syntax reasonably
> z < - 2
> f foreach (j=1:4, .combine=c) %dopar% { f (j) }
[1] 1.732051 2.000000 2.236068 2.449490
http://find/7/28/2019 Foreach Iterators - Lewis
14/25
List comprehension
> foreach (j=-2:2,.combine=c) %:% when (j>=0)
+ %dopar% sqrt (j)
[1] 0.000000 1.000000 1.414214
http://find/http://goback/7/28/2019 Foreach Iterators - Lewis
15/25
Nesting
Foreach loops can be nested. Nesting admits at least two
interesting cases:
Easy loop unrolling
Easy multi-paradigm parallelism
L lli
http://find/7/28/2019 Foreach Iterators - Lewis
16/25
Loop unrolling
Compare (100 iterations of 5 parallel tasks):x
7/28/2019 Foreach Iterators - Lewis
17/25
Multi-paradigm parallelism
> require (doSNOW)
> cl registerDoSNOW (cl)
> foreach (j=, .packages=doMC) %dopar% {+ foreach (k=) %dopar% {
+ registerDoMC ()
+ ...
+ }
+ }
E l V i l b kt ti
http://find/http://goback/7/28/2019 Foreach Iterators - Lewis
18/25
Example: Very simple backtesting
simpleRule
7/28/2019 Foreach Iterators - Lewis
19/25
Brute-force parameter optimization
# Define a return series Ra for the instrument# (below we use the closing price of MSFT), and
# benchmark series Rb
M
7/28/2019 Foreach Iterators - Lewis
20/25
Now in parallel, by rows...
M
7/28/2019 Foreach Iterators - Lewis
21/25
Parallelizing parts of an existing package
Basic idea
Profile code with Rprof (profr is a nice wrapper that visualizes
the results)
Examine bottlenecks for apply-like statements and for loops
with independent code blocks
Rewrite for loops without side-effects as required (may require
a custom combine function)
Unlock the namespace, provisionally replace target
function(s) and experiment (a nice trick)
Example: ipred
http://find/7/28/2019 Foreach Iterators - Lewis
22/25
Example: ipred
(Work through the ipred replacement functions in the filecvx.R.)
Appendix: Fun map/reduce examples
http://find/7/28/2019 Foreach Iterators - Lewis
23/25
Appendix: Fun map/reduce examples
Succint map/reduce...from the mapReduce package by
Christopher Brown:
mapReduce
7/28/2019 Foreach Iterators - Lewis
24/25
mapReduce sequential and parallel examples
# An example
mapReduce (cyl, mean(mpg), mean(hp),data=mtcars, applyfun=sapply)
# With multicore:
require (mutlicore)
mapReduce (cyl, mean(mpg), mean(hp),
data=mtcars, applyfun=mclapply)
# With SNOW parSapply:
require (snow)
cl
7/28/2019 Foreach Iterators - Lewis
25/25
mapReduce parallel examples
# With Rmpi mpi.parSapply:
require (Rmpi)
x