https://www.jottr.org/post/ Recent content in Posts on JottR Hugo -- gohugo.io en-us Thu, 22 Jan 2026 00:00:00 +0000 https://www.jottr.org/2026/01/22/futurize-0.1.0/ Thu, 22 Jan 2026 00:00:00 +0000 https://www.jottr.org/2026/01/22/futurize-0.1.0/ I am incredibly excited to announce the release of the futurize package. This launch marks a major milestone in the decade-long journey of the Futureverse project. Since the inception of the future ecosystem, I (and others) have envisioned a tool that would make concurrent execution as simple as possible with minimal change to your existing code – no refactoring, no new function names to memorize – it should just work and work the same everywhere. https://www.jottr.org/2025/12/03/future.p2p-latinr2025-slides/ Wed, 03 Dec 2025 13:00:00 -0700 https://www.jottr.org/2025/12/03/future.p2p-latinr2025-slides/ Below are the slides for my future.p2p: Peer-to-Peer Compute Clusters in R talk that I presented at the LatinR 2025 online conference. Title: future.p2p: Peer-to-Peer Compute Clusters in R Speaker: Henrik Bengtsson Session: packages Slides: HTML (19 slides; 10 minutes) Video: official recording (10 minutes) I am grateful for the opportunity to present this work at the LatinR conference — LatinR is an amazing community! /Henrik Links LatinR 2025: https://latinr. https://www.jottr.org/2025/10/16/interrupts-crashes-retries/ Thu, 16 Oct 2025 00:00:00 +0000 https://www.jottr.org/2025/10/16/interrupts-crashes-retries/ Interrupts 🛑 Crashes 💥 Retries 🔁 -- The future package celebrates ten years on CRAN as of June 19, 2025. I got a bit stalled over the holidays and while attending the fantastic useR! 2025 conference, but, as promised, here is the fourth in a series of blog posts highlighting recent improvements to the futureverse ecosystem. TL;DR In the past, futures that were interrupted or abruptly terminated were likely putting the future ecosystem in a corrupt state, where you had to manually restart the future backend. https://www.jottr.org/2025/10/08/dofuture-everywhere/ Wed, 08 Oct 2025 00:00:00 +0000 https://www.jottr.org/2025/10/08/dofuture-everywhere/ The future package celebrates ten years on CRAN as of June 19, 2025. I got a bit stalled over the holidays and going to the fantastic useR! 2025 conference, but, as promised, here is the third in a series of blog posts highlighting recent improvements to the futureverse ecosystem. doFuture 1.10.0 is on CRAN as of May 2025 followed by two bug-fix releases in June and July. It comes with several improvements, but here I would like to focus on two new features: https://www.jottr.org/2025/08/14/future.p2p-user2025-slides/ Thu, 14 Aug 2025 17:00:00 -0700 https://www.jottr.org/2025/08/14/future.p2p-user2025-slides/ Below are the slides for my Futureverse P2P: Peer-to-Peer Parallelization in R talk that I presented at the useR! 2025 conference at Duke University, Durham, North Carolina, United States. Title: Futureverse P2P: Peer-to-Peer Parallelization in R – Share compute among friends across the world Speaker: Henrik Bengtsson Session: “Too big to fail”, chaired by Stephanie Zimmer Slides: HTML, PDF (25 slides; 18 minutes) Video: official recording (~18 minutes) https://www.jottr.org/2025/06/25/with-plan/ Wed, 25 Jun 2025 00:00:00 +0000 https://www.jottr.org/2025/06/25/with-plan/ The future package celebrates ten years on CRAN as of June 19, 2025. This is the second in a series of blog posts highlighting recent improvements to the futureverse ecosystem. TL;DR You can now use my_fcn <- function(...) { with(plan(multisession), local = TRUE) ... } to temporarily set a future backend for use in your function. This guarantees that any changes are undone when the function exits, even if there is an error or an interrupt. https://www.jottr.org/2025/06/23/future-got-better-at-finding-global-variables/ Mon, 23 Jun 2025 00:00:00 +0000 https://www.jottr.org/2025/06/23/future-got-better-at-finding-global-variables/ The future package celebrates ten years on CRAN as of June 19, 2025. This is the first in a series of blog posts highlighting recent improvements to the futureverse ecosystem. The globals package is part of the futureverse and has had two recent releases on 2025-04-15 and 2025-05-08. These updates address a few corner cases that would otherwise lead to unexpected errors. They also resulted in several long, outstanding issues reported on the future, future. https://www.jottr.org/2025/06/19/futureverse-10-years/ Thu, 19 Jun 2025 00:00:00 +0000 https://www.jottr.org/2025/06/19/futureverse-10-years/ The future package turns ten on CRAN today – June 19, 2025. (Image credits: Dan LaBar for the future logo; Hadley Wickham and Greg Swinehart for the ggplot2 logo and balloon wall; The future balloon wall was inspired by ggplot2’s recent real-world version and generated with ChatGPT.) The future package turns ten years old today. I released version 0.6.0 to CRAN on June 19, 2015, just days before I presented the package and shared my visions at useR! https://www.jottr.org/2023/07/01/parallelly-managing-workers/ Sat, 01 Jul 2023 18:00:00 +0200 https://www.jottr.org/2023/07/01/parallelly-managing-workers/ parallelly 1.36.0 is on CRAN since May 2023. The parallelly package is part of the Futureverse and enhances the parallel package of base R, e.g. it adds several features you’d otherwise expect to see in parallel. The parallelly package is one of the internal work horses for the future package, but it can also be used outside of the future ecosystem. In this most recent release, parallelly gained several new skills in how cluster nodes (a. https://www.jottr.org/2023/06/26/dofuture/ Mon, 26 Jun 2023 19:00:00 +0200 https://www.jottr.org/2023/06/26/dofuture/ doFuture 1.0.0 is on CRAN since March 2023. It introduces a new foreach operator %dofuture%, which makes it even easier to use foreach() to parallelize via the future ecosystem. This new operator is designed to be an alternative to the existing %dopar% operator for foreach() - an alternative that works in similar ways but better. If you already use foreach() together with futures, or plan on doing so, I recommend using %dofuture% instead of %dopar%. https://www.jottr.org/2023/05/22/future-yegrug-2023-slides/ Mon, 22 May 2023 18:00:00 -0700 https://www.jottr.org/2023/05/22/future-yegrug-2023-slides/ Below are the slides from my presentation at the Edmonton R User Group Meetup (YEGRUG) on May 22, 2023: Title: Futureverse - A Unifying Parallelization Framework in R for Everyone Speaker: Henrik Bengtsson Slides: HTML, PDF (46 slides) Video: official recording (~60 minutes) Thank you Péter Sólymos and the YEGRUG for the invitate and the opportunity! /Henrik Links YEGRUG: https://yegrug.github.io/ Futureverse website: https://www.futureverse.org/ future package CRAN, GitHub, pkgdown https://www.jottr.org/2023/01/18/parallelly-1.34.0-support-for-cgroups-v2-killing-parallel-workers-and-more/ Wed, 18 Jan 2023 14:00:00 -0800 https://www.jottr.org/2023/01/18/parallelly-1.34.0-support-for-cgroups-v2-killing-parallel-workers-and-more/ With the recent releases of parallelly 1.33.0 (2022-12-13) and 1.34.0 (2023-01-13), availableCores() and availableWorkers() gained better support for Linux CGroups, options for avoiding running out of R connections when setting up parallel-style clusters, and killNode() for forcefully terminating one or more parallel workers. I summarize these updates below. For other updates, please see the NEWS. Added support for CGroups v2 availableCores() and availableWorkers() gained support for Linux Control Groups v2 (CGroups v2), besides CGroups v1, which has been supported since parallelly 1. https://www.jottr.org/2023/01/10/progressr-0.13.0/ Tue, 10 Jan 2023 19:00:00 -0800 https://www.jottr.org/2023/01/10/progressr-0.13.0/ progressr 0.13.0 is on CRAN. In the recent releases, progressr gained support for using cli to generate progress bars. Vice versa, cli can now report on progress via the progressr framework. Here are the details. For other updates to progressr, see NEWS. The progressr package, part of the futureverse, provides a minimal API for reporting progress updates in R. The design is to separate the representation of progress updates from how they are presented. https://www.jottr.org/2022/12/05/avoid-detectcores/ Mon, 05 Dec 2022 21:00:00 -0800 https://www.jottr.org/2022/12/05/avoid-detectcores/ The detectCores() function of the parallel package is probably one of the most used functions when it comes to setting the number of parallel workers to use in R. In this blog post, I’ll try to explain why using it is not always a good idea. Already now, I am going to make a bold request and ask you to: Please avoid using parallel::detectCores() in your package! By reading this blog post, I hope you become more aware of the different problems that arise from using detectCores() and how they might affect you and the users of your code. https://www.jottr.org/2022/06/23/future-user2022-slides/ Thu, 23 Jun 2022 17:00:00 -0700 https://www.jottr.org/2022/06/23/future-user2022-slides/ Figure 1: A time chart of logged events for two futures resolved by two parallel workers. This is a screenshot of Slide #18 in my talk. Below are the slides for my Futureverse: Profile Parallel Code talk that I presented at the useR! 2022 conference online and hosted by the Department of Biostatistics at Vanderbilt University Medical Center. Title: Futureverse: Profile Parallel Code Speaker: Henrik Bengtsson https://www.jottr.org/2022/06/09/parallelly-support-for-fujitsu-technical-computing-suite-high-performance-compute-hpc-environments/ Thu, 09 Jun 2022 13:00:00 -0700 https://www.jottr.org/2022/06/09/parallelly-support-for-fujitsu-technical-computing-suite-high-performance-compute-hpc-environments/ parallelly 1.32.0 is now on CRAN. One of the major updates is that availableCores() and availableWorkers(), and therefore also the future framework, gained support for the ‘Fujitsu Technical Computing Suite’ job scheduler. For other updates, please see NEWS. The parallelly package enhances the parallel package - our built-in R package for parallel processing - by improving on existing features and by adding new ones. Somewhat simplified, parallelly provides the things that you would otherwise expect to find in the parallel package. https://www.jottr.org/2022/06/08/parallelly-1.32.0-makeclusterpsock-didnt-work-with-chinese-and-korean-locales/ Wed, 08 Jun 2022 14:00:00 -0700 https://www.jottr.org/2022/06/08/parallelly-1.32.0-makeclusterpsock-didnt-work-with-chinese-and-korean-locales/ parallelly 1.32.0 is on CRAN. This release fixes an important bug that affected users running with the Simplified Chinese, Traditional Chinese (Taiwan), or Korean locale. The bug caused makeClusterPSOCK(), and therefore also future::plan("multisession"), to fail with an error. For other updates, please see NEWS. The parallelly package enhances the parallel package - our built-in R package for parallel processing - by improving on existing features and by adding new ones. https://www.jottr.org/2022/06/03/progressr-0.10.1/ Fri, 03 Jun 2022 13:00:00 -0700 https://www.jottr.org/2022/06/03/progressr-0.10.1/ progressr 0.10.1 is on CRAN. I dedicate this release to all plyr users and developers out there. The progressr package provides a minimal API for reporting progress updates in R. The design is to separate the representation of progress updates from how they are presented. What type of progress to signal is controlled by the developer. How these progress updates are rendered is controlled by the end user. https://www.jottr.org/2022/04/22/parallelly-1.31.1/ Fri, 22 Apr 2022 11:00:00 -0700 https://www.jottr.org/2022/04/22/parallelly-1.31.1/ parallelly 1.31.1 is on CRAN. The parallelly package enhances the parallel package - our built-in R package for parallel processing - by improving on existing features and by adding new ones. Somewhat simplified, parallelly provides the things that you would otherwise expect to find in the parallel package. The future package relies on the parallelly package internally for local and remote parallelization. Since my previous post on parallelly in November 2021, I’ve fixed a few bugs and added some new features to the package; https://www.jottr.org/2022/02/22/future-1.24.0-forwarding-rng-state-also-for-stand-alone-futures/ Tue, 22 Feb 2022 13:00:00 -0800 https://www.jottr.org/2022/02/22/future-1.24.0-forwarding-rng-state-also-for-stand-alone-futures/ future 1.24.0 is on CRAN. It comes with one significant update related to random number generation, further deprecation of legacy future strategies, a slight improvement to plan() and tweaks(), and some bug fixes. Below are the most important changes. One of many possible random number generators. This one was carefully designed by XKCD [CC BY-NC 2.5]. future(…, seed = TRUE) updates RNG state In future (< 1. https://www.jottr.org/2022/01/07/future-during-2021/ Fri, 07 Jan 2022 14:00:00 -0800 https://www.jottr.org/2022/01/07/future-during-2021/ Happy New Year! I made some updates to the future framework during 2021 that involve overall improvements and essential preparations to go forward with some exciting new features that I’m keen to work on during 2022. The future framework makes it easy to parallelize existing R code - often with only a minor change of code. The goal is to lower the barriers so that anyone can quickly and safely speed up their existing R code in a worry-free manner. https://www.jottr.org/2021/11/22/parallelly-1.29.0/ Mon, 22 Nov 2021 21:00:00 -0800 https://www.jottr.org/2021/11/22/parallelly-1.29.0/ parallelly 1.29.0 is on CRAN. The parallelly package enhances the parallel package - our built-in R package for parallel processing - by improving on existing features and by adding new ones. Somewhat simplified, parallelly provides the things that you would otherwise expect to find in the parallel package. The future package rely on the parallelly package internally for local and remote parallelization. Since my previous post on parallelly five months ago, the parallelly package had some bugs fixed, and it gained a few new features; https://www.jottr.org/2021/08/23/matrixstats-gsoc-2021/ Mon, 23 Aug 2021 00:10:00 +0200 https://www.jottr.org/2021/08/23/matrixstats-gsoc-2021/ Author: Angelina Panagopoulou, GSoC student developer, undergraduate in the Department of Informatics & Telecommunications (DIT), University of Athens, Greece We are glad to announce recent CRAN releases of matrixStats with support for handling and returning name attributes. This feature is added to make matrixStats functions handle names in the same manner as the corresponding base R functions. In particular, the behavior of matrixStats functions is now the same as the apply() function in R, resolving previous lack of, or inconsistent, handling of row and column names. https://www.jottr.org/2021/06/11/progressr-0.8.0/ Fri, 11 Jun 2021 19:00:00 -0700 https://www.jottr.org/2021/06/11/progressr-0.8.0/ progressr 0.8.0 is on CRAN. It comes with some new features: A new ‘rstudio’ handler that reports on progress via the RStudio job interface in RStudio withProgressShiny() now updates the detail part, instead of the message part In addition to signalling relative amounts of progress, it’s now also possible to signal total amounts If you’re curious what progressr is about, have a look at my e-Rum 2020 presentation. https://www.jottr.org/2021/06/10/parallelly-1.26.0/ Thu, 10 Jun 2021 15:00:00 -0700 https://www.jottr.org/2021/06/10/parallelly-1.26.0/ parallelly 1.26.0 is on CRAN. It comes with one major improvement and one new function: The setup of parallel workers is now much faster, which comes from using a concurrent, instead of sequential, setup strategy The new freePort() can be used to find a TCP port that is currently available Faster setup of local, parallel workers In R 4.0.0, which was released in May 2020, parallel::makeCluster(n) gained the power of setting up the n local cluster nodes all at the same time, which greatly reduces to total setup time. https://www.jottr.org/2021/04/30/parallelly-1.25.0/ Fri, 30 Apr 2021 15:00:00 -0700 https://www.jottr.org/2021/04/30/parallelly-1.25.0/ A piece of an ice core - more pleasing to look at than yet another illustration of a CPU core (Image credit: Ludovic Brucker, NASA’s Goddard Space Flight Center) parallelly 1.25.0 is on CRAN. It comes with two major improvements: You can now use availableCores(omit = n) to ask for all but n CPU cores makeClusterPSOCK() can finally use the built-in SSH client on MS Windows 10 to set up remote workers https://www.jottr.org/2021/04/08/future-and-kubernetes/ Thu, 08 Apr 2021 19:00:00 -0700 https://www.jottr.org/2021/04/08/future-and-kubernetes/ This is a guest post by Chris Paciorek, Department of Statistics, University of California at Berkeley. In this post, I’ll demonstrate that you can easily use the future package in R on a cluster of machines running in the cloud, specifically on a Kubernetes cluster. This allows you to easily doing parallel computing in R in the cloud. One advantage of doing this in the cloud is the ability to easily scale the number and type of (virtual) machines across which you run your parallel computation. https://www.jottr.org/2021/01/08/future.batchjobs-end-of-life-announcement/ Fri, 08 Jan 2021 09:00:00 -0800 https://www.jottr.org/2021/01/08/future.batchjobs-end-of-life-announcement/ This is an announcement that future.BatchJobs - A Future API for Parallel and Distributed Processing using BatchJobs has been archived on CRAN. The package has been deprecated for years with a recommendation of using future.batchtools instead. The latter has been on CRAN since June 2017 and builds upon the batchtools package, which itself supersedes the BatchJobs package. To wrap up the three-and-a-half year long life of future. https://www.jottr.org/2020/12/19/future-eurobioc2020-slides/ Sat, 19 Dec 2020 10:00:00 -0800 https://www.jottr.org/2020/12/19/future-eurobioc2020-slides/ Luke Zappia's summary of the talk I presented Future: A Simple, Extendable, Generic Framework for Parallel Processing in R at the European Bioconductor Meeting 2020, which took place online during the week of December 14-18, 2020. You’ll find my slides (39 slides + Q&A slides; 35 minutes) below: Title & Abstract HTML (Google Slides; requires online access) PDF (flat slides) Video (YouTube) I want to thank the organizers for inviting me to this Bioconductor conference. https://www.jottr.org/2020/11/12/future-nycmeetup-slides/ Thu, 12 Nov 2020 19:30:00 -0800 https://www.jottr.org/2020/11/12/future-nycmeetup-slides/ I presented Future: Simple, Friendly Parallel Processing for R (67 minutes; 59 slides + Q&A slides) at New York Open Statistical Programming Meetup, on November 9, 2020: HTML (incremental Google Slides; requires online access) PDF (flat slides) Video (presentation starts at 0h10m30s, Q&A starts at 1h17m40s) I like to thanks everyone who attented and everyone who asked lots of brilliant questions during the Q&A. I’d also want to express my gratitude to Amada, Jared, and Noam for the invitation and making this event possible. https://www.jottr.org/2020/11/06/future-1.20.1-the-future-just-got-a-bit-brighter/ Fri, 06 Nov 2020 13:00:00 -0800 https://www.jottr.org/2020/11/06/future-1.20.1-the-future-just-got-a-bit-brighter/ future 1.20.1 is on CRAN. It adds some new features, deprecates old and unwanted behaviors, adds a couple of vignettes, and fixes a few bugs. Interactive debugging First out among the new features, and a long-running feature request, is the addition of argument split to plan(), which allows us to split, or “tee”, any output produced by futures. The default is split = FALSE for which standard output and conditions are captured by the future and only relayed after the future has been resolved, i. https://www.jottr.org/2020/11/04/parallelly-future-cleaning-up-around-the-house/ Wed, 04 Nov 2020 18:00:00 -0800 https://www.jottr.org/2020/11/04/parallelly-future-cleaning-up-around-the-house/ parallelly adverb par·​al·​lel·​ly | \ ˈpa-rə-le(l)li \ Definition: in a parallel manner future noun fu·​ture | \ ˈfyü-chər \ Definition: existing or occurring at a later time I’ve cleaned up around the house - with the recent release of future 1.20.1, the package gained a dependency on the new parallelly package. Now, if you’re like me and concerned about bloating package dependencies, I’m sure you immediately wondered why I chose to introduce a new dependency. https://www.jottr.org/2020/11/04/trust-the-future/ Wed, 04 Nov 2020 14:00:00 -0800 https://www.jottr.org/2020/11/04/trust-the-future/ Each time we use R to analyze data, we rely on the assumption that functions used produce correct results. If we can’t make this assumption, we have to spend a lot of time validating every nitty detail. Luckily, we don’t have to do this. There are many reasons for why we can comfortably use R for our analyses and some of them are unique to R. Here are some I could think of while writing this blog post - I’m sure I forgot something: https://www.jottr.org/2020/09/22/push-for-statistical-sound-rng/ Tue, 22 Sep 2020 19:00:00 -0700 https://www.jottr.org/2020/09/22/push-for-statistical-sound-rng/ Parallel ‘Digital Rain’ by Jahobr After two-and-a-half months, future 1.19.1 is now on CRAN. As usual, there are some bug fixes and minor improvements here and there (NEWS), including things needed by the next version of furrr. For those of you who use Slurm or LSF/OpenLava as a scheduler on your high-performance compute (HPC) cluster, future::availableCores() will now do a better job respecting the CPU resources that those schedulers allocate for your R jobs. https://www.jottr.org/2020/09/21/detect-when-the-random-number-generator-was-used/ Mon, 21 Sep 2020 18:45:00 -0700 https://www.jottr.org/2020/09/21/detect-when-the-random-number-generator-was-used/ If you ever need to figure out if a function call in R generated a random number or not, here is a simple trick that you can use in an interactive R session. Add the following to your ~/.Rprofile(*): if (interactive()) { invisible(addTaskCallback(local({ last <- .GlobalEnv$.Random.seed function(...) { curr <- .GlobalEnv$.Random.seed if (!identical(curr, last)) { msg <- "TRACKER: .Random.seed changed" if (requireNamespace("crayon", quietly=TRUE)) msg <- crayon::blurred(msg) message(msg) last <<- curr } TRUE } }), name = "RNG tracker")) } It works by checking whether or not the state of the random number generator (RNG), that is, . https://www.jottr.org/2020/07/11/future-future.apply-recent-improvements/ Sat, 11 Jul 2020 22:15:00 -0700 https://www.jottr.org/2020/07/11/future-future.apply-recent-improvements/ There are new versions of future and future.apply - your friends in the parallelization business - on CRAN. These updates are mostly maintenance updates with bug fixes, some improvements, and preparations for upcoming changes. It’s been some time since I blogged about these packages, so here is the summary of the main updates this far since early 2020: future: values() for lists and other containers was renamed to value() to simplify the API [future 1. https://www.jottr.org/2020/07/04/progressr-erum2020-slides/ Sat, 04 Jul 2020 17:30:00 -0700 https://www.jottr.org/2020/07/04/progressr-erum2020-slides/ Source: Wiktionary.org I presented Progressr: An Inclusive, Unifying API for Progress Updates (15 minutes; 20 slides) at e-Rum 2020, on June 17, 2020: Abstract HTML (incremental Google Slides; requires online access) PDF (flat slides) Video (starts at 00h49m58s) I am grateful for everyone involved who made e-Rum 2020 possible. I cannot imagine having to cancel the on-site Milano conference that had planned for more than a year and then start over to re-organize and create a fabulous online experience for ~1,500 participants in such short notice. https://www.jottr.org/2020/02/01/future-rstudioconf2020-slides/ Sat, 01 Feb 2020 19:30:00 -0800 https://www.jottr.org/2020/02/01/future-rstudioconf2020-slides/ Design: Dan LaBar I presented Future: Simple Async, Parallel & Distributed Processing in R Why and What’s New? at rstudio::conf 2020 in San Francisco, USA, on January 29, 2020. Below are the slides for my talk (17 slides; ~18+2 minutes): HTML (incremental Google Slides; requires online access) PDF (flat slides) Video with closed captions (official rstudio::conf recording) First of all, a big thank you goes out to Dan LaBar (@embiggenData) for proposing and contributing the original design of the future hex sticker. https://www.jottr.org/2019/11/09/resolved-launches-lazy-futures/ Sat, 09 Nov 2019 11:00:00 -0800 https://www.jottr.org/2019/11/09/resolved-launches-lazy-futures/ No dogs were harmed while making this release future 1.15.0 is now on CRAN, accompanied by a recent, related update of future.callr 0.5.0. The main update is a change to the Future API: resolved() will now also launch lazy futures Although this change does not look much to the world, I’d like to think of this as part of a young person slowly finding themselves. This change in behavior helps us in cases where we create lazy futures upfront; https://www.jottr.org/2019/07/12/future-user2019-slides/ Fri, 12 Jul 2019 16:00:00 +0200 https://www.jottr.org/2019/07/12/future-user2019-slides/ Below are the slides for my Future: Simple Parallel and Distributed Processing in R that I presented at the useR! 2019 conference in Toulouse, France on July 9-12, 2019. My talk (25 slides; ~15+3 minutes): Title: Future: Simple Parallel and Distributed Processing in R HTML (incremental Google Slides; requires online access) PDF (flat slides) Video (official recording) I want to send out a big thank you to everyone making the useR! https://www.jottr.org/2019/05/26/startup-sometimes/ Sun, 26 May 2019 21:00:00 -0700 https://www.jottr.org/2019/05/26/startup-sometimes/ New release: startup 0.12.0 is now on CRAN. This version introduces support for processing some of the R startup files with a certain frequency, e.g. once per day, once per week, or once per month. See below for two examples. startup::startup() is cross platform. The startup package makes it easy to split up a long, complicated .Rprofile startup file into multiple, smaller files in a .Rprofile.d/ folder. For instance, setting R option repos in a separate file ~/. https://www.jottr.org/2019/05/16/future-satrdayla2019-slides/ Thu, 16 May 2019 12:00:00 -0800 https://www.jottr.org/2019/05/16/future-satrdayla2019-slides/ A bit late but here are my slides on Future: Friendly Parallel Processing in R for Everyone that I presented at the satRday LA 2019 conference in Los Angeles, CA, USA on April 6, 2019. My talk (33 slides; ~45 minutes): Title: : Friendly Parallel and Distributed Processing in R for Everyone HTML (incremental slides; requires online access) PDF (flat slides) Video (44 min; YouTube; sorry, different page numbers) Thank you all for making this a stellar satRday event. https://www.jottr.org/2019/03/07/future-satrdayparis2019-slides/ Thu, 07 Mar 2019 12:00:00 -0800 https://www.jottr.org/2019/03/07/future-satrdayparis2019-slides/ Below are links to my slides from my talk on Future: Friendly Parallel Processing in R for Everyone that I presented last month at the satRday Paris 2019 conference in Paris, France (February 23, 2019). My talk (32 slides; ~40 minutes): Title: Future: Friendly Parallel Processing in R for Everyone HTML (incremental slides; requires online access) PDF (flat slides) A big shout out to the organizers, all the volunteers, and everyone else for making it a great satRday. https://www.jottr.org/2019/01/11/parallelize-a-for-loop-by-rewriting-it-as-an-lapply-call/ Fri, 11 Jan 2019 12:00:00 -0800 https://www.jottr.org/2019/01/11/parallelize-a-for-loop-by-rewriting-it-as-an-lapply-call/ A commonly asked question in the R community is: How can I parallelize the following for-loop? The answer almost always involves rewriting the for (...) { ... } loop into something that looks like a y <- lapply(...) call. If you can achieve that, you can parallelize it via for instance y <- future.apply::future_lapply(...) or y <- foreach::foreach() %dopar% { ... }. For some for-loops it is straightforward to rewrite the code to make use of lapply() instead, whereas in other cases it can be a bit more complicated, especially if the for-loop updates multiple variables in each iteration. https://www.jottr.org/2019/01/07/maintenance-updates-of-future-backends-and-dofuture/ Mon, 07 Jan 2019 00:00:00 +0000 https://www.jottr.org/2019/01/07/maintenance-updates-of-future-backends-and-dofuture/ New versions of the following future backends are available on CRAN: future.callr - parallelization via callr, i.e. on the local machine future.batchtools - parallelization via batchtools, i.e. on a compute cluster with job schedulers (SLURM, SGE, Torque/PBS, etc.) but also on the local machine future.BatchJobs - (maintained for legacy reasons) parallelization via BatchJobs, which is the predecessor of batchtools These releases fix a few small bugs and inconsistencies that were identified with help of the future. https://www.jottr.org/2018/07/23/output-from-the-future/ Mon, 23 Jul 2018 00:00:00 +0000 https://www.jottr.org/2018/07/23/output-from-the-future/ future 1.9.0 - Unified Parallel and Distributed Processing in R for Everyone - is on CRAN. This is a milestone release: Standard output is now relayed from futures back to the master R session - regardless of where the futures are processed! Disclaimer: A future’s output is relayed only after it is resolved and when its value is retrieved by the master R process. In other words, the output is not streamed back in a “live” fashion as it is produced. https://www.jottr.org/2018/07/21/suppressgraphics/ Sat, 21 Jul 2018 00:00:00 +0000 https://www.jottr.org/2018/07/21/suppressgraphics/ R.devices 2.16.0 - Unified Handling of Graphics Devices - is on CRAN. With this release, you can now easily suppress unwanted graphics, e.g. graphics produced by one of those do-everything-in-one-call functions that we all bump into once in a while. To suppress graphics, the R.devices package provides graphics device nulldev(), and function suppressGraphics(), which both send any produced graphics into the void. This works on all operating systems, including Windows. https://www.jottr.org/2018/06/23/future.apply_1.0.0/ Sat, 23 Jun 2018 00:00:00 +0000 https://www.jottr.org/2018/06/23/future.apply_1.0.0/ Got compute? future.apply 1.0.0 - Apply Function to Elements in Parallel using Futures - is on CRAN. With this milestone release, all* base R apply functions now have corresponding futurized implementations. This makes it easier than ever before to parallelize your existing apply(), lapply(), mapply(), … code - just prepend future_ to an apply call that takes a long time to complete. That’s it! The default is sequential processing but by using plan(multisession) it’ll run in parallel. https://www.jottr.org/2018/06/18/future-erum2018-slides/ Mon, 18 Jun 2018 00:00:00 +0000 https://www.jottr.org/2018/06/18/future-erum2018-slides/ As promised - though a bit delayed - below are links to my slides and the video of my talk on Future: Parallel & Distributed Processing in R for Everyone that I presented last month at the eRum 2018 conference in Budapest, Hungary (May 14-16, 2018). The conference was very well organized (thank you everyone involved) with a great lineup of several brilliant workshop sessions, talks, and poster presentations (thanks all). https://www.jottr.org/2018/04/12/future-results/ Thu, 12 Apr 2018 00:00:00 +0000 https://www.jottr.org/2018/04/12/future-results/ future 1.8.0 is available on CRAN. This release lays the foundation for being able to capture outputs from futures, perform automated timing and memory benchmarking (profiling) on futures, and more. These features are not yet available out of the box, but thanks to this release we will be able to make some headway on many of the feature requests related to this - hopefully already by the next release. https://www.jottr.org/2018/04/02/coercion-of-indices/ Mon, 02 Apr 2018 00:00:00 +0000 https://www.jottr.org/2018/04/02/coercion-of-indices/ x[idxs + 1] or x[idxs + 1L]? That is the question. Assume that we have a vector $x$ of $n = 100,000$ random values, e.g. > n <- 100000 > x <- rnorm(n) and that we wish to calculate the $n-1$ first-order differences $y=(y_1, y_2, …, y_{n-1})$ where $y_i=x_{i+1} - x_i$. In R, we can calculate this using the following vectorized form: > idxs <- seq_len(n - 1) > y <- x[idxs + 1] - x[idxs] We can certainly do better if we turn to native code, but is there a more efficient way to implement this using plain R code? https://www.jottr.org/2018/03/30/startup-secrets/ Fri, 30 Mar 2018 00:00:00 +0000 https://www.jottr.org/2018/03/30/startup-secrets/ New release: startup 0.10.0 is now on CRAN. If your R startup files (.Renviron and .Rprofile) get long and windy, or if you want to make parts of them public and other parts private, then you can use the startup package to split them up in separate files and directories under .Renviron.d/ and .Rprofile.d/. For instance, the .Rprofile.d/repos.R file can be solely dedicated to setting in the repos option, which specifies from which web servers R packages are installed from. https://www.jottr.org/2017/06/05/many-faced-future/ Mon, 05 Jun 2017 00:00:00 +0000 https://www.jottr.org/2017/06/05/many-faced-future/ The future package defines the Future API, which is a unified, generic, friendly API for parallel processing. The Future API follows the principle of write code once and run anywhere - the developer chooses what to parallelize and the user how and where. The nature of a future is such that it lends itself to be used with several of the existing map-reduce frameworks already available in R. In this post, I’ll give an example of how to apply a function over a set of elements concurrently using plain sequential R, the parallel package, the future package alone, as well as future in combination of the foreach, the plyr, and the purrr packages. https://www.jottr.org/2017/04/01/history-r-help-20-years/ Sat, 01 Apr 2017 00:00:00 +0000 https://www.jottr.org/2017/04/01/history-r-help-20-years/ Today, its been 20 years since Martin Mächler started the R-help community list. The first post was written by Ross Ihaka on 1997-04-01: Screenshot of the very first post to the R-help mailing list. This is a post about R’s memory model. We’re talking R v0.50 beta. I think that the paragraph at the end provides a nice anecdote on the importance not to be overwhelmed by problems ahead: ”(The consumption of one cell per string is perhaps the major memory problem in R - we didn’t design it with large problems in mind. https://www.jottr.org/2017/03/18/dofuture/ Sat, 18 Mar 2017 00:00:00 +0000 https://www.jottr.org/2017/03/18/dofuture/ doFuture 0.4.0 is available on CRAN. The doFuture package provides a universal foreach adaptor enabling any future backend to be used with the foreach() %dopar% { ... } construct. As shown below, this will allow foreach() to parallelize on not only multiple cores, multiple background R sessions, and ad-hoc clusters, but also cloud-based clusters and high performance compute (HPC) environments. 1,300+ R packages on CRAN and Bioconductor depend, directly or indirectly, on foreach for their parallel processing. https://www.jottr.org/2017/02/19/future-rng/ Sun, 19 Feb 2017 00:00:00 +0000 https://www.jottr.org/2017/02/19/future-rng/ future 1.3.0 is available on CRAN. With futures, it is easy to write R code once, which the user can choose to evaluate in parallel using whatever resources s/he has available, e.g. a local machine, a set of local machines, a set of remote machines, a high-end compute cluster (via future.BatchJobs and soon also future.batchtools), or in the cloud (e.g. via googleComputeEngineR). Futures makes it easy to harness any resources at hand. https://www.jottr.org/2016/12/22/startup/ Thu, 22 Dec 2016 00:00:00 +0000 https://www.jottr.org/2016/12/22/startup/ The startup package makes it easy to control your R startup processes and to share part of your startup settings with others (e.g. as a public Git repository) while keeping secret parts to yourself. Instead of having long and windy .Renviron and .Rprofile startup files, you can split them up into short specific files under corresponding .Renviron.d/ and .Rprofile.d/ directories. For example, # Environment variables # (one name=value per line) . https://www.jottr.org/2016/10/22/future-hpc/ Sat, 22 Oct 2016 00:00:00 +0000 https://www.jottr.org/2016/10/22/future-hpc/ A new version of the future.BatchJobs package has been released and is available on CRAN. With a single change of settings, it allows you to switch from running an analysis sequentially on a local machine to running it in parallel on a compute cluster. Our different futures can easily be resolved on high-performance compute clusters. Requirements The future.BatchJobs package implements the Future API, as defined by the future package, on top of the API provided by the BatchJobs package. https://www.jottr.org/2016/10/11/future-remotes/ Tue, 11 Oct 2016 00:00:00 +0000 https://www.jottr.org/2016/10/11/future-remotes/ A new version of the future package has been released and is available on CRAN. With futures, it is easy to write R code once, which later the user can choose to parallelize using whatever resources s/he has available, e.g. a local machine, a set of local notebooks, a set of remote machines, or a high-end compute cluster. The future provides comfortable and friendly long-distance interactions. The new version, future 1. https://www.jottr.org/2016/07/02/future-user2016-slides/ Sat, 02 Jul 2016 00:00:00 +0000 https://www.jottr.org/2016/07/02/future-user2016-slides/ Unless you count DSC 2003 in Vienna, last week’s useR conference at Stanford was my very first time at useR. It was a great event, it was awesome to meet our lovely and vibrant R community in real life, which we otherwise only get know from online interactions, and of course it was very nice to meet old friends and make new ones. The future is promising. At the end of the second day, I presented A Future for R (18 min talk; slides below) on how you can use the future package for asynchronous (parallel and distributed) processing using a single unified API regardless of what backend you have available, e. https://www.jottr.org/2015/12/16/matrixstats-subsetting/ Wed, 16 Dec 2015 00:00:00 +0000 https://www.jottr.org/2015/12/16/matrixstats-subsetting/ The matrixStats package provides highly optimized functions for computing common summaries over rows and columns of matrices. In a previous blog post, I showed that, instead of using apply(X, MARGIN = 2, FUN = median), we can speed up calculations dramatically by using colMedians(X). In the most recent release (version 0.50.0), matrixStats has been extended to perform optimized calculations also on a subset of rows and/or columns specified via new arguments rows and cols, e. https://www.jottr.org/2015/08/12/milestone-cran-7000/ Wed, 12 Aug 2015 00:00:00 +0000 https://www.jottr.org/2015/08/12/milestone-cran-7000/ Another 1,000 packages were added to CRAN, which took less than 9 months. Today (August 12, 2015), the Comprehensive R Archive Network (CRAN) package page reports: “Currently, the CRAN package repository features 7002 available packages.” While the previous 1,000 packages took 355 days, going from 6,000 to 7,000 packages took 286 days - which means that now a new CRAN package is born on average every 6.9 hours (or 3. https://www.jottr.org/2015/06/05/checkuserinterrupt/ Fri, 05 Jun 2015 00:00:00 +0000 https://www.jottr.org/2015/06/05/checkuserinterrupt/ If your native code takes more than a few seconds to finish, it is a nice courtesy to the user to check for user interrupts (Ctrl-C) once in a while, say, every 1,000 or 1,000,000 iteration. The C-level API of R provides R_CheckUserInterrupt() for this (see ‘Writing R Extensions’ for more information on this function). Here’s what the code would typically look like: for (int ii = 0; ii < n; ii++) { /* Some computational expensive code */ if (ii % 1000 == 0) R_CheckUserInterrupt() } This uses the modulo operator % and tests when it is zero, which happens every 1,000 iteration. https://www.jottr.org/2015/03/12/matrixstats-gsoc/ Thu, 12 Mar 2015 00:00:00 +0000 https://www.jottr.org/2015/03/12/matrixstats-gsoc/ We are pleased to announce our proposal ‘Subsetted and parallel computations in matrixStats’ for Google Summer of Code. The project is aimed for a student with experience in R and C, it runs for three months, and the student gets paid 5500 USD by Google. Students from (almost) all over the world can apply. Application deadline is March 27, 2015. I, Henrik Bengtsson, and Héctor Corrada Bravo will be joint mentors. https://www.jottr.org/2015/02/21/how-to-plain-latex-vignettes/ Sat, 21 Feb 2015 00:00:00 +0000 https://www.jottr.org/2015/02/21/how-to-plain-latex-vignettes/ Ever wanted to include a plain-LaTeX vignette in your package and have it compiled into a PDF? The R.rsp package provides a four-line solution for this. But, first, what’s R.rsp? R.rsp is an R package that implements a compiler for the RSP markup language. RSP can be used to embed dynamic R code in any text-based source document to be compiled into a final document, e.g. RSP-embedded LaTeX into PDF, RSP-embedded Markdown into HTML, RSP-embedded HTML into HTML and so on. https://www.jottr.org/2015/01/25/matrixstats-0.13.1/ Sun, 25 Jan 2015 00:00:00 +0000 https://www.jottr.org/2015/01/25/matrixstats-0.13.1/ A new release 0.13.1 of matrixStats is now on CRAN. The source code is available on GitHub. What does it do? The matrixStats package provides highly optimized functions for computing common summaries over rows and columns of matrices, e.g. rowQuantiles(). There are also functions that operate on vectors, e.g. logSumExp(). Their implementations strive to minimize both memory usage and processing time. They are often remarkably faster compared to good old apply() solutions. https://www.jottr.org/2014/10/29/milestone-cran-6000/ Wed, 29 Oct 2014 00:00:00 +0000 https://www.jottr.org/2014/10/29/milestone-cran-6000/ Another 1,000 packages were added to CRAN and this time in less than 12 months. Today (2014-10-29) on The Comprehensive R Archive Network (CRAN) package page: “Currently, the CRAN package repository features 6000 available packages.” Going from 5,000 to 6,000 packages took 355 days - which means that it on average was only ~8.5 hours between each new packages added. It is actually even more frequent since dropped packages are not accounted for. https://www.jottr.org/2014/06/17/matrixna-wrong-way/ Tue, 17 Jun 2014 00:00:00 +0000 https://www.jottr.org/2014/06/17/matrixna-wrong-way/ Are you a good R citizen and preallocates your matrices? If you are allocating a numeric matrix in one of the following two ways, then you are doing it the wrong way! x <- matrix(nrow = 500, ncol = 100) or x <- matrix(NA, nrow = 500, ncol = 100) Why? Because it is counter productive. And why is that? In the above, x becomes a logical matrix, and not a numeric matrix as intended. https://www.jottr.org/2014/05/26/captureoutput/ Mon, 26 May 2014 00:00:00 +0000 https://www.jottr.org/2014/05/26/captureoutput/ The R function capture.output() can be used to “collect” the output of functions such as cat() and print() to strings. For example, > s <- capture.output({ + cat("Hello\nworld!\n") + print(pi) + }) > s [1] "Hello" "world!" "[1] 3.141593" More precisely, it captures all output sent to the standard output and returns a character vector where each element correspond to a line of output. By the way, it does not capture the output sent to the standard error, e. https://www.jottr.org/2013/05/25/trick-fast-rm/ Sat, 25 May 2013 00:00:00 +0000 https://www.jottr.org/2013/05/25/trick-fast-rm/ When processing large data sets in R you often also end up creating large temporary objects. In order to keep the memory footprint small, it is always good to remove those temporary objects as soon as possible. When done, removed objects will be deallocated from memory (RAM) the next time the garbage collection runs. Better: Use rm(list = "x") instead of rm(x), if using rm() To remove an object in R, one can use the rm() function (with alias remove()). https://www.jottr.org/2013/04/01/history-r-help/ Mon, 01 Apr 2013 00:00:00 +0000 https://www.jottr.org/2013/04/01/history-r-help/ Today it’s 16 years ago and 367,496 messages later since Martin Mächler started the R-help (321,119 msgs), R-devel (45,830 msgs) and R-announce (547 msgs) mailing lists [1] - a great benefit to all of us. Special thanks to Martin and also thanks to everyone else contributing to these forums. [1] https://stat.ethz.ch/pipermail/r-help/1997-April/001490.html https://www.jottr.org/2013/01/07/trick-unlist/ Mon, 07 Jan 2013 00:00:00 +0000 https://www.jottr.org/2013/01/07/trick-unlist/ Sometimes a minor change to your R code can make a big difference in processing time. Here is an example showing that if you’re don’t care about the names attribute when unlist():ing a list, specifying argument use.names = FALSE can speed up the processing lots! > x <- split(sample(1000, size = 1e6, rep = TRUE), rep(1:1e5, times = 10)) > t1 <- system.time(y1 <- unlist(x)) > t2 <- system.time(y2 <- unlist(x, use. https://www.jottr.org/2012/10/22/config-help-start/ Mon, 22 Oct 2012 00:00:00 +0000 https://www.jottr.org/2012/10/22/config-help-start/ The below code shows how to configure the help.ports option in R such that the built-in R help server always uses the same URL port. Just add it to the .Rprofile file in your home directory (iff missing, create it). For more details, see help("Startup"). # Force the URL of the help to http://127.0.0.1:21510 options(help.ports = 21510) A slighter fancier version is to use a environment variable to set the port(s): https://www.jottr.org/2012/09/27/config-repos/ Thu, 27 Sep 2012 00:00:00 +0000 https://www.jottr.org/2012/09/27/config-repos/ The below code shows how to configure the repos option in R such that install.packages() etc. will locate the packages without having to explicitly specify the repository. Just add it to the .Rprofile file in your home directory (iff missing, create it). For more details, see help("Startup"). local({ repos <- getOption("repos") # http://cran.r-project.org/ # For a list of CRAN mirrors, see getCRANmirrors(). repos["CRAN"] <- "http://cran.stat.ucla.edu" # http://www.stats.ox.ac.uk/pub/RWin/ReadMe if (.Platform$OS.type == "windows") { repos["CRANextra"] <- "http://www.