Automatically downloading and processing TRMM rainfall data

TRMM rainfall data maybe is the most accurate rainfall data derived from satellite measurements and a valuable source in regions with scarse weather-stations. It has a good spatial (0.25°) and temporal (daily) resolution and is available since 1998. However, downloading and processing may be a lot of work, if not scripted. The following script may be badly coded, but it works. All you need is the open source software R. This script is written for a Linux environment.

First, load some packages:

library(raster)
 library(rgdal)
 library(RCurl)

Then, download the data from the FTP server: ftp://disc2.nascom.nasa.gov/data/TRMM/Gridded/. This example downloads 3B43 monthly data version 7 for 2012. The first line has to be changed for the desired data:

url = "ftp://disc2.nascom.nasa.gov/data/TRMM/Gridded/3B43_V7/2012/"
filenames = getURL(url, ftp.use.epsv = FALSE, ftplistonly=TRUE, crlf=TRUE)
 filenames = paste(url, strsplit(filenames, "\r*\n") [[1]], sep="")
filenames = filenames[grep(filenames, pattern="*precipitation.accum")] # this line allows to download only files with a certain pattern, e.g. only certain months or days. "*precipitation.accum" means monthly accumulated rainfall here.
filenames # list all files
mapply(download.file, filenames, basename(filenames)) # download files

Now I create a virtual file as the downloaded TRMM data come as binarys. The VRT-file contains all the downloaded binary files with the appropriate geo-informations. To automate the process, the following script generates the VRT-file (TRMM.vrt) for all 2012 data. Change “3B43.12” accordingly to the downloaded data. Save this example as trmm2012.sh.

echo '<VRTDataset rasterXSize="1440" rasterYSize="400"> 
 <Geotransform>-180,0.25,0,50, 0, -0.25</Geotransform> 
 <SRS>WGS84</SRS>' >TRMM.vrt;
for i in 3B43.12*
do echo '<VRTRasterBand dataType="Float32" band="1" subClass="VRTRawRasterBand"> 
 <SourceFilename relativeToVRT="1">'$i'</SourceFilename> 
 <ByteOrder>MSB</ByteOrder>
 <ImageOffset>0</ImageOffset>
 <PixelOffset>4</PixelOffset>
 <LineOffset>5760</LineOffset>
 </VRTRasterBand>' >>TRMM.vrt
done;
echo '</VRTDataset>' >>TRMM.vrt

Within R, the script (trmm2012.sh) is executed and the virtual-file (TRMM.vrt) loaded as a rasterbrick. This is flipped in y direction and the files written as multilayer Geotiff. This Geotiff contains all the layers for 2012 and can be opened in every GIS software.

system("./trmm2012.sh")
 b=brick("TRMM.vrt")
 b
 trmm = flip(b, direction='y')
 writeRaster(trmm, filename="trmm_acc_2012.tif", format="GTiff", overwrite=TRUE)
Advertisements

30 thoughts on “Automatically downloading and processing TRMM rainfall data

  1. It doesn’t work properly.
    Final values are incorrect:
    min values : -3.397748e+38, -3.358339e+38, -3.356937e+38, -3.379672e+38, -3.381341e+38, -3.379652e+38, -3.392844e+38, -3.394530e+38, -3.399306e+38, -3.393002e+38, -3.392979e+38, -3.385703e+38
    max values : 3.400084e+38, 3.391985e+38, 3.379663e+38, 3.401695e+38, 3.379725e+38, 3.380926e+38, 3.392971e+38, 3.386065e+38, 3.363687e+38, 3.400968e+38, 3.367997e+38, 3.398838e+38

    Any idea?

    Thnak you

  2. Pingback: The new “hddtools”, an R package for Hydrological Data Discovery | Claudia @ ICL

  3. Hi Martin.
    I have a problem when I execute the trmm2012.sh.
    I get the following error.
    I need your help I am running on Windows with Cygwin.
    regards.
    KARLOZ@KARLOZ-PC /cygdrive/c/Program Files/R/R-3.0.2/library/hddtools
    $ bash trmm2012.sh
    trmm2012.sh: línea 3: $’\r’: no se encontró la orden
    trmm2012.sh: línea 12: $’\r’: no se encontró la orden

  4. Hello,
    I am new in R and I will like to download the daily TRMM data (1998-2014) of my study area (all Tanzania)
    thanks for your help

  5. brick(“Filename.vrt”) is not recognized by library(raster). Or detailed:
    `Path\TRMM.vrt’ does not exist in the file system, and is not recognised as a supported dataset name.

      • According to documentation it(brick()) should be able to read everything that reads readGDAL(), but R says there is no such a function. So I guess it is x64 Windows related problem…
        going to test script on Ubuntu to find out. I was not going to say that your script is “fake”.
        Could You specify version of library(raster) installed on your R?

      • -180,0.25,0,50, 0, -0.25
        WGS84

        3B43.120601.7.precipitation.accum
        MSB
        0
        4
        5760

        Also could you say if anything wrong in in generated VRT(above first entry)? Thank you

      • Well on Ubuntu it works fine, after installing RGdal package with some workarounds. Made it work on Windows too, and opened result image with MapWindowGIS. It looks like random color pixels are set in bounding box. Is it normal?

  6. For those who are on windows platform be warned:
    “writeRaster(trmm, filename=”trmm_acc_2012.tif”, format=”GTiff”, overwrite=TRUE)”
    remove underscores from [filename=”trmm_acc_2012.tif”] => [filename=”trmmacc2012.tif”]
    I lost tons of time because of this, R however for some reason was giving me error on VRT:
    `Path\TRMM.vrt’ does not exist in the file system, and is not recognised as a supported dataset name.

  7. I am using 64 bit windows 7 and and got the problem. Could you help me. Error is as follows:

    `C:\Users\pc\Documents\TRMM.vrt’ does not exist in the file system,
    and is not recognised as a supported dataset name.

    In addition: Warning message:
    running command ‘sh ~/myTRMM.sh’ had status 127
    Error in .rasterObjectFromFile(x, objecttype = “RasterBrick”, …) :
    Cannot create a RasterLayer object from this file. (file does not exist)

  8. Hi Martin! Thanks for sharing your code! I adapted it to download 3B42_V7 data:

    library(raster)
    library(rgdal)
    library(RCurl)
    library(lubridate)

    setwd(“~/”)
    url <- "ftp://disc2.nascom.nasa.gov/data/TRMM/Gridded/3B42_V7/&quot;

    #defining period
    seq <- seq( ymd("1998-01-01"), ymd("2016-05-01"), by="month")
    seqn <- as.numeric(gsub("-", "", substring(seq, 1, 7)))

    for (i in 1:length(seqn)){
    urlt <- paste(url,seqn[i],"/",sep="")
    filenames <- getURL(urlt, http://ftp.use.epsv = FALSE, ftplistonly=TRUE, crlf=TRUE)
    filenames <- paste(urlt, strsplit(filenames, "\r*\n") [[1]], sep="")
    filenames <- filenames[grep(filenames, pattern="*precipitation.bin")] # this line allows to download only files with a certain pattern, e.g. only certain months or days. "*precipitation.accum" means monthly accumulated rainfall here.
    mapply(download.file, filenames, basename(filenames)) # download files
    }

    Cheers,

    Diego

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s