Import data

The amd_read_base() imports a BASE file, either from a zipped file or an unzipped comma-separated file (.csv). The parse_timestamp parameter can be used if additional time-keeping columns (e.g., year, month, day, hour) are desired.

# read the BASE from a zip file, without additional parsed time-keeping columns
base1 <- amf_read_base(
  file = floc2,
  unzip = TRUE,
  parse_timestamp = FALSE
)
pander::pandoc.table(base1[c(1:3),])

# read the BASE from a csv file, with additional parsed time-keeping columns
base2 <- amf_read_base(
  file = floc3,
  unzip = FALSE,
  parse_timestamp = TRUE
)
pander::pandoc.table(base2[c(1:3), c(1:10)])

Parse and interpret data

The details of the BASE data product’s format and variable definitions can be found on AmeriFlux website. In short, the BASE data product contains flux, meteorological, and soil observations that are reported at regular intervals of time, generally half-hourly or hourly, for a certain time period. TIMESTAMP_START and TIMESTAMP_END columns (i.e., YYYYMMDDHHMM 12 digits) denote the starting and ending time of each reporting interval (i.e., row).

All other variables use the format of {base name}_{qualifier}, e.g., FC_1, CO2_1_1_1. Base names indicate fundamental quantities that are either measured or calculated / derived. Qualifiers are suffixes appended to variable base names that provide additional information (e.g., gap-filling, position) about the variable. In some cases, qualifiers are omitted if only one variable is provided for a site.

The amf_variable() retrieves the latest list of base names and default units. For sites that have relatively fewer variables and less complicated qualifiers, the users could easily interpret variables and qualifiers. The amf_variable() also returns the expected maximal and minimal values based on physically plausible ranges or network reported values.

# get a list of latest base names and units. 
FP_ls <- amf_variables()
pander::pandoc.table(FP_ls[c(11:20), ])

Alternatively, the amf_parse_basename() can programmatically parse the the variable names into base names and qualifiers. This function can be helpful for sites with many variables and relatively complicated qualifiers, as a prerequisite for handling data from many sites. The function returns a data frame with information about each variable’s base name, qualifier, and whether a variable is gap-filled, layer-aggregated, or replicate aggregated.

# parse the variable name
basename_decode <- amf_parse_basename(var_name = colnames(base1))
pander::pandoc.table(basename_decode[c(1, 2, 3, 4, 6, 11, 12),])

Data filtering

While BASE data products are quality-checked before release, the data may not be filtered for all outliers. The amf_filter_base() can be use to filter the data based on the expected physically ranges (i.e., obtained through amf_variables()). By default, a ±5% buffer is applied to account for possible edge values near the lower and upper bounds, which are commonly observed for certain variables like radiation, relative humidity, and snow depth.

# filter data, using default physical range +/- 5% buffer
base_f <- amf_filter_base(data_in = base1)

Measurement height information

Measurement height information contains height/depth and instrument model information of the BASE data products. The info can be downloaded directly using the amf_var_info() function. The function returns a data frame for all available sites, and can be subset using the “Site_ID” column. The “Height” column refers to the distance from the ground surface in meters. Positive values are heights, and negative values are depths. See the web page for explanation.

# obtain the latest measurement height information
var_info <- amf_var_info()

# subset the variable by target Site ID
var_info <- var_info[var_info$Site_ID == "US-CRT", ]
pander::pandoc.table(var_info[c(1:10), ])

Import BADM data

Biological, Ancillary, Disturbance, and Metadata (BADM) are non-continuous information that describe and complement continuous flux and meteorological data (e.g., BASE data product). BADM include general site description, metadata about the sensors and their setup, maintenance and disturbance events, and biological and ecological data that characterize a site’s ecosystem. See link for details.

The amf_read_bif() can be used to import the BADM data file. The function returns a data frame for all available sites, and can subset using the “SITE_ID” column.

# read the BADM BIF file, using an example data file
bif <- amf_read_bif(file = floc1)

# subset by target Site ID
bif <- bif[bif$SITE_ID == "US-CRT", ]
pander::pandoc.table(bif[c(1:15), ])

# get a list of all BADM variable groups and variables
unique(bif$VARIABLE_GROUP)

[1] “GRP_ACKNOWLEDGEMENT” “GRP_CLIM_AVG” “GRP_COUNTRY”
[4] “GRP_DOI” “GRP_DOI_CONTRIBUTOR” “GRP_DOI_ORGANIZATION” [7] “GRP_DOM_DIST_MGMT” “GRP_FLUX_MEASUREMENTS” “GRP_HEADER”
[10] “GRP_HEIGHTC” “GRP_IGBP” “GRP_LAND_OWNERSHIP”
[13] “GRP_LOCATION” “GRP_NETWORK” “GRP_REFERENCE_PAPER”
[16] “GRP_SHIPPING_ADDRESS” “GRP_SITE_CHAR” “GRP_SITE_DESC”
[19] “GRP_SITE_FUNDING” “GRP_STATE” “GRP_TEAM_MEMBER”
[22] “GRP_TOWER_POWER” “GRP_TOWER_TYPE” “GRP_URL”
[25] “GRP_URL_AMERIFLUX” “GRP_UTC_OFFSET”

length(unique(bif$VARIABLE))

[1] 59

As shown above, BADM data contain information from a variety of variable groups (i.e., GRP_{BADM_GROUPS}). Browse the definitions of all available variable groups here.

To get the BADM data for a certain variable group, use amf_extract_badm() function. The function also renders the data format (i.e., display all variables by columns) for human readability.

# extract the FLUX_MEASUREMENTS group
bif_flux <- amf_extract_badm(bif_data = bif, select_group = "GRP_FLUX_MEASUREMENTS")
pander::pandoc.table(bif_flux)

# extract the HEIGHTC (canopy height) group
bif_hc <- amf_extract_badm(bif_data = bif, select_group = "GRP_HEIGHTC")
pander::pandoc.table(bif_hc)

Note: amf_extract_badm() returns all columns in characters. Certain groups of BADM variables contain columns of time stamps (i.e., ISO format) and data values, and need to be converted before further use.

# convert HEIGHTC_DATE to POSIXlt
bif_hc$TIMESTAMP <- strptime(bif_hc$HEIGHTC_DATE, format = "%Y%m%d", tz = "GMT")

# convert HEIGHTC column to numeric
bif_hc$HEIGHTC <- as.numeric(bif_hc$HEIGHTC)

# plot time series of canopy height
plot(bif_hc$TIMESTAMP, bif_hc$HEIGHTC, xlab = "TIMESTAMP", ylab = "canopy height (m)")

Last, the contacts of the site members and data DOI can be obtained from the BADM data. The AmeriFlux data policy requires proper attribution (e.g., data DOI).

In some case, for example, using data shared under Legacy Data Policy for publication, data users are required to contact data contributors directly, so that they have the opportunity to contribute substantively and become a co-author.

# get a list of contacts
bif_contact <- amf_extract_badm(bif_data = bif, select_group = "GRP_TEAM_MEMBER")
pander::pandoc.table(bif_contact)

# get data DOI
bif_doi <- amf_extract_badm(bif_data = bif, select_group = "GRP_DOI")
pander::pandoc.table(bif_doi)