---
title: "PLANTPOPNET Data Cleaning"
output:
  html_document: 
   code_folding: show
---

```{r date, echo = FALSE}
date <- format(Sys.Date(), "%d %B, %Y")
```

# `r date`

# Packages required
```{r, packages, echo = T}
library(dplyr)
library(stringr)
```


# Introduction 
This is the primary file for plantpopnet data cleaning.It is designed to run on the compiled data set (.csv) and has been tested with updated data-sets. 

Each error was assigned a unique identifier. Once an error was identified by the diagnostic, the solution can be called from a script (of the same identifier name) and solved, before moving on to the next error. Error logs should be recorded in annotated text within this document. Error types were initially diagnosed and solved for a static test compiled data file (ca. 7000 plants). Once this script was complete and tested successfully on the test data using both Mac and PC platforms, a new static compiled data file was used for further testing and error diagnosis (complete data from 2014- May 2019). 

Create an R project in a new folder and copy all files provided to this working directory, including this file.

```{r, git, login}
# this has been removed for anonymisation, static files are provided with the manuscript files for review

```


# Load in Data
The demographic census file (for year 0 (Y0) only) is input as 'data_orig' which is kept unchanged for reference. Ensure that sheet 1 in the excel file is the datasheet. All changes will be made to 'mydata'.  NOTE: using the "xlsx" package does not require data file to be in CSV format. 
The site description file is input as 'sitedata' as it is used in some variable diagnosis and cleaning.
```{r, load_data}

data_orig <- read.csv("fullPPN_datasetY0_2020-11-18_.csv",header = T, na.strings = c("NA", "", "Na", "na", "nA"))
mydata <- data_orig 

sitedata <- read.csv("Coordinates_Oct2020_site_level.csv", header = T)
```

```{r, setup_df}

#str(mydata)

#change site code to factor
mydata$site_code <- as.factor(mydata$site_code)
mydata$s_year <- as.factor(mydata$s_year)
mydata$survival <- as.factor(mydata$survival)

```

```{r, tidy_sitedata}
## tidy up structure of sitedata 
str(sitedata)
sitedata$site_code <- as.factor(sitedata$site_code)

```

# Global Fixes
## Global1 - blank rows
Script for a function to output data frame which removes rows/cases with all NA variables. Takes 'mydata' and outputs 'mydata'.
Applies function to the dataset.
```{r, remove_rows}
source("Global1_remove_na_rows_func.R")

#1428 NA rows removed (Oct 2020)
#3467 NA rows removed (March 2021)

```
## Global2 - Rename rosette_number column
The naming of separate variables 'no_of_rosettes' (total number of rosettes recorded for a unique plant) and 'rosette_number' (disambiguation of rosettes within a plant, typically not retained in consecutive years) were confusing. We rename 'rosette number' to 'rosette_ID' but note that this ID can change from year to year as individual rosettes are not usually tagged.
```{r, rename_rosette}
names(mydata)[names(mydata) == "rosette_number"] <- "rosette_ID"
```

## Global 3 Operators & functions used in multiple scripts
"Not in" operator is defined.
```{r, operators}
## create "not in" operator
'%nin%' = Negate('%in%')
```


# Variables
Code chunks should be run in the sequence below, where variable number equals col number in mydata. Some variables are sequenced out of order in this markdown as they are required to be cleaned before a variable which comes before them numerically. 

### Var1 Site code
1. Identify empty site code (NA) and resolve
2. Checking the site codes in mydata match the Site description codes in sitedata

```{r, site_code}
# levels(mydata$site_code)

source("var1_site_code.R")


# levels(mydata$site_code)

#notes: giving false warning messages about all site codes 
## fixed in var1_site_code.R


```



### Var 2:5 GPS Coordinates.
Lat and Lon must be stored in decimal degrees. The measurements package can be used to convert other formats.

This script contains a number of issues and checks throughout. There is alot of manual checking and fixing required. May require additional input when new sites are added. Refer to comments within script and you may need to run commented code for diagnosis when new sites are added.

** NOTE: 20/10/2020 this code is buggy - but not essential for the rest of the codes to work - see the site summary file for site level coordinates, do not use the transect coordinates **

solution
```{r, coordinates}
Sys.setlocale('LC_ALL','C') ## needed to deal with some encoding issues (Mac to PC) if all ok print-out will read "[1] "C/C/C/C/C/en_IE.UTF-8" " . No action necessary


source("var2-5_coordinate_locations.R")
```

### Var 6_7 Transect and Plot
levels(mydata$plot)

1. a,b,c,d plots are from HV site which surveyed using a 1m squared quadrat. We divided into 4 plots to make 50cm squared as per PPN protocol. These are to be left as is for now

2. Duplicate plot numbers for P01-P09 and P1-P9. remove 0s for consistency.

3. Plot numbers should range between 1 & 20. One sites continues to P43. Will ignore for now as Plot number is unique and should not cause any issues 


levels(mydata$transect)
1. replicate of T2 with a space

solution
```{r, transect_plot2}
source("var6_7_transect_plot_numbers.R")

```

### Var 8 No_seedlings
Variable is a factor with text na - needs to be numerical

Solution
```{r, seedlings}
source("var8_number_seedling.R")

```

### Var 9 Plant ID
Fixes issues with missing plant IDs and non-unique IDs. Creates a new variable called plant_unique_id

```{r, plantID}
source("var9_PlantID.R")


```

### Go back and run Transect & Plot
Run the Transect &Plot script again to fix problems with NA's in the BG transect/plot names

```{r, transect_plot}

source("var6_7_transect_plot_numbers.R")

```
### Var 10 & 11 Within plot Coordinates (x,y)
Variable is a factor - should be numerical and between 0 and 50

```{r, xy_coords_detect, Eval=FALSE}
##levels(mydata$x_coord)
##levels(mydata$y_coord)



```

Solution
```{r, xy_coords, echo=T}
source("var10_11_xy_coords.R")


######### Note Values still range above 50

```


### Var 12 suspected_clone
tasks accomplished in this script:
suspected_clone_binary column created 
suspected_clone column renamed and preserved as suspected_clone_notes

solution
```{r, clone, echo=T}
source("var12_suspected_clone_binary.R")

```
### Var 28_29 c_year & s_year
This script must be run before "survival"
Any errors found will be printed with a warning
Script is diagnostic only, does not make changes to source data
```{r, c_year_s_year, echo=T}
source("var28_29_c_year_and_s_year.R")
```

### Var 13 survival
Ensuring the survival column is binary (yes/no)
``` {r, survival_check2, echo = F, eval=F}
write.csv(mydata, "mydata_surv_check.csv") ## use this data file for checks to survival code - includes all changes up to here
```

```{r, survival, echo=T}

source("var13_survival.R")

```

### Var 14 No. of rosettes
Checking no_rosette column, should be a numeric value
```{r, no_rosettes, echo=T }
source("var14_no_rosettes.r")  
```

SURVIVAL SCRIPT NEEDS TO BE RE-RUN  
Need to re-run plant survival script to ensure that correct values are recorded in plant_survival due to changes in survival as a result of no_rosettes tidying
```{r, no_rosettes_survival, echo = T}
source("var13_survival.R")

```

``` {r, rosette_check, echo = F, eval=F}
write.csv(mydata, "mydata_ros_check.csv") ## use this data file for checks to rosette ID code - includes all changes up to here
```



### Var 16:21 Outlier detection
searching for outliers in the numerical columns

```{r, outliers, eval=T}
source("var16-21_outlier_detection.R")

```
 
### Var 23_24 Disease_yes_no
ensuring binary yes/no values in column

solution
```{r, disease, echo=T}
source("var23_disease_yes_no.R")


levels(mydata$disease..yes.no.)
```

### Var 25_26 Herbivory
```{r, herbivory, echo=T}
source("var25_herbivory_yes_no.R")


levels(mydata$herbivory..yes.no.)
```


### Var 15 Rosette ID
Site AC has only recorded 1 rosette for multi-rosette plants. No_leaves column is total number for the whole plant not just the rosette from which leaf/inflor measurements are taken.
The script below just diagnoses issues with number of rosettes and rosette id, no changes are made to the data.

```{r, rosette_id, echo = T}
source("var15_rosette_id.r")  
```


# Production of Y0 data product
```{r, year0 } 
mydataY0 <- subset(mydata, s_year == "Y0")

```

## Export Y0

```{r, export_data_Y0}
output_name <- paste("PLANTPOPNET_Y0_V1.4",Sys.Date(), ".csv",  sep = "_") 

write.csv(mydataY0, output_name, row.names = F)

```
