Comparing multiple irregular non-aligned sequences

title: “distantia: an R package to compute the dissimilarity between multivariate time-series” output: github_document: toc: true toc_depth: 2 pandoc_args: –webtex always_allow_html: yes —

DOI

Summary

The package distantia allows to measure the dissimilarity between multivariate time-series (sequences hereafter). The package assumes that the target sequences are ordered along a given dimension, being depth and time the most common ones, but others such as latitude or elevation are also suitable. Furthermore, the target sequences can be regular or irregular, and have their samples aligned (same age/time/depth) or unaligned (different age/time/depth). The only requirement is that the sequences must have at least two (but ideally more) columns with the same name and units.

In this document I briefly explain the logics behind the method, show how to use it, and demonstrate how the distantia package introduces useful tools to compare multivariate time-series. The topics covered in this document are:

  • Installation of the package.
  • Comparing two irregular non-aligned sequences.
  • Comparing multiple irregular non-aligned sequences.
  • Comparing regular aligned sequences.
  • Restricted permutation test to assess the significance of dissimilarity values.
  • Assessing the contribution of a variable to the dissimilarity between two sequences.
  • Finding the section in a long sequence more similar to a given short sequence (irregular sequences).
  • Finding the section in a long sequence more similar to a given short sequence (regular sequences).
  • Sequence slotting: combining samples of two sequences into a single composite sequence.
  • Tranferring an attribute from one sequence to another: direct and interpolated modes.

Installation

You can install the released version of distantia from CRAN with:

install.packages("distantia")

And the development version from GitHub with:

install.packages("devtools")
library(devtools)
devtools::install_github("BlasBenito/distantia")

Loading the library, plus other helper libraries:

library(distantia)
library(ggplot2)
library(viridis)
library(kableExtra)
library(qgraph)
library(tidyr)

Comparing two irregular non-aligned sequences

This section assumes that the user wants to compare two sequences. The package provides two example datasets based on the Abernethy pollen core (Birks and Mathewes, 1978):

Sequence A
betula pinus corylu junipe empetr gramin cypera artemi rumex
79 271 36 0 4 7 25 0 0
113 320 42 0 4 3 11 0 0
51 420 39 0 2 1 12 0 0
130 470 6 0 0 2 4 0 0
31 450 6 0 3 2 3 0 0
59 425 12 0 0 2 3 0 0
78 386 29 2 0 0 2 0 0
71 397 52 2 0 6 3 0 0
140 310 50 2 0 4 3 0 0
150 323 34 2 0 11 2 0 0
175 317 37 2 0 11 3 0 0
181 345 28 3 0 7 3 0 0
153 285 36 2 0 8 3 0 1
214 315 54 2 1 13 5 0 0
200 210 41 6 0 10 4 0 0 
kable(sequenceB[1:15, ], caption = "Sequence B")
Sequence B
betula pinus corylu junipe gramin cypera artemi rumex
19 175 NA 2 34 39 1 0
18 119 28 1 36 44 0 4
30 99 37 0 2 20 0 1
26 101 29 0 0 18 0 0
31 99 30 0 1 10 0 0
24 97 28 0 2 9 0 0
23 105 34 0 1 6 0 0
48 112 46 0 0 12 0 0
29 108 16 0 6 3 0 0
23 110 21 0 2 11 0 1
5 119 19 0 1 1 0 0
30 105 NA 0 9 7 0 0
22 116 17 0 1 7 0 0
24 115 20 0 2 4 0 0
26 119 23 0 4 0 0 0

Notice that sequenceB has a few NA values (that were introduced to serve as an example). The function prepareSequences gets them ready for analysis by matching colum names and handling empty data. It allows to merge two (or more) multivariate time-series into a single table ready for further analyses. Note that, since the data represents pollen abundances, a Hellinger transformation (square root of the relative proportions of each taxa) is applied.

id betula pinus corylu junipe empetr gramin cypera artemi rumex
A 0.4327 0.8014 0.2921 0.0002 0.0974 0.1288 0.2434 2e-04 0.0002
A 0.4788 0.8057 0.2919 0.0001 0.0901 0.0780 0.1494 1e-04 0.0001
A 0.3117 0.8944 0.2726 0.0001 0.0617 0.0436 0.1512 1e-04 0.0001
A 0.4609 0.8763 0.0990 0.0001 0.0001 0.0572 0.0808 1e-04 0.0001
A 0.2503 0.9535 0.1101 0.0001 0.0778 0.0636 0.0778 1e-04 0.0001
A 0.3432 0.9210 0.1548 0.0001 0.0001 0.0632 0.0774 1e-04 0.0001
A 0.3962 0.8813 0.2416 0.0634 0.0001 0.0001 0.0634 1e-04 0.0001
A 0.3657 0.8647 0.3129 0.0614 0.0001 0.1063 0.0752 1e-04 0.0001
A 0.5245 0.7804 0.3134 0.0627 0.0001 0.0886 0.0768 1e-04 0.0001
A 0.5361 0.7866 0.2552 0.0619 0.0001 0.1452 0.0619 1e-04 0.0001
A 0.5667 0.7627 0.2606 0.0606 0.0001 0.1421 0.0742 1e-04 0.0001
A 0.5650 0.7800 0.2222 0.0727 0.0001 0.1111 0.0727 1e-04 0.0001
A 0.5599 0.7642 0.2716 0.0640 0.0001 0.1280 0.0784 1e-04 0.0453
A 0.5952 0.7222 0.2990 0.0575 0.0407 0.1467 0.0910 1e-04 0.0001
A 0.6516 0.6677 0.2950 0.1129 0.0001 0.1457 0.0922 1e-04 0.0001

The computation of dissimlarity between two sequences requires several steps.

1. Computation of a distance matrix among the samples of both sequences. It is computed by the distanceMatrix function, which allows the user to select a distance metric (so far the ones implemented are manhattan, euclidean, chi, and hellinger). The function plotMatrix allows an easy visualization of the distance matrix.

Distance matrix of two irregular and unaligned multivariate sequences. Warmer colors indicate higher distance.

2. Computation of the least-cost path within the distance matrix. This step uses a dynamic programming algorithm to find the least-cost when moving between the cell 1,1 of the matrix (lower left in the image above) and the last cell of the matrix (opposite corner). It does so by solving first every partial solution (in the neighborhood of every cell in the matrix), and then propagating the sum towards the opposite extreme of the matrix.

The value of the upper-right cell in the plotted matrix (actually, the lower-right cell in the actual data matrix, the matrix is rotated in the plot) is the sum of the minimum distance across all samples of both time-series.

Least cumulative cost of moving between the lower-left cell and the upper-right cell. The value of the upper-right cell corresponds with the minimized sum of distances between the samples of both sequences.

Optional In this optional step, the function leastCostPath finds the least-cost path in the matrix above. It returns a dataframe with the coordinates of the cells within the path, the distance between consecutive cells, and the cumulative distance of any given step in the path. It can be used to plot the path, which helps to better understand the alignmnet between both sequences.

This dataframe will be used later on to generate other products, such the slotting of both sequences (a unique sequences with the samples of both sequences in the order that minimizes the distance between consecutive samples), or to transfer attributes such as age/time/depth from one sequence with them to another without them. Note: these applications are not yet implemented.

A B distance cumulative.distance
49 41 2.0278 77.5398
49 40 1.7083 74.4425
48 40 1.0461 72.7342
48 39 0.9972 71.6881
47 39 1.0389 70.6909
46 39 0.9777 69.6520
45 39 1.3507 68.6743
44 39 1.3165 67.3236
43 39 1.4085 66.0071
42 39 1.3132 64.5986
42 38 1.5524 63.2853
42 37 1.2139 61.7329
42 36 1.6284 60.5191
42 35 1.7988 58.8907
41 35 1.2870 57.0919

least-cost path plotted on top of the distance matrix.

Least-cost path plotted on top of the least-cost matrix.

Optional: least cost path using diagonals.

The least cost path can be computed also in diagonal, which allows to pair together these samples with the higher similarity that contribute to minimize the overall cost of the least cost path.

Least cumulative cost of moving between the lower-left cell and the upper-right cell. The value of the upper-right cell corresponds with the minimized sum of distances between the samples of both sequences.

Using diagonals is useful when the goal is to transfer attributes (time, age, depth, etc) from one sequence with them to another sequence without them. This will be shown later NOTE: not implemented yet.

3. Getting the least-cost value. The Upper right cell of the least-cost matrix in the image of the least-cost matrix (lower right cell in the actual data-matrix) holds the cumulative sum of the least-cost path. This value will be used later to compute dissimilarity.

4. Autosum, or sum of the distances among adjacent samples on each sequence. This is another requirement to compute a normalized measure of dissimilarity. It just computes the distance between adjacent samples of each sequence and sums them up. Notice that the output is a list with named slots. Lists are the main output device of this package, I’ll explain later why.

5. Compute dissimilarity. The dissimilarity measure used in this package is named psi, that was first described in the book “Numerical methods in Quaternary pollen analysis” (Birks and Gordon, 1985). Psi is computed as follows:

$$\psi = \frac{LC - (\sum{A_{i-j}} + \sum{B_{i-j}})}{\sum{A_{i-j}} + \sum{B_{i-j}}} $$

where:

  • $LC$ is the least-cost computed by leastCostMatrix.
  • $\sum{A_{i-j}}$ is the autosum of one of the sequences.
  • $\sum{B_{i-j}}$ is the autosum of the other sequence.

Which basically is the least-cost normalizado by the autosum of both sequences. The psi function only requires the least cost, and the autosum of both sequences, as follows. Notice that in the output list, the slot with the psi value is named after the two sequences separated by a vertical line (“A|B”). This convention will be followed by any function in the package that returns objects resulting from comparing two sequences.

The output of psi is a list, that can be transformed to a dataframe or a matrix by using the formatPsi function.

A B psi
A B 2.3093
A B
A 0.0000 2.3093
B 2.3093 0.0000

Or can also be transformed from matrix to dataframe, or from dataframe to matrix, as convenient.

A B psi
A B 2.3093
A B
A 0.0000 2.3093
B 2.3093 0.0000

All the steps required to compute psi, including the format options provided by formatPsi are wrapped together in the function workflowPsi, that works as follows:

#cleaning environment for next example
rm(AB.autosum, AB.distance.matrix, AB.least.cost, AB.least.cost.matrix, AB.least.cost.path, AB.psi, AB.psi.dataframe, AB.psi.matrix, AB.sequences, sequenceA, sequenceB, diagonal.least.cost, diagonal.least.cost.path)

The package can work seamlessly with any given number of sequences, as long as there is memory enough available. To do so, almost every function uses the package “foreach”, that allows to parallelize the execution of the functions by using all the processors in your machine but one. This speeds up operations considerably.

The example dataset sequencesMIS contains 12 sections of the same sequence belonging to different marine isotopic stages identified by a column named “MIS”. MIS stages with odd numbers are generally interpreted as warm periods (interglacials), while the odd ones are interpreted as cold periods (glacials). In any case, this interpretation is not important to illustrate this capability of the library.

data(sequencesMIS)
kable(head(sequencesMIS, n=15), digits = 4)
MIS Quercus Betula Pinus Alnus Tilia Carpinus
MIS-1 55 1 5 3 4 5
MIS-1 86 21 35 8 0 10
MIS-1 120 15 8 1 0 1
MIS-1 138 16 12 6 1 3
MIS-1 130 12 17 2 1 1
MIS-1 128 0 6 4 2 2
MIS-1 140 0 19 9 4 0
MIS-1 113 0 15 12 2 5
MIS-1 98 0 27 2 2 0
MIS-1 92 1 16 7 3 0
MIS-1 73 3 22 3 0 0
MIS-1 91 1 21 3 7 0
MIS-1 148 1 22 1 4 0
MIS-1 148 0 1 7 13 0
MIS-1 149 1 2 5 4 0

The dataset is checked and prepared with prepareSequences (note the change in the argument grouping.column).

The dissimilarity measure psi can be computed for every combination of sequences through the function workflowPsi shown below. Note the argument output that allows to select either “dataframe” (ordered from lower to higher dissimilarity) or “matrix” to obtain an output with a given structure (if empty, the function returns a list).

A B psi
65 MIS-10 MIS-12 0.4656
24 MIS-3 MIS-6 0.5412
13 MIS-2 MIS-4 0.5560
62 MIS-9 MIS-11 0.5639
30 MIS-3 MIS-12 0.5686
61 MIS-9 MIS-10 0.5772
59 MIS-8 MIS-11 0.5841
12 MIS-2 MIS-3 0.5843
66 MIS-11 MIS-12 0.5949
60 MIS-8 MIS-12 0.6004
63 MIS-9 MIS-12 0.6266
51 MIS-6 MIS-12 0.6487
22 MIS-3 MIS-4 0.6830
53 MIS-7 MIS-9 0.6877
32 MIS-4 MIS-6 0.6964
28 MIS-3 MIS-10 0.7231
45 MIS-5 MIS-12 0.7274
64 MIS-10 MIS-11 0.7361
56 MIS-7 MIS-12 0.7378
21 MIS-2 MIS-12 0.7444
26 MIS-3 MIS-8 0.7621
47 MIS-6 MIS-8 0.7630
57 MIS-8 MIS-9 0.7648
19 MIS-2 MIS-10 0.7737
27 MIS-3 MIS-9 0.7884
40 MIS-5 MIS-7 0.7900
15 MIS-2 MIS-6 0.7912
54 MIS-7 MIS-10 0.8147
29 MIS-3 MIS-11 0.8182
49 MIS-6 MIS-10 0.8244
58 MIS-8 MIS-10 0.8452
43 MIS-5 MIS-10 0.8882
42 MIS-5 MIS-9 0.9065
48 MIS-6 MIS-9 0.9500
44 MIS-5 MIS-11 0.9556
55 MIS-7 MIS-11 0.9618
38 MIS-4 MIS-12 0.9874
52 MIS-7 MIS-8 1.0042
25 MIS-3 MIS-7 1.0658
23 MIS-3 MIS-5 1.0807
34 MIS-4 MIS-8 1.1241
50 MIS-6 MIS-11 1.1681
46 MIS-6 MIS-7 1.2160
17 MIS-2 MIS-8 1.2658
41 MIS-5 MIS-8 1.2964
18 MIS-2 MIS-9 1.4517
36 MIS-4 MIS-10 1.4584
39 MIS-5 MIS-6 1.4667
20 MIS-2 MIS-11 1.4832
6 MIS-1 MIS-7 1.7293
35 MIS-4 MIS-9 1.8136
10 MIS-1 MIS-11 1.8177
16 MIS-2 MIS-7 1.8869
8 MIS-1 MIS-9 1.8992
33 MIS-4 MIS-7 1.9877
37 MIS-4 MIS-11 1.9949
7 MIS-1 MIS-8 2.0264
5 MIS-1 MIS-6 2.0409
4 MIS-1 MIS-5 2.1046
14 MIS-2 MIS-5 2.1585
11 MIS-1 MIS-12 2.1738
2 MIS-1 MIS-3 2.4295
9 MIS-1 MIS-10 2.7392
31 MIS-4 MIS-5 2.7452
3 MIS-1 MIS-4 2.7457
1 MIS-1 MIS-2 2.7670

A dataframe like this can be plotted as an adjacency network with the qgraph package as follows:

Similarity between MIS sequences represented as a network. More similar sites are closer, and linked by a wider edge. Note that glacials are colored in blue and interglacials in green

Or as a matrix with ggplot2.

#ordering factors to get a triangular matrix
MIS.psi$A <- factor(MIS.psi$A, levels=unique(sequencesMIS$MIS))
MIS.psi$B <- factor(MIS.psi$B, levels=unique(sequencesMIS$MIS))

#plotting matrix
ggplot(data=na.omit(MIS.psi), aes(x=A, y=B, size=psi, color=psi)) + 
  geom_point() +
  viridis::scale_color_viridis(direction = -1) +
  guides(size = FALSE)

Dissimilarity between MIS sequences. Darker colors indicate a higher dissimilarity.

The dataframe of dissimilarities between pairs of sequences can be also used to analyze the drivers of dissimilarity. To do so, attributes such as differences in time (when sequences represent different times) or distance (when sequences represent different sites) between sequences, or differences between physical/climatic attributes between sequences such as topography or climate can be added to the table, so models such as $psi = A + B + C$ (were A, B, and C are these attributes) can be fitted.

#cleaning environment for next example
rm(MIS.distance, MIS.psi, MIS.psi.matrix, MIS.sequences, sequencesMIS)

Comparing regular aligned sequences

This section assumes that the target multivariate time-series have the same number of samples/rows, and samples have the same time/depth/order if there is a time/age/depth column available. In this particular case, distances are computed only between samples with the same time/depth/order, and no distance matrix (nor least cost analysis) is required. When the argument paired.samples in prepareSequences is set to TRUE, the function checks if the sequences have the same number of rows, and, if time.column is provided, it selects the samples that have valid time/depth columns for every sequence in the dataset.

Here we test these ideas with the climate dataset included in the library. It represents simulated palaeoclimate over 200 ky. at four sites identified by the column sequenceId. Note that this time the transformation applied is “scaled”, which uses the scale function of R base to center and scale the data.

The function distancePairedSamples computes the between paired samples (same order/time/age/depth), and therefore does not use the least cost algorithm to minimize the distances between sequences shown above. If the argument time.column is not provided, or if it’s provided but same.time is false, the samples are paired by their order (first sample of one sequence vs. the first sample of the other sequence, the second against the second, and so on), and samples that only have an age for one of the sequences are removed from the comparison. This is useful to compare subsets of the same sequence, or subsets of different sequences belonging to different times.

If there is a time.column and same.time is set to true, only samples with the same time/age/depth are compared. This is useful to compare sequences taken at the same time in different sites.

The result is a list, each slot named as the id of the given sequence according to grouping.columns, and containing a vector with pairwise distances. Each vector position is named after time.column if available.

If the argument sum.distances is set to TRUE, it sums them up, doing in fact the same work done by distanceMatrix, leastCostMatrix and leastCost at once.

Computing psi from there only requires the autosum of each sequence, and the application of the psi function.

The function workflowPsi executes this sequences when the argument paired.samples is set to TRUE, as follows.

A B psi
1 2 3.3788
1 3 2.7679
1 4 4.0435
2 3 3.7611
2 4 2.5517
3 4 3.1698 
#cleaning workspace for next example
rm(climate, climate.autosum, climate.distances, climate.psi)

Restricted permutation test to assess the significance of dissimilarity values

One question that may arise when comparing time series is “to what extent are dissimilarity values random?”. Answering this question requires to compare a given dissimilarity value (psi) with a distribution of dissimilarity values resulting from chance. However… how do we simulate chance in a multivariate time-series? The natural answer is “permutation”. Since samples in a multivariate time-series are ordered, re-shuffling samples is out of the question, but it is possible to re-shuffle the values of the variables within each sample. This kind of permutation is named “restricted permutation”.

A restricted permutation test on psi values requires the following steps:

  • Compute the real psi on two given sequences A and B.
  • Repeat the following steps several times (99 to 999):
    • For each case of each column of A and B, randomly apply one of these actions:
      • Leave it as is.
      • Replace it with the previous case.
      • Replace it with the next case.
    • Compute randomized psi between A and B and store the value.
  • Add real psi to the pool of randomized psi.
  • Compute the proportion of randomized psi that is equal or lower than real psi.

Such a proportion represents the probability of obtaining a value lower than real psi by chance.

Since the restricted permutation only happens at a local scale within each column of each sequence, the probability values returned are very conservative and shouldn’t be interpreted in the same way p-values are interpreted.

The process described above has been implemented in the workflowNullPsi function. We will apply it to three groups of the sequencesMIS dataset.

The computation of the null psi values goes as follows:

Note that the number of repetitions has been set to 9 in order to speed-up execution. The actual number should be 99, and ideally, 999.

The output is a list with two dataframes, psi and p.

The dataframe psi contains the real and random psi values. The column psi contains the dissimilarity between the sequences in the columns A and B. The columns r1 to r9 contain the psi values obtained from permutations of the sequences.

kable(random.psi$psi, digits = 4)
A B psi r1 r2 r3 r4 r5 r6 r7 r8 r9
MIS-4 MIS-5 2.7452 3.5801 1.0108 2.0398 3.5858 1.2728 2.0929 3.9632 1.0159 2.1560
MIS-4 MIS-6 0.6964 3.4371 1.0574 1.9265 3.6948 1.1597 1.9834 3.4991 1.0731 1.9729
MIS-5 MIS-6 1.4667 3.5061 1.1663 2.1745 3.8673 1.1517 2.4507 3.7685 1.0278 2.0605

The information in this dataframe can be easily plotted as follows:

#extracting the dataframe
psi <- random.psi[[1]]

#colors
cols <- viridis::viridis(2, begin = 0.2, end = 0.8)

#multipanel plot
par(mfrow=c(1,3))
for(i in 1:nrow(psi)){
  plot(density(x = t(psi[i, 3:ncol(psi)]), from = 0), 
       main = paste(psi[i, 1], " vs. ",  psi[i, 2], sep=""),
       xlab = "Psi",
       col = cols[1],
       lwd = 3
       )
  abline(v = psi[i, 3], 
         col = cols[2], 
         lwd=3
         )
}

Real psi values versus psi values computed from randomized sequences. Real psi values are represented by the green vertical line.

According to this data, the psi value obtained for MIS-4 and MIS-6 is more robust (less likely obtained by chance) the the other two. The dataframe p contains the probability of obtaining the real psi value by chance for each combination of sequences.

kable(random.psi$p)
A B p
MIS-4 MIS-5 0.7
MIS-4 MIS-6 0.1
MIS-5 MIS-6 0.4

Assessing the contribution of a variable to the dissimilarity between two sequences

What variables are more important in explaining the dissimilarity between two sequences?, or in other words, what variables contribute the most to the dissimilarity between two sequences? One reasonable answer is: the one that reduces dissimilarity the most when removed from the data.

This section explains how to use the function workflowImportance to evaluate the importance of given variables in explaining differences between sequences.

First, we prepare the data. It is again sequencesMIS, but with only three groups selected (MIS 4 to 6) to simplify the analysis.

The workflow function is pretty similar to the ones explained above. It allows to work with paired samples (aligned time-series) or with unaligned ones by switching between workflowPsi and workflowPsiPairedSamples as requested by the user through the argument paired.samples. Unlike the other functions in the package, that parallelize the execution of combinations of sequences, this one parallelizes the computation of psi on combinations of columns, removing one column each time.

WARNING: the argument ‘exclude.columns’ of ‘workflowImportance’ does not work in version 1.0.0 (available in CRAN), but the bug is fixed in version 1.0.1 (available in GitHub). If you are using 1.0.0, I recommend you to subset ‘sequences’ so only the grouping column and the numeric columns to be compared are available for the function.

The output is a list with two slots named psi and psi.drop.

The dataframe psi contains psi values for each combination of variables (named in the coluns A and B) computed for all columns in the column All variables, and one column per variable named Without variable_name containing the psi value when that variable is removed from the compared sequences.

kable(psi.importance$psi, digits = 4)
A B All variables Without Carpinus Without Tilia Without Alnus Without Pinus Without Betula Without Quercus
MIS-4 MIS-5 2.2093 2.2259 2.2257 2.2786 2.8885 2.2505 0.9186
MIS-4 MIS-6 0.7145 0.7070 0.7139 0.6955 0.7377 0.7045 0.6033
MIS-5 MIS-6 1.4622 1.4546 1.4714 1.4858 1.7097 1.5020 0.7573

This table can be plotted as a bar plot as follows:

#extracting object
psi.df <- psi.importance$psi

#to long format
psi.df.long <- tidyr::gather(psi.df, variable, psi, 3:ncol(psi.df))

#creating column with names of the sequences
psi.df.long$name <- paste(psi.df.long$A, psi.df.long$B, sep=" - ")

#plot
ggplot(data=psi.df.long, aes(x=variable, y=psi, fill=psi)) + 
  geom_bar(stat = "identity") + 
  coord_flip() + 
  facet_wrap("name") +
  scale_fill_viridis() +
  ggtitle("Contribution of separated variables to dissimilarity.") +
  labs(fill = "Psi")

Variable importance analysis of three combinations of sequences. The plot suggest that MIS-4 and MIS-6 are more similar (both are glacial periods), and that the column Quercus is the one with a higher contribution to dissimilarity between sequences.

The second table, named psi.drop contains the drop in psi values, in percentage, when the given variable is removed from the analysis. Large positive numbers indicate that dissimilarity drops (increase in similarity) when the given variable is removed, confirming that the variable is important to explain the dissimilarity between both sequences. Negative values indicate an increase in dissimilarity between the sequences when the variable is dropped.

In summary:

  • High psi-drop value: variable contributes to dissimilarity.
  • Low or negative psi-drop value: variable contributes to similarity.
kable(psi.importance$psi.drop)
A B Carpinus Tilia Alnus Pinus Betula Quercus
MIS-4 MIS-5 -0.75 -0.74 -3.13 -30.74 -1.86 58.42
MIS-4 MIS-6 1.04 0.07 2.66 -3.25 1.40 15.56
MIS-5 MIS-6 0.52 -0.63 -1.61 -16.93 -2.72 48.21 
#extracting object
psi.drop.df <- psi.importance$psi.drop

#to long format
psi.drop.df.long <- tidyr::gather(psi.drop.df, variable, psi, 3:ncol(psi.drop.df))

#creating column with names of the sequences
psi.drop.df.long$name <- paste(psi.drop.df.long$A, psi.drop.df.long$B, sep=" - ")

#plot
ggplot(data=psi.drop.df.long, aes(x=variable, y=psi, fill=psi)) + 
  geom_bar(stat = "identity") + 
  coord_flip() + 
  facet_wrap("name") +
  scale_fill_viridis(direction = -1) +
  ggtitle("Drop in dissimilarity when variables are removed.") +
  ylab("Drop in dissimilarity (%)") +
  labs(fill = "Psi drop (%)")

Drop in psi values, represented as percentage, when a variable is removed from the analysis. Negative values indicate a contribution to similarity, while positive values indicate a contribution to dissimilarity. The plot suggest that Quercus is the variable with a higher contribution to dissimilarity, while Pinus has the higher contribution to similarity.

#cleaning environment for next example
rm(psi.df, psi.df.long, psi.drop.df, psi.drop.df.long, psi.importance, sequences, sequencesMIS)

Finding the section in a long sequence more similar to a given short sequence (irregular time-series)

In this scenario the user has one short and one long sequence, and the goal is to find the section in the long sequence that better matches the short one. To recreate this scenario we will use the dataset sequencesMIS. We will extract the first 10 samples as short sequence, and the first 40 samples as long sequence. These small subsets are selected to speed-up the execution time of this example.

A more realistic example would be to select, for example, the group “MIS-5” as short sequence, and the complete sequence as long sequence, in order to find what segments in sequencesMIS is more similar to the “MIS-5” group.

The sequences have to be prepared and transformed. For simplicity, the sequences are named short and long, and the grouping column is named id, but the user can name them at will. Since the data represents community composition, a Hellinger transformation is applied.

id Quercus Betula Pinus Alnus Tilia Carpinus
short 0.8680 0.1170 0.2617 0.2027 0.2341 0.2617
short 0.7331 0.3623 0.4677 0.2236 0.0002 0.2500
short 0.9097 0.3216 0.2349 0.0830 0.0003 0.0830
short 0.8855 0.3015 0.2611 0.1846 0.0754 0.1306
short 0.8931 0.2713 0.3229 0.1108 0.0783 0.0783
short 0.9494 0.0003 0.2056 0.1678 0.1187 0.1187
short 0.9022 0.0002 0.3324 0.2287 0.1525 0.0002
short 0.8768 0.0003 0.3194 0.2857 0.1166 0.1844
short 0.8716 0.0003 0.4575 0.1245 0.1245 0.0003
short 0.8793 0.0917 0.3667 0.2425 0.1588 0.0003
long 0.8680 0.1170 0.2617 0.2027 0.2341 0.2617
long 0.7331 0.3623 0.4677 0.2236 0.0002 0.2500
long 0.9097 0.3216 0.2349 0.0830 0.0003 0.0830
long 0.8855 0.3015 0.2611 0.1846 0.0754 0.1306
long 0.8931 0.2713 0.3229 0.1108 0.0783 0.0783
long 0.9494 0.0003 0.2056 0.1678 0.1187 0.1187
long 0.9022 0.0002 0.3324 0.2287 0.1525 0.0002
long 0.8768 0.0003 0.3194 0.2857 0.1166 0.1844
long 0.8716 0.0003 0.4575 0.1245 0.1245 0.0003
long 0.8793 0.0917 0.3667 0.2425 0.1588 0.0003
long 0.8502 0.1723 0.4667 0.1723 0.0003 0.0003
long 0.8601 0.0902 0.4132 0.1562 0.2386 0.0003
long 0.9170 0.0754 0.3536 0.0754 0.1508 0.0002
long 0.9358 0.0002 0.0769 0.2035 0.2774 0.0002
long 0.9620 0.0788 0.1115 0.1762 0.1576 0.0002
long 0.8692 0.0005 0.4714 0.1491 0.0005 0.0005
long 0.9315 0.0003 0.2227 0.0909 0.2727 0.0003
long 0.8919 0.1508 0.3482 0.1741 0.1741 0.0003
long 0.8722 0.2265 0.2615 0.2615 0.2265 0.0003
long 0.8882 0.3496 0.2108 0.0003 0.2108 0.0003
long 0.7977 0.0010 0.4264 0.4264 0.0010 0.0010
long 0.9295 0.1549 0.1549 0.2828 0.0894 0.0003
long 0.9225 0.1786 0.1031 0.3262 0.0003 0.0003
long 0.9010 0.1407 0.2814 0.1990 0.2225 0.0003
long 0.8898 0.3819 0.2500 0.0005 0.0005 0.0005
long 0.8935 0.2593 0.3550 0.0003 0.0917 0.0003
long 0.8452 0.3780 0.3780 0.0008 0.0008 0.0008
long 0.6070 0.3244 0.6489 0.0007 0.3244 0.0007
long 0.0010 0.5222 0.8528 0.0010 0.0010 0.0010
long 0.0011 0.4714 0.8819 0.0011 0.0011 0.0011
long 0.5000 0.6124 0.6124 0.0011 0.0011 0.0011
long 0.0010 0.4472 0.8944 0.0010 0.0010 0.0010
long 0.0006 0.3922 0.8987 0.1961 0.0006 0.0006
long 0.0007 0.3086 0.9512 0.0007 0.0007 0.0007
long 0.2041 0.2887 0.9129 0.2041 0.0005 0.0005
long 0.0007 0.3162 0.9220 0.2236 0.0007 0.0007
long 0.2611 0.3371 0.8919 0.1508 0.0005 0.0005
long 0.0006 0.3536 0.9186 0.1768 0.0006 0.0006
long 0.1104 0.2469 0.9627 0.0003 0.0003 0.0003
long 0.0004 0.1796 0.9837 0.0004 0.0004 0.0004

The function workflowPartialMatch shown below is going to subset the long sequence in sizes between min.length and max.length. In the example below this search space is reduced to the minimum (the rows of MIS.short plus and minus one) to speed-up the execution of this example. If left empty, the length of the segment in the long sequence to be matched will have the same number of samples as the short sequence. In the example below we look for segments of the same length, two samples shorter, and two samples longer than the shorter sequence.

The function returns a dataframe with three columns: first.row (first row of the matched segment of the long sequence), last.row (last row of the matched segment of the long sequence), and psi (ordered from lower to higher). In this case, since the long sequence contains the short sequence, the first row shows a perfect match.

kable(MIS.psi[1:10, ], digits = 4)
first.row last.row psi
1 10 0.0000
1 9 0.0419
1 11 0.0490
1 12 0.0540
1 8 0.1168
2 10 0.3338
4 12 0.3408
4 11 0.3507
4 13 0.3514
2 12 0.3603

Subsetting the long sequence to obtain the segment best matching with the short sequence goes as follows.

#cleaning workspace
rm(best.match, MIS.long, MIS.psi, MIS.short, MIS.short.long, sequencesMIS, best.match.indices)

Finding the section in a long sequence more similar to a given short sequence (regular time-series, paired samples)

When regular time-series are provided, and the analysis needs to be done comparing only aligned samples, the argument paired.samples can be set to TRUE, and instead of the least-cost algorithm to find the minimized distance between sequences, the function distancePairedSamples (shown above) is used to compute the distance between sequences of the same length (short sequence versus subsets of the long sequence). In such a case, the arguments min.length and max.length are irrelevant, and the sample-size of the matching segments in the long sequence have equals the sample-size of the short sequence. This application is ideal when the user has regular time-series.

The example below shows how it is done with the climateShort and climateLong datasets. The goal is to find the subset in climateLong, with lengths between min.length and max.length, better matching climateShort, with the same number of rows (11) as climateShort.

first.row last.row psi
17 27 0.0000
17 28 0.0083
17 26 0.0367
16 27 0.0880
16 26 0.1268
16 25 0.1756
18 27 0.3294
18 28 0.3355
18 29 0.3747
15 26 0.3906

Since climateShort is a subset of climateLong, there is a perfect match from the rows 17 to 27.

The output table can be used to easily plot similarity over the indices of the long dataset.

Pattern of dissimilarity between climateShort and climateLong across samples of the latter.

#cleaning workspace
rm(climateLong, climateShort, climate.short.long, climate.psi)

Sequence slotting: combining samples of two sequences into a single composite sequence

Under this scenario, the objective is to combine two sequences in order to obtain a single composite sequence. The basic assumption followed by the algorithm building the composite sequence is most similar samples should go together, but respecting the original ordering of the sequences. Therefore, the output will contain the samples in both sequences ordered in a way that minimizes the multivariate distance between consecutive samples. This scenario assumes that at least one of the sequences do not have a time/age/depth column, or that the values in such a column are uncertain. In any case, time/age/depth is not considered as a factor in the generation of the composite sequence.

The example below uses the pollenGP dataset, which contains 200 samples, with 40 pollen types each. To create a smalle case study, the code below separates the first 20 samples of the sequence into two different sequences with 10 randomly selected samples each. Even though this scenario assumes that these sequences do not have depth or age, these columns will be kept so the result can be assessed. That is why these columns are added to the exclude.columns argument. Also, note that the argument transformation is set to “none”, so the output is not transformed, and the outcome can be easily interpreted. This will give more weight to the most abundant taxa, which will in fact guide the slotting.

id depth age Abies Juniperus Hedera Plantago Boraginaceae Crassulaceae Pinus Ranunculaceae Rhamnus Caryophyllaceae Dipsacaceae Betula Acer Armeria Tilia Hippophae Salix Thalictrum Labiatae Valeriana Nymphaea Umbelliferae Sanguisorba_minor Plantago.lanceolata Campanulaceae Asteroideae Gentiana Fraxinus Cichorioideae Taxus Rumex Cedrus Ranunculus.subgen..Batrachium Cyperaceae Corylus Myriophyllum Filipendula Vitis Rubiaceae Polypodium
A 3 3.97 11243 0 5 12 21 95 0 20 0 0 6 1 0 7 0 0 1 0 0 2 0 11 1 0 0 0 0 0 0 0 0 0 0 0 0 4 4 0 0 0
A 6 4.05 11459 0 20 73 94 1 0 0 0 0 1 10 0 2 0 0 3 0 1 3 0 0 3 0 0 0 0 3 0 0 2 0 0 0 0 0 0 1 0 0
A 7 4.07 11514 0 20 80 100 0 0 0 0 0 10 4 0 0 1 0 0 0 0 3 0 0 1 0 0 0 0 8 1 0 0 0 0 0 1 2 1 0 1 0
A 8 4.10 11595 0 34 80 155 0 0 0 2 0 2 13 0 0 0 0 1 0 0 2 0 0 2 0 0 0 0 6 0 0 0 0 0 0 1 1 0 0 1 0
A 9 4.17 11784 0 22 44 131 0 0 0 0 0 1 13 0 0 0 0 5 0 0 5 0 0 3 0 0 0 0 6 0 0 0 0 2 0 1 0 0 2 0 0
A 10 4.20 11865 0 35 30 112 0 0 0 0 0 4 8 0 0 0 2 2 0 1 2 0 0 7 0 1 0 0 10 1 0 0 0 1 0 1 1 0 2 1 0
A 11 4.22 11919 0 30 45 150 0 0 0 0 0 4 11 0 0 0 0 2 0 3 1 0 0 1 0 0 0 0 13 1 0 0 0 0 0 0 0 0 2 1 0
A 12 4.25 12000 0 44 35 150 0 0 0 0 0 2 8 0 0 0 0 3 0 1 0 0 0 5 0 0 0 0 6 3 0 0 0 0 0 0 2 0 1 0 0
A 15 4.32 12189 0 43 17 120 0 0 0 0 0 2 15 0 0 1 1 2 0 0 2 0 0 5 1 0 0 0 6 2 0 2 0 1 0 2 1 0 0 0 0
A 16 4.35 12270 0 36 25 91 0 0 2 0 0 6 17 0 0 1 0 5 0 4 3 0 1 6 0 0 1 0 1 2 0 0 1 0 0 0 1 0 1 1 1
B 1 3.92 11108 0 7 0 5 20 0 13 0 2 1 0 2 41 0 0 0 0 0 0 0 0 1 0 8 0 0 0 0 0 0 0 0 0 0 0 60 2 0 0
B 2 3.95 11189 0 3 3 15 47 0 15 0 3 1 0 3 28 0 0 0 0 0 0 0 3 0 0 2 0 4 1 0 0 0 0 1 0 0 0 87 0 0 0
B 4 4.00 11324 0 8 43 60 65 0 10 0 0 2 4 0 0 0 0 0 0 0 2 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 1 11 0 2 0
B 5 4.02 11378 0 44 76 110 0 0 0 0 0 2 11 0 0 0 0 3 0 1 3 0 0 5 0 0 1 0 6 0 0 1 0 1 0 0 0 0 1 1 1
B 13 4.27 12054 0 37 30 150 0 0 1 0 0 6 10 0 0 0 0 7 0 2 2 0 0 6 0 0 0 0 7 1 0 0 0 0 0 0 3 0 4 0 0 
rm(sampling.indices)

Once the sequences are prepared, the function workflowSlotting will allow to combine (slot) them. The function computes a distance matrix between the samples in both sequences according to the method argument, computes the least cost matrix, and generates the least cost path.

Distance matrix and least cost path of the example sequences 'A' and 'B'..

The function reads the least cost path in order to find the combination of samples of both sequences that minimizes dissimilarity, constrained by the order of the samples on each sequence. The output dataframe has a column named original.index, which has the index of each sample in the original datasets.

kable(AB.combined[1:15,1:10], format = "html", digits = 4) %>% 
  row_spec(c(4, 6, 11, 12, 13), bold = T)
id original.index depth age Abies Juniperus Hedera Plantago Boraginaceae Crassulaceae
11 B 1 1 3.92 11108 0 7 0 5 20
12 B 2 2 3.95 11189 0 3 3 15 47
13 B 3 4 4.00 11324 0 8 43 60 65
1 A 1 3 3.97 11243 0 5 12 21 95
2 A 2 6 4.05 11459 0 20 73 94 1
14 B 4 5 4.02 11378 0 44 76 110 0
3 A 3 7 4.07 11514 0 20 80 100 0
4 A 4 8 4.10 11595 0 34 80 155 0
5 A 5 9 4.17 11784 0 22 44 131 0
15 B 5 13 4.27 12054 0 37 30 150 0
6 A 6 10 4.20 11865 0 35 30 112 0
7 A 7 11 4.22 11919 0 30 45 150 0
8 A 8 12 4.25 12000 0 44 35 150 0
16 B 6 14 4.30 12135 0 50 10 120 0
9 A 9 15 4.32 12189 0 43 17 120 0

Note that several samples (highlighted in bold) show inverted ages with respect to the previous samples. This is expected, since the slotting algorithm only takes into account distance/dissimilarity between adjacent samples to generate the ordering.

Sequences A (green) and B (blue) with their ordered samples (upper panel), and the composite sequence resulting from them (lower panel) after applying the sequence slotting algorithm. Notice that the slotting takes into account all columns in both datasets, and therefore, a single column, as shown in the plot, might not be totally representative of the slotting solution.

#cleaning workspace
rm(A, B, AB, AB.combined, pollenGP, cols, taxon, y.lim)

Transferring an attribute from one sequence to another

This scenario assumes that the user has two multivariate sequences, one of them with a given attribute (age/time) that needs to be transferred to the other sequence by using similarity/dissimilarity (constrained by sample order) as a transfer criterion.

The code below prepares the data for the example. The sequence pollenGP is the reference sequence, and contains the column age. The sequence pollenX is the target sequence, without an age column. We generate it by taking 40 random samples between the samples 50 and 100 of pollenGP. The sequences are prepared with prepareSequences, as usual, with the identificators “GP” and “X”

The transfer of “age” values from GP to X can be done in two ways, both constrained by sample order:

  • Direct: each sample in X gets the age of its most similar sample in GP.
  • Interpolated: each sample in X gets an age interpolated from the ages of the two most similar samples in GP. The interpolation is weighted by the similarity between the samples.

Direct transfer

A direct transfer of an attribute from the samples of one sequence to the samples of another requires to compute a distance matrix between samples, the least cost matrix and its least cost path (both with the option diagonal activated), and to parse the least cost path file to assign attribute values. This is done by the function workflowTransfer with the option $mode = “direct”$.

id depth age Abies Juniperus Hedera Plantago Boraginaceae Crassulaceae Pinus Ranunculaceae Rhamnus Caryophyllaceae Dipsacaceae Betula Acer Armeria Tilia Hippophae Salix Thalictrum Labiatae Valeriana Nymphaea Umbelliferae Sanguisorba_minor Plantago.lanceolata Campanulaceae Asteroideae Gentiana Fraxinus Cichorioideae Taxus Rumex Cedrus Ranunculus.subgen..Batrachium Cyperaceae Corylus Myriophyllum Filipendula Vitis Rubiaceae Polypodium
41 X 0 3.92 11108 0 7 0 5 20 0 13 0 2 1 0 2 41 0 0 0 0 0 0 0 0 1 0 8 0 0 0 0 0 0 0 0 0 0 0 60 2 0 0
42 X 0 4.00 11324 0 8 43 60 65 0 10 0 0 2 4 0 0 0 0 0 0 0 2 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 1 11 0 2 0
43 X 0 4.02 11378 0 44 76 110 0 0 0 0 0 2 11 0 0 0 0 3 0 1 3 0 0 5 0 0 1 0 6 0 0 1 0 1 0 0 0 0 1 1 1
44 X 0 4.05 11459 0 20 73 94 1 0 0 0 0 1 10 0 2 0 0 3 0 1 3 0 0 3 0 0 0 0 3 0 0 2 0 0 0 0 0 0 1 0 0
45 X 0 4.07 11514 0 20 80 100 0 0 0 0 0 10 4 0 0 1 0 0 0 0 3 0 0 1 0 0 0 0 8 1 0 0 0 0 0 1 2 1 0 1 0
46 X 0 4.07 11595 0 34 80 155 0 0 0 2 0 2 13 0 0 0 0 1 0 0 2 0 0 2 0 0 0 0 6 0 0 0 0 0 0 1 1 0 0 1 0
47 X 0 4.17 11784 0 22 44 131 0 0 0 0 0 1 13 0 0 0 0 5 0 0 5 0 0 3 0 0 0 0 6 0 0 0 0 2 0 1 0 0 2 0 0
48 X 0 4.20 11865 0 35 30 112 0 0 0 0 0 4 8 0 0 0 2 2 0 1 2 0 0 7 0 1 0 0 10 1 0 0 0 1 0 1 1 0 2 1 0
49 X 0 4.22 11919 0 30 45 150 0 0 0 0 0 4 11 0 0 0 0 2 0 3 1 0 0 1 0 0 0 0 13 1 0 0 0 0 0 0 0 0 2 1 0
50 X 0 4.25 12000 0 44 35 150 0 0 0 0 0 2 8 0 0 0 0 3 0 1 0 0 0 5 0 0 0 0 6 3 0 0 0 0 0 0 2 0 1 0 0
51 X 0 4.25 12054 0 37 30 150 0 0 1 0 0 6 10 0 0 0 0 7 0 2 2 0 0 6 0 0 0 0 7 1 0 0 0 0 0 0 3 0 4 0 0
52 X 0 4.32 12135 0 50 10 120 0 0 0 0 0 1 7 0 0 0 0 1 0 0 1 0 0 8 0 0 0 0 8 2 0 2 0 0 0 0 0 0 1 0 0
53 X 0 4.32 12189 0 43 17 120 0 0 0 0 0 2 15 0 0 1 1 2 0 0 2 0 0 5 1 0 0 0 6 2 0 2 0 1 0 2 1 0 0 0 0
54 X 0 4.40 12324 0 50 11 86 0 0 0 0 0 2 15 0 0 2 1 4 0 5 3 0 0 6 0 0 0 0 5 1 0 2 0 0 1 1 0 0 3 1 0
55 X 0 4.40 12405 0 51 6 70 0 0 0 0 0 1 16 0 0 4 1 2 0 4 2 0 0 5 2 0 0 0 1 2 0 1 0 0 0 0 0 0 0 3 1

The algorithm finds the most similar samples, and transfers attribute values directly between them. This can result in duplicated attribute values, as highlighted in the table above. The Pearson correlation between the original ages (stored in pollenX.age) and the assigned ones is 0.9996, so it can be concluded that in spite of its simplicity, this algorithm yields accurate results.

#cleaning workspace
rm(GP.X, pollenX, X.new, pollenX.age)

Interpolated transfer

If we consider:

  • Two samples $A_{i}$ and $A_{j}$ of the sequence $A$.
  • Each one with a value the attribute $t$, $At_{i}$ and $At_{j}$.
  • One sample $B_{k}$ of the sequence $B$.
  • With an unknown attribute $Bt_{k}$.
  • The multivariate distance $D_{B_{k}A_{i}}$ between the samples $A_{i}$ and $B_{k}$.
  • The multivariate distance $D_{B_{k}A_{j}}$ between the samples $A_{j}$ and $B_{k}$.
  • The weight $w_{i}$, computed as $D_{B_{k}A_{i}} / (D_{B_{k}A_{i}} + D_{B_{k}A_{j}})$
  • The weight $w_{j}$, computed as $D_{B_{k}A_{j}} / (D_{B_{k}A_{i}} + D_{B_{k}A_{j}})$

The unknwon value $Bt_{k}$ is computed as:

$$Bt_{k} = w_{i} \times At_{i} + w_{j} \times At_{j}$$

The code below exemplifies the operation, using the samples 1 and 4 of the dataset pollenGP as $Ai$ and $Aj$, and the sample 3 as $Bk$.

The table below shows the observed versus the predicted values for $Btk$.

temp.df <- data.frame(Observed = pollenGP[3, "age"], Predicted = Btk)
kable(t(temp.df), digits = 4)
Observed 3.9700
Predicted 3.9735

Below we create some example data, where a subset of pollenGP will be the donor of age values, and another subset of it, named pollenX will be the receiver of the age values.

To transfer attributes from GP to X we use the workflowTransfer function with the option mode = “interpolate”.

id depth age Abies Juniperus Hedera Plantago Boraginaceae Crassulaceae Pinus Ranunculaceae Rhamnus Caryophyllaceae Dipsacaceae Betula Acer Armeria Tilia Hippophae Salix Thalictrum Labiatae Valeriana Nymphaea Umbelliferae Sanguisorba_minor Plantago.lanceolata Campanulaceae Asteroideae Gentiana Fraxinus Cichorioideae Taxus Rumex Cedrus Ranunculus.subgen..Batrachium Cyperaceae Corylus Myriophyllum Filipendula Vitis Rubiaceae Polypodium
41 X 0 3.9498 11108 0 7 0 5 20 0 13 0 2 1 0 2 41 0 0 0 0 0 0 0 0 1 0 8 0 0 0 0 0 0 0 0 0 0 0 60 2 0 0
42 X 0 3.9705 11324 0 8 43 60 65 0 10 0 0 2 4 0 0 0 0 0 0 0 2 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 1 11 0 2 0
43 X 0 4.0003 11378 0 44 76 110 0 0 0 0 0 2 11 0 0 0 0 3 0 1 3 0 0 5 0 0 1 0 6 0 0 1 0 1 0 0 0 0 1 1 1
44 X 0 4.0212 11459 0 20 73 94 1 0 0 0 0 1 10 0 2 0 0 3 0 1 3 0 0 3 0 0 0 0 3 0 0 2 0 0 0 0 0 0 1 0 0
45 X 0 4.0512 11514 0 20 80 100 0 0 0 0 0 10 4 0 0 1 0 0 0 0 3 0 0 1 0 0 0 0 8 1 0 0 0 0 0 1 2 1 0 1 0
46 X 0 4.1308 11595 0 34 80 155 0 0 0 2 0 2 13 0 0 0 0 1 0 0 2 0 0 2 0 0 0 0 6 0 0 0 0 0 0 1 1 0 0 1 0
47 X 0 4.1985 11784 0 22 44 131 0 0 0 0 0 1 13 0 0 0 0 5 0 0 5 0 0 3 0 0 0 0 6 0 0 0 0 2 0 1 0 0 2 0 0
48 X 0 NA 11865 0 35 30 112 0 0 0 0 0 4 8 0 0 0 2 2 0 1 2 0 0 7 0 1 0 0 10 1 0 0 0 1 0 1 1 0 2 1 0
49 X 0 4.2014 11919 0 30 45 150 0 0 0 0 0 4 11 0 0 0 0 2 0 3 1 0 0 1 0 0 0 0 13 1 0 0 0 0 0 0 0 0 2 1 0
50 X 0 4.2223 12000 0 44 35 150 0 0 0 0 0 2 8 0 0 0 0 3 0 1 0 0 0 5 0 0 0 0 6 3 0 0 0 0 0 0 2 0 1 0 0
51 X 0 4.2301 12054 0 37 30 150 0 0 1 0 0 6 10 0 0 0 0 7 0 2 2 0 0 6 0 0 0 0 7 1 0 0 0 0 0 0 3 0 4 0 0
52 X 0 4.2726 12135 0 50 10 120 0 0 0 0 0 1 7 0 0 0 0 1 0 0 1 0 0 8 0 0 0 0 8 2 0 2 0 0 0 0 0 0 1 0 0
53 X 0 NA 12189 0 43 17 120 0 0 0 0 0 2 15 0 0 1 1 2 0 0 2 0 0 5 1 0 0 0 6 2 0 2 0 1 0 2 1 0 0 0 0
54 X 0 4.3521 12324 0 50 11 86 0 0 0 0 0 2 15 0 0 2 1 4 0 5 3 0 0 6 0 0 0 0 5 1 0 2 0 0 1 1 0 0 3 1 0
55 X 0 4.4177 12405 0 51 6 70 0 0 0 0 0 1 16 0 0 4 1 2 0 4 2 0 0 5 2 0 0 0 1 2 0 1 0 0 0 0 0 0 0 3 1

When interpolated values of the age column (transferred attribute via interpolation) show the value NA, it means that the interpolation yielded an age lower than the previous one. This happens when the same $Ai$ and $Aj$ are used to evaluate two or more different samples $Bk$, and the second $Bk$ is more similar to $Ai$ than the first one. These NA values can be removed with na.omit(), or interpolated with the functions imputeTS::na.interpolation or zoo::na.approx.

Without taking into account these NA values, the Pearson correlation of the interpolated ages with the real ones is 0.9984.

IMPORTANT: the interpretation of the interpolated ages requires a careful consideration. Please, don’t do it blindly, because this algorithm has its limitations. For example, significant hiatuses in the data can introduce wild variations in interpolated ages.