Batch transformations

We perform the same analysis for the target models of the batch transformations.

target_model_statistics <- trainbenchmark_log %>%
  filter(experiment == 'viewModel-batch-physical' &
           modificationMix == 'none' &
           rerun == 0 &
           checkpoint == 'batch' &
           category == 'target') %>%
  group_by(modelSize, transformationCase, variable) %>%
  summarize(value = first(value)) %>%
  mutate(variable = paste0('target_', variable)) %>%
  spread(variable, value)

tstat_cases <- TransformationCaseFactor(target_model_statistics$transformationCase)
tstat_df <- data.frame(target_model_statistics$modelSize, tstat_cases, target_model_statistics$target_count, target_model_statistics$target_referenceCount, target_model_statistics$target_attributeCount)

colnames(tstat_df) <- c("Scale factor", "Case study", "Target objects", "Target references", "Target attributes")
knitr::kable(tstat_df, format='markdown') %>% cat(sep = '\n')

For sanity, we check that all transformations resulted in the same number of target model elements.

bad_batch_transformations <- trainbenchmark_log %>%
  filter(modificationMix == 'none' &
           checkpoint == 'batch' &
           category == 'target') %>%
  select(-c(modificationMix, checkpoint, category)) %>%
  mutate(variable = paste0('actual_', variable)) %>%
  spread(variable, value) %>%
  inner_join(target_model_statistics, by=c('transformationCase', 'modelSize')) %>%
  filter(actual_rootCount != target_rootCount |
           actual_count != target_count |
           actual_referenceCount != target_referenceCount |
           actual_attributeCount != target_attributeCount)

if (nrow(bad_batch_transformations) != 0) {
  print(bad_batch_transformations)
  stop("Unexpected batch transformation results")
} else {
  message("All correct")
}

## All correct

We may see that every batch transformation resulted in the expected number of target elements.

Now we count the variables and constraints in the partial models.

partial_size <- trainbenchmark_log %>%
  filter(modificationMix == 'none' &
           checkpoint == 'batch' & category == 'trace' &
           variable %in% c('variableCount', 'constraintCount') &
           rerun == 0) %>%
  select(c(transformationCase, variable, value, modelSize)) %>%
  spread(variable, value)

partial_cases <- TransformationCaseFactor(partial_size$transformationCase)
partial_df = data.frame(partial_size$modelSize, partial_cases, partial_size$variableCount, partial_size$constraintCount)

colnames(partial_df) <- c("Scale factor", "Case study", "Partial model variables", "Partial model constraints")
knitr::kable(partial_df, format='markdown') %>% cat(sep = '\n')

Change-driven transformations

Different modification mixed produce different output models, so we also collect statistics for the target model after each modification mix separately.

incremental_target_model_statistics <- trainbenchmark_log %>%
  filter(experiment == 'viewModel-incremental-physical' &
           modificationMix != 'none' &
           rerun == 0 &
           checkpoint == 'after' &
           category == 'target') %>%
  group_by(modelSize, transformationCase, modificationMix, variable) %>%
  summarize(value = first(value)) %>%
  mutate(variable = paste0('target_', variable)) %>%
  spread(variable, value) %>%
  ungroup()

incremental_tstat_df <- data.frame(
  incremental_target_model_statistics$modelSize,
  TransformationCaseFactor(incremental_target_model_statistics$transformationCase),
  ModificationMixFactor(incremental_target_model_statistics$modificationMix),
  incremental_target_model_statistics$target_count,
  incremental_target_model_statistics$target_referenceCount,
  incremental_target_model_statistics$target_attributeCount) %>%
  arrange_at(c(2, 1, 3))

colnames(incremental_tstat_df) <- c("Scale factor", "Case study", "Modification mix", "Target objects", "Target references", "Target attributes")
knitr::kable(incremental_tstat_df, format='markdown') %>% cat(sep = '\n')

We make sure that each experiment resulted in the same number of target model elements when executed with the same source model and modification mix.

incremental_target_model_statistics_by_experiment <- trainbenchmark_log %>%
  filter(modificationMix != 'none' &
           checkpoint == 'after' &
           category == 'target') %>%
  group_by(modelSize, transformationCase, modificationMix, experiment, rerun, variable) %>%
  summarize(value = first(value)) %>%
  mutate(variable = paste0('actual_', variable)) %>%
  spread(variable, value) %>%
  ungroup()

bad_incremental_experiments <- incremental_target_model_statistics_by_experiment %>%
  inner_join(incremental_target_model_statistics,
    by = c('modelSize', 'transformationCase', 'modificationMix')) %>%
  filter(actual_rootCount != target_rootCount |
           actual_count != target_count |
           actual_referenceCount != target_referenceCount |
           actual_attributeCount != target_attributeCount)


if (nrow(bad_incremental_experiments) != 0) {
  print(bad_incremental_experiments)
  stop("Unexpected incremental transformation results")
} else {
  message("All correct")
}

## All correct

We may see that every incremental transformation resulted in the expected number of target elements.

First execution (RQ2)

We will “formally” treat model query and first execution as two modification mixes, which simplifies our data frames. They will be shown on the same plots as the incremental sychronization times, anyway.

We collect the execution time of the model query in the first (also known as “batch” in the logs) execution. This is relatively simple, as all experiments contain a modelQuery checkpoint.

modelQuery <- trainbenchmark_log %>%
  filter(modificationMix == 'none' &
           variable == 'duration' &
           checkpoint == 'modelQuery') %>%
  mutate(modificationMix='modelQuery') %>%
  RenameExperiment()

We extract the first execution time of the hand-written VIATRA transformations, too. Only the hand-written transformations have an execute checkpoint in the log.

execution_viatra <- trainbenchmark_log %>%
  filter(modificationMix == 'none' &
           variable == 'duration' &
           checkpoint == 'execute') %>%
  mutate(modificationMix='execute') %>%
  RenameExperiment()

Extracting the first run of the ViewModel transformations is a bit more involved, as different steps were logged separately. We simply sum their durations.

execution_viewmodel <- trainbenchmark_log %>%
  filter(modificationMix == 'none' &
           variable == 'duration' &
           checkpoint %in% c('pt2tExecute', 'pt2tRete', 's2ptExecute', 's2ptRete')) %>%
  group_by(transformationCase, experiment, rerun, category, variable, modelSize) %>%
  summarize(checkpoint='execute', modificationMix='execute', value=sum(value)) %>%
  ungroup() %>%
  RenameExperiment()

Incremental transformations (RQ3)

The execution time of change-driven synchronization is split between the modelModification (propagation of source model changes trough the RETE net) and synchronization (firing of change-driven transformation rules) phases. We sum the two durations.

incremental <- trainbenchmark_log %>%
  filter(modificationMix != 'none' &
           variable == 'duration' &
           checkpoint %in% c('synchronization', 'modelModification')) %>%
  group_by(transformationCase, experiment, rerun, category, variable, modelSize, modificationMix) %>%
  summarize(checkpoint='synchronization', value=sum(value)) %>%
  ungroup() %>%
  RenameExperiment()

Putting it together

We bind the data frames from the previous two sections, and join them to the target and source model statistics. Then we prepare for creating the plots as follows:

• The total model size is the geometric mean of the number of source and target model objects. This measure was selected because it can indicate scaleability in both case studies considered.
• Only the median of the execution times is kept.
• Execution times smaller than 1 ms are replaced with 1 ms so that a log-log plot can be drawn. Zero execution times would lead to NaN values after taking their logarithm.

durations_plot <- rbind(incremental, modelQuery, execution_viatra, execution_viewmodel) %>%
  inner_join(target_model_statistics, by = c('transformationCase', 'modelSize')) %>%
  inner_join(source_model_statistics, by = c('modelSize')) %>%
  mutate(total_count = sqrt(source_count * target_count)) %>%
  group_by(modelSize, transformationCase, modificationMix, experiment) %>%
  summarize(total_count = median(total_count), value = median(value)) %>%
  mutate(value = ifelse(value < 1, 1, value))

We add some factor labels in order to generate appropriate legends in the plots.

durations_plot$modificationMix <- ModificationMixFactor(durations_plot$modificationMix)
durations_plot$transformationCase <- TransformationCaseFactor(durations_plot$transformationCase)
durations_plot$experiment <- ExperimentFactor(durations_plot$experiment)
durations_plot <- durations_plot %>%
  arrange(transformationCase, modelSize, modificationMix, experiment)

A large table can be assembled for viewing by spreading the three experiments side by side. Note that different modification mixes were evaluated on different virtual machines, so executing times are only comparable within a single modification mix.

durations_table <- durations_plot %>%
  spread(experiment, value) %>%
  arrange(modificationMix, transformationCase, modelSize)
colnames(durations_table)[1:4] <- c("Scale factor", "Case study", "Modification mix", "Total size")
knitr::kable(durations_table, format='markdown') %>% cat(sep = '\n')