message: Update IWSLT configuration for transformer

diff:
@@ -36,6 +36,31 @@ $ python generate.py data-bin/iwslt14.tokenized.de-en \
```
+To train transformer model on IWSLT'14 German to English:
+```
+# Preparation steps are the same as for fconv model.
+
+# Train the model (better for a single GPU setup):
+$ mkdir -p checkpoints/transformer
+$ CUDA_VISIBLE_DEVICES=0 python train.py data-bin/iwslt14.tokenized.de-en \
+ -a transformer_iwslt_de_en --optimizer adam --lr 0.0005 -s de -t en \
+ --label-smoothing 0.1 --dropout 0.3 --max-tokens 4000 \
+ --min-lr '1e-09' --lr-scheduler inverse_sqrt --weight-decay 0.0001 \
+ --criterion label_smoothed_cross_entropy --max-update 50000 \
+ --warmup-updates 4000 --warmup-init-lr '1e-07' \
+ --adam-betas '(0.9, 0.98)' --save-dir checkpoints/transformer
+
+# Average 10 latest checkpoints:
+$ python scripts/average_checkpoints.py --inputs checkpoints/transformer \
+ --num-epoch-checkpoints 10 --output checkpoints/transformer/model.pt
+
+# Generate:
+$ python generate.py data-bin/iwslt14.tokenized.de-en \
+ --path checkpoints/transformer/model.pt \
+ --batch-size 128 --beam 5 --remove-bpe
+
+```
+
### prepare-wmt14en2de.sh

diff:
@@ -588,14 +588,14 @@ def base_architecture(args):
@register_model_architecture('transformer', 'transformer_iwslt_de_en')
def transformer_iwslt_de_en(args):
- args.encoder_embed_dim = getattr(args, 'encoder_embed_dim', 256)
- args.encoder_ffn_embed_dim = getattr(args, 'encoder_ffn_embed_dim', 512)
+ args.encoder_embed_dim = getattr(args, 'encoder_embed_dim', 512)
+ args.encoder_ffn_embed_dim = getattr(args, 'encoder_ffn_embed_dim', 1024)
args.encoder_attention_heads = getattr(args, 'encoder_attention_heads', 4)
- args.encoder_layers = getattr(args, 'encoder_layers', 3)
- args.decoder_embed_dim = getattr(args, 'decoder_embed_dim', 256)
- args.decoder_ffn_embed_dim = getattr(args, 'decoder_ffn_embed_dim', 512)
+ args.encoder_layers = getattr(args, 'encoder_layers', 6)
+ args.decoder_embed_dim = getattr(args, 'decoder_embed_dim', 512)
+ args.decoder_ffn_embed_dim = getattr(args, 'decoder_ffn_embed_dim', 1024)
args.decoder_attention_heads = getattr(args, 'decoder_attention_heads', 4)
- args.decoder_layers = getattr(args, 'decoder_layers', 3)
+ args.decoder_layers = getattr(args, 'decoder_layers', 6)
base_architecture(args)


message: Reset gnorm after each epoch

change_type: MODIFY

diff:
@@ -155,7 +155,7 @@ def train(args, trainer, task, epoch_itr):
progress.print(stats)
# reset training meters
- for k in ['train_loss', 'train_nll_loss', 'wps', 'ups', 'wpb', 'bsz', 'clip']:
+ for k in ['train_loss', 'train_nll_loss', 'wps', 'ups', 'wpb', 'bsz', 'clip', 'gnorm']:
meter = trainer.get_meter(k)
if meter is not None:
meter.reset()

license: MIT License

original_message: Reset gnorm after each epoch


message: Fix adaptive softmax cutoff comment

diff:
@@ -25,7 +25,7 @@ class AdaptiveSoftmax(nn.Module):
cutoff = cutoff + [vocab_size]
else:
assert vocab_size == cutoff[
- -1], 'cannot specify cutoff smaller than vocab size'
+ -1], 'cannot specify cutoff larger than vocab size'
output_dim = cutoff[0] + len(cutoff) - 1

original_message: Fix adaptive softmax cutoff comment


message: Old checkpoints can't be loaded because of a new meter

diff:
@@ -107,7 +107,7 @@ class Trainer(object):
self._num_updates = last_optim['num_updates']
if extra_state is not None and 'train_meters' in extra_state:
- self.meters = extra_state['train_meters']
+ self.meters.update(extra_state['train_meters'])
del extra_state['train_meters']
# reset TimeMeters, since their start times don't make sense anymore


message: Update readme with WMT'18 model

diff:
@@ -7,7 +7,7 @@ of various sequence-to-sequence models, including:
- **Convolutional Neural Networks (CNN)**
- [Dauphin et al. (2017): Language Modeling with Gated Convolutional Networks](https://arxiv.org/abs/1612.08083)
- [Gehring et al. (2017): Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122)
- - **_New_** [Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning](https://arxiv.org/abs/1711.04956)
+ - [Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning](https://arxiv.org/abs/1711.04956)
- **_New_** [Fan et al. (2018): Hierarchical Neural Story Generation](https://arxiv.org/abs/1805.04833)
- **Long Short-Term Memory (LSTM) networks**
- [Luong et al. (2015): Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025)
@@ -15,6 +15,7 @@ of various sequence-to-sequence models, including:
- **Transformer (self-attention) networks**
- [Vaswani et al. (2017): Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- **_New_** [Ott et al. (2018): Scaling Neural Machine Translation](https://arxiv.org/abs/1806.00187)
+ - **_New_** [Edunov et al. (2018): Understanding Back-Translation at Scale](https://arxiv.org/abs/1808.09381)
Fairseq features:
- multi-GPU (distributed) training on one machine or across multiple machines
@@ -64,6 +65,7 @@ Convolutional
 ([Gehring et al., 2017](https://arxiv.org/abs/1705.03122)) |
Convolutional
 ([Gehring et al., 2017](https://arxiv.org/abs/1705.03122)) | [WMT17 English-German](http://statmt.org/wmt17/translation-task.html#Download) | [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/models/wmt17.v2.en-de.fconv-py.tar.bz2) | newstest2014:
 [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/data/wmt17.v2.en-de.newstest2014.tar.bz2)
Transformer
 ([Ott et al., 2018](https://arxiv.org/abs/1806.00187)) | [WMT14 English-French](http://statmt.org/wmt14/translation-task.html#Download) | [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/models/wmt14.en-fr.joined-dict.transformer.tar.bz2) | newstest2014 (shared vocab):
 [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/data/wmt14.en-fr.joined-dict.newstest2014.tar.bz2)
Transformer
 ([Ott et al., 2018](https://arxiv.org/abs/1806.00187)) | [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) | [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/models/wmt16.en-de.joined-dict.transformer.tar.bz2) | newstest2014 (shared vocab):
 [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/data/wmt16.en-de.joined-dict.newstest2014.tar.bz2)
+Transformer
 ([Edunov et al. (2018)](https://arxiv.org/abs/1808.09381)) | [WMT'18 English-German](http://www.statmt.org/wmt18/translation-task.html) | [download (.tar.bz2)](https://s3.amazonaws.com/fairseq-py/models/wmt18.en-de.ensemble.tar.bz2) | See NOTE in the archive
### Language models

original_message: Update readme with WMT'18 model (#433)


message: Fix adaptive loss logging

diff:
@@ -74,6 +74,7 @@ class AdaptiveLoss(FairseqCriterion):
sample_size = sum(log.get('sample_size', 0) for log in logging_outputs)
agg_output = {
'loss': loss_sum / sample_size / math.log(2),
+ 'nll_loss': loss_sum / sample_size / math.log(2),
'ntokens': ntokens,
'nsentences': nsentences,
'sample_size': sample_size,

original_message: Fix adaptive loss logging


message: Parallel preprocessing

diff:
@@ -52,9 +52,15 @@ def data_file_path(prefix_path):
class IndexedDataset(torch.utils.data.Dataset):
"""Loader for TorchNet IndexedDataset"""
- def __init__(self, path, fix_lua_indexing=False):
+ def __init__(self, path, fix_lua_indexing=False, read_data=True):
super().__init__()
self.fix_lua_indexing = fix_lua_indexing
+ self.read_index(path)
+ self.data_file = None
+ if read_data:
+ self.read_data(path)
+
+ def read_index(self, path):
with open(index_file_path(path), 'rb') as f:
magic = f.read(8)
assert magic == b'TNTIDX\x00\x00'
@@ -66,7 +72,6 @@ class IndexedDataset(torch.utils.data.Dataset):
self.dim_offsets = read_longs(f, self.size + 1)
self.data_offsets = read_longs(f, self.size + 1)
self.sizes = read_longs(f, self.s)
- self.read_data(path)
def read_data(self, path):
self.data_file = open(data_file_path(path), 'rb', buffering=0)
@@ -76,6 +81,7 @@ class IndexedDataset(torch.utils.data.Dataset):
raise IndexError('index out of range')
def __del__(self):
+ if self.data_file:
self.data_file.close()
def __getitem__(self, i):
@@ -193,6 +199,26 @@ class IndexedDatasetBuilder(object):
self.sizes.append(s)
self.dim_offsets.append(self.dim_offsets[-1] + len(tensor.size()))
+ def merge_file_(self, another_file):
+ index = IndexedDataset(another_file, read_data=False)
+ assert index.dtype == self.dtype
+
+ begin = self.data_offsets[-1]
+ for offset in index.data_offsets[1:]:
+ self.data_offsets.append(begin + offset)
+ self.sizes.extend(index.sizes)
+ begin = self.dim_offsets[-1]
+ for dim_offset in index.dim_offsets[1:]:
+ self.dim_offsets.append(begin + dim_offset)
+
+ with open(data_file_path(another_file), 'rb') as f:
+ while True:
+ data = f.read(1024)
+ if data:
+ self.out_file.write(data)
+ else:
+ break
+
def finalize(self, index_file):
self.out_file.close()
index = open(index_file, 'wb')

diff:
# can be found in the PATENTS file in the same directory.
from collections import Counter
-import re
+import os, re
import torch
-
+from multiprocessing import Pool
SPACE_NORMALIZER = re.compile("\s+")
@@ -20,28 +20,74 @@ def tokenize_line(line):
return line.split()
+def safe_readline(f):
+ pos = f.tell()
+ while True:
+ try:
+ return f.readline()
+ except UnicodeDecodeError:
+ pos -= 1
+ f.seek(pos) # search where this character begins
+
class Tokenizer:
@staticmethod
- def add_file_to_dictionary(filename, dict, tokenize):
+ def add_file_to_dictionary_single_worker(filename, tokenize, eos_word, worker_id=0, num_workers=1):
+ counter = Counter()
with open(filename, 'r') as f:
- for line in f:
+ size = os.fstat(f.fileno()).st_size
+ chunk_size = size // num_workers
+ offset = worker_id * chunk_size
+ end = offset + chunk_size
+ f.seek(offset)
+ if offset > 0:
+ safe_readline(f) # drop first incomplete line
+ line = f.readline()
+ while line:
for word in tokenize(line):
- dict.add_symbol(word)
- dict.add_symbol(dict.eos_word)
+ counter.update([word])
+ counter.update([eos_word])
+ if f.tell() > end:
+ break
+ line = f.readline()
+ return counter
+
+ @staticmethod
+ def add_file_to_dictionary(filename, dict, tokenize, num_workers):
+ def merge_result(counter):
+ for w, c in counter.items():
+ dict.add_symbol(w, c)
+ if num_workers > 1:
+ pool = Pool(processes=num_workers)
+ results = []
+ for worker_id in range(num_workers):
+ results.append(pool.apply_async(
+ Tokenizer.add_file_to_dictionary_single_worker,
+ (filename, tokenize, dict.eos_word, worker_id, num_workers)
+ ))
+ pool.close()
+ pool.join()
+ for r in results:
+ merge_result(r.get())
+ else:
+ merge_result(Tokenizer.add_file_to_dictionary_single_worker(filename, tokenize, dict.eos_word))
@staticmethod
def binarize(filename, dict, consumer, tokenize=tokenize_line,
- append_eos=True, reverse_order=False):
+ append_eos=True, reverse_order=False,
+ offset=0, end=-1):
nseq, ntok = 0, 0
replaced = Counter()
-
def replaced_consumer(word, idx):
if idx == dict.unk_index and word != dict.unk_word:
replaced.update([word])
-
with open(filename, 'r') as f:
- for line in f:
+ f.seek(offset)
+ # next(f) breaks f.tell(), hence readline() must be used
+ line = safe_readline(f)
+ while line:
+ if end > 0 and f.tell() > end:
+ break
ids = Tokenizer.tokenize(
line=line,
dict=dict,
@@ -52,10 +98,22 @@ class Tokenizer:
reverse_order=reverse_order,
)
nseq += 1
-
- consumer(ids)
ntok += len(ids)
- return {'nseq': nseq, 'nunk': sum(replaced.values()), 'ntok': ntok, 'replaced': len(replaced)}
+ consumer(ids)
+ line = f.readline()
+ return {'nseq': nseq, 'nunk': sum(replaced.values()), 'ntok': ntok, 'replaced': replaced}
+
+ @staticmethod
+ def find_offsets(filename, num_chunks):
+ with open(filename, 'r') as f:
+ size = os.fstat(f.fileno()).st_size
+ chunk_size = size // num_chunks
+ offsets = [0 for _ in range(num_chunks + 1)]
+ for i in range(1, num_chunks):
+ f.seek(chunk_size * i)
+ safe_readline(f)
+ offsets[i] = f.tell()
+ return offsets
@staticmethod
def tokenize(line, dict, tokenize=tokenize_line, add_if_not_exist=True,

diff:
@@ -10,12 +10,16 @@ Data pre-processing: build vocabularies and binarize training data.
"""
import argparse
+from collections import Counter
from itertools import zip_longest
import os
import shutil
+
from fairseq.data import indexed_dataset, dictionary
from fairseq.tokenizer import Tokenizer, tokenize_line
+from multiprocessing import Pool, Manager, Process
+
def get_parser():
@@ -41,6 +45,7 @@ def get_parser():
parser.add_argument('--only-source', action='store_true', help='Only process the source language')
parser.add_argument('--padding-factor', metavar='N', default=8, type=int,
help='Pad dictionary size to be multiple of N')
+ parser.add_argument('--workers', metavar='N', default=1, type=int, help='number of parallel workers')
return parser
@@ -52,7 +57,7 @@ def main(args):
def build_dictionary(filenames):
d = dictionary.Dictionary()
for filename in filenames:
- Tokenizer.add_file_to_dictionary(filename, d, tokenize_line)
+ Tokenizer.add_file_to_dictionary(filename, d, tokenize_line, args.workers)
return d
def train_path(lang):
@@ -70,11 +75,6 @@ def main(args):
def dict_path(lang):
return dest_path('dict', lang) + '.txt'
- def dataset_dest_path(output_prefix, lang, extension):
- base = f'{args.destdir}/{output_prefix}'
- lang_part = f'.{args.source_lang}-{args.target_lang}.{lang}' if lang is not None else ''
- return f'{base}{lang_part}.{extension}'
-
if args.joined_dictionary:
assert not args.srcdict, 'cannot combine --srcdict and --joined-dictionary'
assert not args.tgtdict, 'cannot combine --tgtdict and --joined-dictionary'
@@ -111,25 +111,54 @@ def main(args):
)
tgt_dict.save(dict_path(args.target_lang))
- def make_binary_dataset(input_prefix, output_prefix, lang):
+ def make_binary_dataset(input_prefix, output_prefix, lang, num_workers):
dict = dictionary.Dictionary.load(dict_path(lang))
print('| [{}] Dictionary: {} types'.format(lang, len(dict) - 1))
+ n_seq_tok = [0, 0]
+ replaced = Counter()
- ds = indexed_dataset.IndexedDatasetBuilder(dataset_dest_path(output_prefix, lang, 'bin'))
-
- def consumer(tensor):
- ds.add_item(tensor)
+ def merge_result(worker_result):
+ replaced.update(worker_result['replaced'])
+ n_seq_tok[0] += worker_result['nseq']
+ n_seq_tok[1] += worker_result['ntok']
input_file = '{}{}'.format(input_prefix, ('.' + lang) if lang is not None else '')
- res = Tokenizer.binarize(input_file, dict, consumer)
+ offsets = Tokenizer.find_offsets(input_file, num_workers)
+ pool = None
+ if num_workers > 1:
+ pool = Pool(processes=num_workers-1)
+ for worker_id in range(1, num_workers):
+ prefix = "{}{}".format(output_prefix, worker_id)
+ pool.apply_async(binarize, (args, input_file, dict, prefix, lang,
+ offsets[worker_id],
+ offsets[worker_id + 1]), callback=merge_result)
+ pool.close()
+
+ ds = indexed_dataset.IndexedDatasetBuilder(dataset_dest_file(args, output_prefix, lang, 'bin'))
+ merge_result(Tokenizer.binarize(input_file, dict, lambda t: ds.add_item(t),
+ offset=0, end=offsets[1]))
+ if num_workers > 1:
+ pool.join()
+ for worker_id in range(1, num_workers):
+ prefix = "{}{}".format(output_prefix, worker_id)
+ temp_file_path = dataset_dest_prefix(args, prefix, lang)
+ ds.merge_file_(temp_file_path)
+ os.remove(indexed_dataset.data_file_path(temp_file_path))
+ os.remove(indexed_dataset.index_file_path(temp_file_path))
+
+
+ ds.finalize(dataset_dest_file(args, output_prefix, lang, 'idx'))
+
+
print('| [{}] {}: {} sents, {} tokens, {:.3}% replaced by {}'.format(
- lang, input_file, res['nseq'], res['ntok'],
- 100 * res['nunk'] / res['ntok'], dict.unk_word))
- ds.finalize(dataset_dest_path(output_prefix, lang, 'idx'))
+ lang, input_file, n_seq_tok[0], n_seq_tok[1],
+ 100 * sum(replaced.values()) / n_seq_tok[1], dict.unk_word))
+
+
- def make_dataset(input_prefix, output_prefix, lang):
+ def make_dataset(input_prefix, output_prefix, lang, num_workers=1):
if args.output_format == 'binary':
- make_binary_dataset(input_prefix, output_prefix, lang)
+ make_binary_dataset(input_prefix, output_prefix, lang, num_workers)
elif args.output_format == 'raw':
# Copy original text file to destination folder
output_text_file = dest_path(
@@ -140,7 +169,7 @@ def main(args):
def make_all(lang):
if args.trainpref:
- make_dataset(args.trainpref, 'train', lang)
+ make_dataset(args.trainpref, 'train', lang, num_workers=args.workers)
if args.validpref:
for k, validpref in enumerate(args.validpref.split(',')):
outprefix = 'valid{}'.format(k) if k > 0 else 'valid'
@@ -196,6 +225,28 @@ def main(args):
print('{} {}'.format(src_dict[k], tgt_dict[v]), file=f)
+
+def binarize(args, filename, dict, output_prefix, lang, offset, end):
+
+ ds = indexed_dataset.IndexedDatasetBuilder(dataset_dest_file(args, output_prefix, lang, 'bin'))
+ def consumer(tensor):
+ ds.add_item(tensor)
+
+ res = Tokenizer.binarize(filename, dict, consumer, offset=offset, end=end)
+ ds.finalize(dataset_dest_file(args, output_prefix, lang, 'idx'))
+ return res
+
+def dataset_dest_prefix(args, output_prefix, lang):
+ base = f'{args.destdir}/{output_prefix}'
+ lang_part = f'.{args.source_lang}-{args.target_lang}.{lang}' if lang is not None else ''
+ return f'{base}{lang_part}'
+
+
+def dataset_dest_file(args, output_prefix, lang, extension):
+ base = dataset_dest_prefix(args, output_prefix, lang)
+ return f'{base}.{extension}'
+
+
if __name__ == '__main__':
parser = get_parser()
args = parser.parse_args()

original_message: Parallel preprocessing


message: Add back secondary set

diff:
@@ -45,7 +45,7 @@ class TranslationTask(FairseqTask):
@staticmethod
def add_args(parser):
"""Add task-specific arguments to the parser."""
- parser.add_argument('data', help='path to data directory')
+ parser.add_argument('data', nargs='+', help='path(s) to data directorie(s)')
parser.add_argument('-s', '--source-lang', default=None, metavar='SRC',
help='source language')
parser.add_argument('-t', '--target-lang', default=None, metavar='TARGET',
@@ -80,13 +80,13 @@ class TranslationTask(FairseqTask):
# find language pair automatically
if args.source_lang is None or args.target_lang is None:
- args.source_lang, args.target_lang = data_utils.infer_language_pair(args.data)
+ args.source_lang, args.target_lang = data_utils.infer_language_pair(args.data[0])
if args.source_lang is None or args.target_lang is None:
raise Exception('Could not infer language pair, please provide it explicitly')
# load dictionaries
- src_dict = Dictionary.load(os.path.join(args.data, 'dict.{}.txt'.format(args.source_lang)))
- tgt_dict = Dictionary.load(os.path.join(args.data, 'dict.{}.txt'.format(args.target_lang)))
+ src_dict = Dictionary.load(os.path.join(args.data[0], 'dict.{}.txt'.format(args.source_lang)))
+ tgt_dict = Dictionary.load(os.path.join(args.data[0], 'dict.{}.txt'.format(args.target_lang)))
assert src_dict.pad() == tgt_dict.pad()
assert src_dict.eos() == tgt_dict.eos()
assert src_dict.unk() == tgt_dict.unk()
@@ -102,8 +102,8 @@ class TranslationTask(FairseqTask):
split (str): name of the split (e.g., train, valid, test)
"""
- def split_exists(split, src, tgt, lang):
- filename = os.path.join(self.args.data, '{}.{}-{}.{}'.format(split, src, tgt, lang))
+ def split_exists(split, src, tgt, lang, data_path):
+ filename = os.path.join(data_path, '{}.{}-{}.{}'.format(split, src, tgt, lang))
if self.args.raw_text and IndexedRawTextDataset.exists(filename):
return True
elif not self.args.raw_text and IndexedInMemoryDataset.exists(filename):
@@ -120,29 +120,35 @@ class TranslationTask(FairseqTask):
src_datasets = []
tgt_datasets = []
+ data_paths = self.args.data
+
+ for data_path in data_paths:
for k in itertools.count():
split_k = split + (str(k) if k > 0 else '')
# infer langcode
src, tgt = self.args.source_lang, self.args.target_lang
- if split_exists(split_k, src, tgt, src):
- prefix = os.path.join(self.args.data, '{}.{}-{}.'.format(split_k, src, tgt))
- elif split_exists(split_k, tgt, src, src):
- prefix = os.path.join(self.args.data, '{}.{}-{}.'.format(split_k, tgt, src))
+ if split_exists(split_k, src, tgt, src, data_path):
+ prefix = os.path.join(data_path, '{}.{}-{}.'.format(split_k, src, tgt))
+ elif split_exists(split_k, tgt, src, src, data_path):
+ prefix = os.path.join(data_path, '{}.{}-{}.'.format(split_k, tgt, src))
else:
if k > 0:
break
else:
- raise FileNotFoundError('Dataset not found: {} ({})'.format(split, self.args.data))
+ raise FileNotFoundError('Dataset not found: {} ({})'.format(split, data_path))
src_datasets.append(indexed_dataset(prefix + src, self.src_dict))
tgt_datasets.append(indexed_dataset(prefix + tgt, self.tgt_dict))
- print('| {} {} {} examples'.format(self.args.data, split_k, len(src_datasets[-1])))
+ print('| {} {} {} examples'.format(data_path, split_k, len(src_datasets[-1])))
if not combine:
break
+
+
+
assert len(src_datasets) == len(tgt_datasets)
if len(src_datasets) == 1:

original_message: Add back secondary set


diff:
@@ -4,7 +4,7 @@ This tutorial will walk you through pretraining RoBERTa over your own data.
### 1) Preprocess the data
-Data should be preprocessed following the [language modeling format](/examples/language_model).
+Data should be preprocessed following the [language modeling format](/examples/language_model), i.e. each document should be separated by an empty line (only useful with `--sample-break-mode complete_doc`). Lines will be concatenated as a 1D text stream during training.
We'll use the [WikiText-103 dataset](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/)
to demonstrate how to preprocess raw text data with the GPT-2 BPE. Of course

original_message:
Explain the language modelling format in RoBERTa pretraining readme

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/1174

Differential Revision: D17627767

Pulled By: myleott

fbshipit-source-id: 7b5f77146b8776a5967699e430136039c066c851


message:
Fix load_dataset signature
Summary:
Fix for
Tested with MaskedLMTask.
Pull Request resolved:

diff:
@@ -71,7 +71,7 @@ class MaskedLMTask(FairseqTask):
print('| dictionary: {} types'.format(len(dictionary)))
return cls(args, dictionary)
- def load_dataset(self, split, epoch=0, combine=False):
+ def load_dataset(self, split, epoch=0, combine=False, **kwargs):
"""Load a given dataset split.
Args:

diff:
@@ -110,7 +110,7 @@ class MultiLingualMaskedLMTask(FairseqTask):
smoothed_prob = smoothed_prob / smoothed_prob.sum()
return smoothed_prob
- def load_dataset(self, split, epoch=0, combine=False):
+ def load_dataset(self, split, epoch=0, combine=False, **kwargs):
"""Load a given dataset split.
Args:

original_message:
Fix load_dataset signature (#1281)

Summary:
Fix for https://github.com/pytorch/fairseq/issues/1240
Tested with MaskedLMTask.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1281

Differential Revision: D18051472

fbshipit-source-id: 0aeff60c71489655f5e621349f780ba9cd8c027a


message:
Add whole word masking for SentencepieceBPE
Summary:
Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly.
Pull Request resolved:

diff:
@@ -31,3 +31,13 @@ class SentencepieceBPE(object):
def decode(self, x: str) -> str:
return x.replace(' ', '').replace('\u2581', ' ').strip()
+
+ def is_beginning_of_word(self, x: str) -> bool:
+ if x in ['<unk>', '<s>', '</s>', '<pad>']:
+ # special elements are always considered beginnings
+ # HACK: this logic is already present in fairseq/tasks/masked_lm.py
+ # but these special tokens are also contained in the sentencepiece
+ # vocabulary which causes duplicate special tokens. This hack makes
+ # sure that they are all taken into account.
+ return True
+ return x.startswith('\u2581')

diff:
@@ -108,7 +108,7 @@ class MaskedLMTask(FairseqTask):
# create masked input and targets
if self.args.mask_whole_words:
bpe = encoders.build_bpe(self.args)
- if bpe is not None:
+ assert bpe is not None
def is_beginning_of_word(i):
if i < self.source_dictionary.nspecial:

original_message:
Add whole word masking for SentencepieceBPE (#1292)

Summary:
Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/1292

Differential Revision: D18338307

Pulled By: myleott

fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704


message:
Camembert model and code
Summary:
Check locally that everything works fine.
Model is uploaded to fbaipublicfiles.
I fixed a few inconsistencies in the bpe encoding along the way, e.g. related to
Pull Request resolved:

diff:
@@ -6,6 +6,7 @@ modeling and other text generation tasks.
### What's New:
+- November 2019: [CamemBERT model and code released](examples/camembert/README.md)
- November 2019: [BART model and code released](examples/bart/README.md)
- November 2019: [XLM-R models and code released](examples/xlmr/README.md)
- September 2019: [Nonautoregressive translation code released](examples/nonautoregressive_translation/README.md)

diff:
@@ -8,7 +8,8 @@ RoBERTa iterates on BERT's pretraining procedure, including training the model l
### What's New:
-- November 2019: Multilingual encoder (XLM-RoBERTa) is available [XLM-R](https://github.com/pytorch/fairseq/examples/xlmr).
+- November 2019: French model (CamemBERT) is available [CamemBERT](https://github.com/pytorch/fairseq/tree/master/examples/camembert).
+- November 2019: Multilingual encoder (XLM-RoBERTa) is available [XLM-R](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).
- September 2019: TensorFlow and TPU support via the [transformers library](https://github.com/huggingface/transformers).
- August 2019: RoBERTa is now supported in the [pytorch-transformers library](https://github.com/huggingface/pytorch-transformers).
- August 2019: Added [tutorial for finetuning on WinoGrande](https://github.com/pytorch/fairseq/tree/master/examples/roberta/wsc#roberta-training-on-winogrande-dataset).

diff:
@@ -58,7 +58,7 @@ class RobertaHubInterface(nn.Module):
for s in addl_sentences:
bpe_sentence += (' </s>' if not no_separator else '')
bpe_sentence += ' ' + self.bpe.encode(s) + ' </s>'
- tokens = self.task.source_dictionary.encode_line(bpe_sentence, append_eos=False)
+ tokens = self.task.source_dictionary.encode_line(bpe_sentence, append_eos=False, add_if_not_exist=False)
return tokens.long()
def decode(self, tokens: torch.LongTensor):
@@ -146,8 +146,9 @@ class RobertaHubInterface(nn.Module):
[self.bpe.encode(text_span.rstrip()) for text_span in text_spans]
).strip()
tokens = self.task.source_dictionary.encode_line(
- '<s> ' + text_spans_bpe,
- append_eos=True,
+ '<s> ' + text_spans_bpe + ' </s>',
+ append_eos=False,
+ add_if_not_exist=False,
)
masked_index = (tokens == self.task.mask_idx).nonzero()
@@ -168,6 +169,9 @@ class RobertaHubInterface(nn.Module):
topk_filled_outputs = []
for index, predicted_token_bpe in enumerate(topk_predicted_token_bpe.split(' ')):
predicted_token = self.bpe.decode(predicted_token_bpe)
+ # Quick hack to fix https://github.com/pytorch/fairseq/issues/1306
+ if predicted_token_bpe.startswith('\u2581'):
+ predicted_token = ' ' + predicted_token
if " {0}".format(masked_token) in masked_input:
topk_filled_outputs.append((
masked_input.replace(

diff:
@@ -218,6 +218,29 @@ class XLMRModel(RobertaModel):
return RobertaHubInterface(x['args'], x['task'], x['models'][0])
+@register_model('camembert')
+class CamembertModel(RobertaModel):
+ @classmethod
+ def hub_models(cls):
+ return {
+ 'camembert.v0': 'http://dl.fbaipublicfiles.com/fairseq/models/camembert.v0.tar.gz',
+ }
+
+ @classmethod
+ def from_pretrained(cls, model_name_or_path, checkpoint_file='model.pt', data_name_or_path='.', bpe='sentencepiece', **kwargs):
+ from fairseq import hub_utils
+ x = hub_utils.from_pretrained(
+ model_name_or_path,
+ checkpoint_file,
+ data_name_or_path,
+ archive_map=cls.hub_models(),
+ bpe=bpe,
+ load_checkpoint_heads=True,
+ **kwargs,
+ )
+ return RobertaHubInterface(x['args'], x['task'], x['models'][0])
+
+
class RobertaLMHead(nn.Module):
"""Head for masked language modeling."""

original_message:
Camembert model and code (#904)

Summary:
Check locally that everything works fine.
Model is uploaded to fbaipublicfiles.

I fixed a few inconsistencies in the bpe encoding along the way, e.g. related to https://github.com/pytorch/fairseq/issues/1306..
Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/904

Reviewed By: ngoyal2707

Differential Revision: D18418345

Pulled By: louismartin

fbshipit-source-id: 53acb4d021581968d70430ee9babee07d6573c17


message:
Invoke onNext closure's call() method explicitly
Fix

diff:
@@ -36,4 +36,23 @@ task listImages(type: DockerListImages) {
then:
noExceptionThrown()
}
+
+ def "can list images and handle empty result"() {
+ buildFile << """
+import com.bmuschko.gradle.docker.tasks.image.DockerListImages
+
+task listImages(type: DockerListImages) {
+ showAll = true
+ filters = "project=none-image-match"
+ onNext {
+ }
+}
+"""
+
+ when:
+ build('listImages')
+
+ then:
+ noExceptionThrown()
+ }
}

diff:
@@ -60,7 +60,7 @@ class DockerCopyFileFromContainer extends DockerExistingContainer {
tarStream = containerCommand.exec()
if(onNext) {
- onNext(tarStream)
+ onNext.call(tarStream)
} else {
def hostDestination = new File(hostPath)

diff:
@@ -163,7 +163,7 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
logger.quiet "Created container with ID '$container.id'."
containerId = container.id
if(onNext) {
- onNext(container)
+ onNext.call(container)
}
}

diff:
@@ -27,7 +27,7 @@ class DockerInspectContainer extends DockerExistingContainer {
def container = dockerClient.inspectContainerCmd(getContainerId()).exec()
if (onNext) {
- onNext(container)
+ onNext.call(container)
} else if (responseHandler) {
responseHandler.handle(container)
}

diff:
@@ -104,7 +104,7 @@ class DockerCommitImage extends DockerExistingContainer {
imageId = commitCmd.exec()
logger.quiet "Created image with ID '$imageId'."
if(onNext) {
- onNext(imageId)
+ onNext.call(imageId)
}
}
}

diff:
@@ -26,7 +26,7 @@ class DockerInspectImage extends DockerExistingImage {
logger.quiet "Inspecting image for with ID '${getImageId()}'."
def image = dockerClient.inspectImageCmd(getImageId()).exec()
if(onNext) {
- onNext(image)
+ onNext.call(image)
} else {
responseHandler.handle(image)
}

diff:
@@ -45,9 +45,8 @@ class DockerListImages extends AbstractDockerRemoteApiTask {
}
def images = listImagesCmd.exec()
-
if (onNext) {
- onNext(images)
+ onNext.call(images)
} else if (responseHandler) {
responseHandler.handle(images)
}

diff:
@@ -513,7 +513,7 @@ class DockerThreadContextClassLoader implements ThreadContextClassLoader {
enhancer.setCallback([
invoke: { Object proxy, Method method, Object[] args ->
if ("onNext" == method.name) {
- onNext(args[0])
+ onNext.call(args[0])
}
try {
method.invoke(defaultHandler, args)
@@ -547,7 +547,7 @@ class DockerThreadContextClassLoader implements ThreadContextClassLoader {
enhancer.setCallback([
invoke: { Object proxy, Method method, Object[] args ->
if ("onNext" == method.name) {
- onNext(args[0])
+ onNext.call(args[0])
}
try {
method.invoke(defaultHandler, args)

license: Apache License 2.0

repo: bmuschko/gradle-docker-plugin

original_message:
Invoke onNext closure's call() method explicitly (#368)

Fix #367


message:
Inspect exec container
Remove obsolate logging handler

change_type: ADD

diff:
+package com.bmuschko.gradle.docker
+
+import org.gradle.testkit.runner.BuildResult
+import spock.lang.Requires
+
+/**
+ * System tests for {@link com.bmuschko.gradle.docker.tasks.container.DockerInspectExecContainer} class.
+ */
+@Requires({ TestPrecondition.DOCKER_SERVER_INFO_URL_REACHABLE })
+class DockerInspectExecContainerFunctionalTest extends AbstractFunctionalTest {
+
+ def "Inspect executed command within running container"() {
+ buildFile << """
+ import com.bmuschko.gradle.docker.tasks.image.DockerPullImage
+ import com.bmuschko.gradle.docker.tasks.container.DockerCreateContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerStartContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerExecContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerInspectExecContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerRemoveContainer
+
+ task pullImage(type: DockerPullImage) {
+ repository = 'alpine'
+ tag = '3.4'
+ }
+
+ task createContainer(type: DockerCreateContainer) {
+ dependsOn pullImage
+ targetImageId { pullImage.repository + ":" + pullImage.tag }
+ cmd = ['sleep','10']
+ }
+
+ task startContainer(type: DockerStartContainer) {
+ dependsOn createContainer
+ targetContainerId { createContainer.getContainerId() }
+ }
+
+ task execContainer(type: DockerExecContainer) {
+ dependsOn startContainer
+ targetContainerId { startContainer.getContainerId() }
+ cmd = ['touch', '/tmp/test.txt']
+ }
+
+ task inspectExec(type: DockerInspectExecContainer) {
+ dependsOn execContainer
+ targetExecId { execContainer.execId }
+ }
+
+ task removeContainer(type: DockerRemoveContainer) {
+ dependsOn inspectExec
+ removeVolumes = true
+ force = true
+ targetContainerId { startContainer.getContainerId() }
+ }
+
+ task workflow {
+ dependsOn removeContainer
+ }
+ """
+
+ expect:
+ BuildResult result = build('workflow')
+ result.output.contains("Exec ID: ")
+ }
+
+ def "Inspect exit code of executed command within running container"() {
+ buildFile << """
+ import com.bmuschko.gradle.docker.tasks.image.DockerPullImage
+ import com.bmuschko.gradle.docker.tasks.container.DockerCreateContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerStartContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerExecContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerInspectExecContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerRemoveContainer
+
+ task pullImage(type: DockerPullImage) {
+ repository = 'alpine'
+ tag = '3.4'
+ }
+
+ task createContainer(type: DockerCreateContainer) {
+ dependsOn pullImage
+ targetImageId { pullImage.repository + ":" + pullImage.tag }
+ cmd = ['sleep','10']
+ }
+
+ task startContainer(type: DockerStartContainer) {
+ dependsOn createContainer
+ targetContainerId { createContainer.getContainerId() }
+ }
+
+ task execContainer(type: DockerExecContainer) {
+ dependsOn startContainer
+ targetContainerId { startContainer.getContainerId() }
+ cmd = ['test', '-e', '/not_existing_file']
+ }
+
+ task inspectExec(type: DockerInspectExecContainer) {
+ dependsOn execContainer
+ targetExecId { execContainer.execId }
+ onNext { r ->
+ if(r.exitCode) {
+ throw new GradleException("Docker container exec failed with exit code: " + r.exitCode)
+ }
+ }
+ }
+
+ task removeContainer(type: DockerRemoveContainer) {
+ dependsOn inspectExec
+ removeVolumes = true
+ force = true
+ targetContainerId { startContainer.getContainerId() }
+ }
+
+ task workflow {
+ dependsOn removeContainer
+ }
+ """
+
+ expect:
+ BuildResult result = buildAndFail('workflow')
+ result.output.contains("Docker container exec failed with exit code: ")
+ }
+}

diff:
@@ -31,6 +31,11 @@ class DockerExecContainer extends DockerExistingContainer {
@Optional
Boolean attachStderr = true
+ String execId
+
+ DockerExecContainer() {
+ ext.getExecId = { execId }
+ }
@Override
void runRemoteCommand(dockerClient) {
@@ -38,7 +43,9 @@ class DockerExecContainer extends DockerExistingContainer {
def execCallback = onNext ? threadContextClassLoader.createExecCallback(onNext) : threadContextClassLoader.createExecCallback(System.out, System.err)
def execCmd = dockerClient.execCreateCmd(getContainerId())
setContainerCommandConfig(execCmd)
- dockerClient.execStartCmd(execCmd.exec().getId()).withDetach(false).exec(execCallback).awaitCompletion();
+ execId = execCmd.exec().getId()
+ logger.quiet "Exec ID '${execId}'."
+ dockerClient.execStartCmd(execId).withDetach(false).exec(execCallback).awaitCompletion()
}
private void setContainerCommandConfig(containerCommand) {

diff:
+package com.bmuschko.gradle.docker.tasks.container
+
+import com.bmuschko.gradle.docker.tasks.AbstractDockerRemoteApiTask
+import org.gradle.api.tasks.Input
+
+/**
+ * Inspects task executed inside container
+ * with {@link DockerExecContainer} command.
+ */
+class DockerInspectExecContainer extends AbstractDockerRemoteApiTask {
+
+ /**
+ * Exec ID used to perform operation. The exec for the provided
+ * ID has to be created and started first.
+ */
+ String execId
+
+ void targetExecId(Closure execId) {
+ conventionMapping.execId = execId
+ }
+
+ @Input
+ String getExecId() {
+ execId
+ }
+
+ @Override
+ void runRemoteCommand(Object dockerClient) {
+ logger.quiet "Inspecting exec with ID '${getExecId()}'."
+ def result = dockerClient.inspectExecCmd(getExecId()).exec()
+ if (onNext) {
+ onNext.call(result)
+ } else {
+ logger.quiet("Exec ID: {}", result.id)
+ logger.quiet("Container ID: {}", result.containerID)
+ logger.quiet("Is running: {}", result.running)
+ logger.quiet("Exit code: {}", result.exitCode)
+ }
+ }
+}

original_message:
Inspect exec container (#362)

Remove obsolate logging handler


message:
Allow to build image with multiple tags
Adds tags property to DockerBuildImage task and deprecates tag.
To preserve backward compatibility if tag is defined, tags are ignored.
Fix

diff:
package com.bmuschko.gradle.docker.tasks.image
import com.bmuschko.gradle.docker.AbstractFunctionalTest
-import com.bmuschko.gradle.docker.TestPrecondition
import org.gradle.testkit.runner.BuildResult
-import spock.lang.Requires
import spock.lang.Unroll
class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
@@ -55,13 +53,26 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
gradleTaskDefinition << [imageCreationWithBuildArgs(), imageCreationWithLabelParameter()]
}
+ def "can build image with multiple tags"() {
+ buildFile << buildImageWithTags()
+
+ when:
+ BuildResult result = build('buildImageWithTags', 'buildImageWithTag')
+
+ then:
+ result.output.contains("Using tags 'test/image:123', 'registry.com:5000/test/image:123' for image.")
+ result.output.contains("Using tag 'test/image:123' for image.")
+ }
+
+
+
private String imageCreation() {
"""
import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
task dockerfile(type: Dockerfile) {
- from 'ubuntu:12.04'
+ from 'alpine'
}
task buildImage(type: DockerBuildImage) {
@@ -78,7 +89,7 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
import com.bmuschko.gradle.docker.tasks.image.DockerInspectImage
task dockerfile(type: Dockerfile) {
- from 'ubuntu:12.04'
+ from 'alpine'
arg('arg1')
arg('arg2')
@@ -106,7 +117,7 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
import com.bmuschko.gradle.docker.tasks.image.DockerInspectImage
task dockerfile(type: Dockerfile) {
- from 'ubuntu:12.04'
+ from 'alpine'
}
task buildImage(type: DockerBuildImage) {
@@ -121,4 +132,27 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
}
"""
}
+
+ private String buildImageWithTags() {
+ """
+ import com.bmuschko.gradle.docker.tasks.image.Dockerfile
+ import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
+
+ task dockerfile(type: Dockerfile) {
+ from 'alpine'
+ }
+
+ task buildImageWithTags(type: DockerBuildImage) {
+ dependsOn dockerfile
+ inputDir = file("build/docker")
+ tags = ['test/image:123', 'registry.com:5000/test/image:123']
+ }
+
+ task buildImageWithTag(type: DockerBuildImage) {
+ dependsOn dockerfile
+ inputDir = file("build/docker")
+ tag = 'test/image:123'
+ }
+ """
+ }
}

diff:
@@ -36,11 +36,20 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
@Optional
File dockerFile
+ /**
+ * Tags for image.
+ */
+ @Input
+ @Optional
+ Set<String> tags = []
+
/**
* Tag for image.
+ * @deprecated use {@link #tags}
*/
@Input
@Optional
+ @Deprecated
String tag
@Input
@@ -98,6 +107,10 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
if(getTag()) {
logger.quiet "Using tag '${getTag()}' for image."
buildImageCmd.withTag(getTag())
+ } else if (getTags()) {
+ def tagListString = getTags().collect {"'${it}'"}.join(", ")
+ logger.quiet "Using tags ${tagListString} for image."
+ buildImageCmd.withTags(getTags())
}
if (getNoCache()) {

original_message:
Allow to build image with multiple tags (#380)

Adds tags property to DockerBuildImage task and deprecates tag.
To preserve backward compatibility if tag is defined, tags are ignored.
Fix #286


message:
Add DockerInspectExecContainer to README
[skip ci]

diff:
@@ -117,6 +117,7 @@ The plugin provides the following custom task types for managing containers:
|link:http://bmuschko.github.io/gradle-docker-plugin/docs/groovydoc/com/bmuschko/gradle/docker/tasks/container/DockerWaitContainer.html[DockerWaitContainer] |Blocks until container for a given id stops.
|link:http://bmuschko.github.io/gradle-docker-plugin/docs/groovydoc/com/bmuschko/gradle/docker/tasks/container/DockerLogsContainer.html[DockerLogsContainer] |Copies the container output to the Gradle process standard out/err.
|link:http://bmuschko.github.io/gradle-docker-plugin/docs/groovydoc/com/bmuschko/gradle/docker/tasks/container/DockerExecContainer.html[DockerExecContainer] |Executes a command within a running container.
+|link:http://bmuschko.github.io/gradle-docker-plugin/docs/groovydoc/com/bmuschko/gradle/docker/tasks/container/DockerInspectExecContainer.html[DockerInspectExecContainer] |Inspects task executed inside container with DockerExecContainer command.
|=======


message:
Use org.ajoberstar.release-opinion plugin for versioning
gradle-git was upgraded to 1.7.1.
Publising to GhPages was refactored to use separate plugin.

diff:
+buildscript {
+ repositories {
+ jcenter()
+ }
+
+ dependencies {
+ classpath "org.ajoberstar:gradle-git:1.7.1"
+ }
+}
+
group = 'com.bmuschko'
-version = '3.0.8'
apply plugin: 'groovy'
apply plugin: 'idea'
-apply from: "$rootDir/gradle/dependencies.gradle"
+apply plugin: 'org.ajoberstar.release-opinion'
apply from: "$rootDir/gradle/test-setup.gradle"
apply from: "$rootDir/gradle/integration-test.gradle"
apply from: "$rootDir/gradle/functional-test.gradle"
apply from: "$rootDir/gradle/additional-artifacts.gradle"
+apply from: "$rootDir/gradle/release.gradle"
apply from: "$rootDir/gradle/publishing.gradle"
apply from: "$rootDir/gradle/documentation.gradle"
-apply from: "$rootDir/gradle/release.gradle"
repositories {
mavenCentral()

change_type: DELETE

diff:
-ext.depVersions = ['gradleGit': '1.4.2']
\ No newline at end of file

diff:
@@ -4,29 +4,16 @@ buildscript {
}
dependencies {
- classpath "org.ajoberstar:gradle-git:$depVersions.gradleGit"
+ classpath "org.ajoberstar:gradle-git-publish:0.2.1"
}
}
-apply plugin: org.ajoberstar.gradle.git.ghpages.GithubPagesPlugin
+apply plugin: org.ajoberstar.gradle.git.publish.GitPublishPlugin
-ext {
- repoPath = 'bmuschko/gradle-docker-plugin'
- ghToken = System.getenv('GH_TOKEN')
-}
-
-githubPages {
- if (ghToken) {
- repoUri = "https://github.com/${repoPath}.git"
- credentials {
- username = ghToken
- password = ''
- }
- } else {
- repoUri = "git@github.com:${repoPath}.git"
- }
+gitPublish {
+ repoUri = 'git@github.com:bmuschko/gradle-docker-plugin.git'
- pages {
+ contents {
from(javadoc.outputs.files) {
into 'docs/javadoc'
}
@@ -36,4 +23,4 @@ githubPages {
}
}
-publishGhPages.dependsOn javadoc, groovydoc
+gitPublishPush.dependsOn javadoc, groovydoc

diff:
+import org.ajoberstar.grgit.Grgit
+import org.ajoberstar.gradle.git.release.opinion.Strategies
+
buildscript {
repositories {
jcenter()
}
dependencies {
- classpath "org.ajoberstar:gradle-git:$depVersions.gradleGit"
+ classpath "org.ajoberstar:gradle-git:1.7.1"
}
}
-import org.ajoberstar.grgit.Grgit
-
ext.repo = Grgit.open(rootDir)
-task createTag {
- description = 'Creates repository tag with current project version.'
- ext.tagName = "v${project.version}"
-
- doLast {
- logger.quiet "Creating tag '$tagName'."
-
- // Remove potentially existing tag
- repo.tag.remove(names: [tagName])
-
- // Create tag
- repo.tag.add {
- name = tagName
- message = "Version ${project.version}"
- }
- }
-}
-
-task pushTag {
- description = 'Pushes tag to remote repository.'
- dependsOn createTag
-
- doLast {
- logger.quiet "Pushing tag '$createTag.tagName' to remote."
+release {
+ grgit = Grgit.open(rootDir)
+ versionStrategy Strategies.FINAL
+ defaultVersionStrategy = Strategies.SNAPSHOT
- repo.push {
- refsOrSpecs = [createTag.tagName]
+ tagStrategy {
+ generateMessage = { version -> "Version ${project.version.version}" }
}
}
-}
-
-task tag {
- dependsOn pushTag
- mustRunAfter bintrayUpload
-}
-task release {
- description = 'Publishes artifacts to Bintray and tags repository with current project version.'
- dependsOn assemble, bintrayUpload, tag
-}
\ No newline at end of file
+tasks.release.dependsOn 'assemble', 'bintrayUpload'


message: Release via Travis stage on Git tag

diff:
@@ -11,8 +11,16 @@ before_install:
jdk:
- oraclejdk8
-script:
- - ./gradlew build -s
+jobs:
+ include:
+ - script: ./gradlew build -s
+ - stage: release
+ - script: skip
+ deploy:
+ provider: script
+ script: ./gradlew release
+ on:
+ tags: true
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock

original_message: Release via Travis stage on Git tag


message:
release task doesn't upload to bintray
Bintray upload is done via Travis stage.

diff:
@@ -11,14 +11,15 @@ before_install:
jdk:
- oraclejdk8
+script: ./gradlew build -s
+
jobs:
include:
- - script: ./gradlew build -s
- stage: release
- script: skip
deploy:
provider: script
- script: ./gradlew release
+ script: ./gradlew assemble bintrayUpload
on:
tags: true

diff:
@@ -22,5 +22,3 @@ release {
generateMessage = { version -> "Version ${project.version.version}" }
}
}
-
-tasks.release.dependsOn 'assemble', 'bintrayUpload'


message: Skip default script in release stage

diff:
@@ -16,7 +16,8 @@ script: ./gradlew build -s
jobs:
include:
- stage: release
- - script: skip
+ install: skip
+ script: skip
deploy:
provider: script
script: ./gradlew assemble bintrayUpload

original_message: Skip default script in release stage


message: Document release process

diff:
+# Introduction
+This document describes the release process designed and implemented for `gradle-docker-plugin` (plugin). Its main purpose is to explain developers and maintainers how to prepare and release new version of this software.
+
+# Tools
+The release process use some external libraries and services described in some details below.
+
+## `gradle-git`
+[`gradle-git`](https://github.com/ajoberstar/gradle-git) Gradle plugin is used to automatically determine project version. `org.ajoberstar.release-opinion` is applied in main [build.gradle](build.gradle#L15) and configured in [gradle/release.gradle](gradle/release.gradle#L16). Please reffer the plugin [documentation](https://github.com/ajoberstar/gradle-git/wiki/Release%20Plugins#how-do-i-use-the-opinion-plugin) for details.
+
+## Travis CI
+[Travis CI](https://travis-ci.com) service is used as [CI/CD](https://en.wikipedia.org/wiki/CI/CD) server. Build and deploy jobs are configured in [.travis.yml](.travis.yml) file. Please reffer the [documentation](https://docs.travis-ci.com/) for details.
+
+## Bintray
+[Bintray](https://bintray.com) service is used to publish plugin versions. With [BintrayPlugin](https://github.com/bintray/gradle-bintray-plugin) artifacts are uploaded to remote reposiotry. Plugin configuration is in [gradle/publishing.gradle](gradle/publishing.gradle) file.
+
+# Workflow
+The release process is automated to some extent. The following steps describe the workflow.
+1. Developer updates `README.md` and `RELEASE_NOTES.md` with new planned version
+2. Developer commits all changes in local working copy
+3. Developer triggers new version release using the following command: `$ ./gradlew release -Prelease.stage=final -Prelease.scope=[SCOPE]`, where `[SCOPE]` can be one of: `major`, `minor`, `patch` and determines which part of the version string `<major>.<minor>.<patch>` will be increased.
+
+4. Gradle builds new version string, creates new tag with it and pushes to the `origin`
+5. Travis detects new tag and triggers a build
+6. Travis [is instructed](.travis.yml#L23) to execute [release stage](https://docs.travis-ci.com/user/build-stages/) when on Git tag
+7. In this stage the following command is executed: `./gradlew assemble bintrayUpload` which assembles plugin binaries (with new version) and uploads them to Bintray (credentials are stored as [secure variables](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings) in Travis).
+
+# Useful links
+* [Semantic Versioning](http://semver.org/)
+* [gradle-git version inference](https://github.com/ajoberstar/gradle-git/wiki/Release%20Plugins#version-inference)
+* [Travis script deployment](https://docs.travis-ci.com/user/deployment/script/)
\ No newline at end of file

original_message: Document release process


message: Publish JavaDoc to gh-pages on release

diff:
@@ -18,7 +18,7 @@ jobs:
script: skip
deploy:
provider: script
- script: ./gradlew -s assemble bintrayUpload
+ script: ./gradlew -s assemble bintrayUpload gitPublishPush
on:
tags: true

diff:
@@ -12,6 +12,7 @@ apply plugin: org.ajoberstar.gradle.git.publish.GitPublishPlugin
gitPublish {
repoUri = 'git@github.com:bmuschko/gradle-docker-plugin.git'
+ branch = 'gh-pages'
contents {
from(javadoc.outputs.files) {

original_message: Publish JavaDoc to gh-pages on release


message: Clarify the release process doc

diff:
@@ -18,10 +18,11 @@ The release process is automated to some extent. The following steps describe th
1. Developer updates `README.md` and `RELEASE_NOTES.md` with new planned version.
2. Developer commits all changes in local working copy.
3. Developer triggers new version release using the following command: `$ ./gradlew release -Prelease.stage=final -Prelease.scope=[SCOPE]`, where `[SCOPE]` can be one of: `major`, `minor`, `patch` and determines which part of the version string `<major>.<minor>.<patch>` will be increased.
-4. Gradle builds new version string, creates new tag with it and pushes to the `origin`.
-5. Travis detects new tag and triggers a build.
-6. Travis [is instructed](.travis.yml#L23) to execute [release stage](https://docs.travis-ci.com/user/build-stages/) when on Git tag.
-7. In this stage [Gradle script](.travis.yml#L21) assembles plugin binaries (with new version) and uploads them to Bintray (credentials are stored as [secure variables](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings) in Travis).
+4. Gradle executes a build on developer's machine which calculates new version string, creates new tag with it and pushes to the `origin`.
+5. When Gradle build is finished, developer's work is done and the rest of the release process is automated.
+6. After push to the `origin`, Travis detects new tag and triggers a build.
+7. Travis [is instructed](.travis.yml#L23) to execute [release stage](https://docs.travis-ci.com/user/build-stages/) when on Git tag.
+8. In this stage [Gradle script](.travis.yml#L21) assembles plugin binaries (with new version) and uploads them to Bintray (credentials are stored as [secure variables](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings) in Travis). Also JavaDoc is published to `gh-pages` branch.
# Useful links
* [Semantic Versioning](http://semver.org/)

original_message: Clarify the release process doc


message: Fix publishing JavaDoc

diff:
@@ -18,7 +18,7 @@ jobs:
script: skip
deploy:
provider: script
- script: ./gradlew -s assemble bintrayUpload gitPublishPush
+ script: ./gradlew -s assemble bintrayUpload gitPublishPush -Dorg.ajoberstar.grgit.auth.username=bmuschko -Dorg.ajoberstar.grgit.auth.password=$GH_TOKEN
on:
tags: true

diff:
@@ -7,6 +7,9 @@ The release process uses some external libraries and services described in detai
## gradle-git
The [`gradle-git`](https://github.com/ajoberstar/gradle-git) Gradle plugin is used to automatically determine the project version. `org.ajoberstar.release-opinion` is applied in the main [build.gradle](build.gradle#L15) and configured in [gradle/release.gradle](gradle/release.gradle#L16). Please refer to the plugin [documentation](https://github.com/ajoberstar/gradle-git/wiki/Release%20Plugins#how-do-i-use-the-opinion-plugin) for more details.
+## gradle-git-publish
+The [`gradle-git-publish`](https://github.com/ajoberstar/gradle-git-publish) Gradle plugin is used to publish the documentation to `gh-pages` branch. It is applied and configured in the [gradle/documentation.gradle](gradle/documentation.gradle) file.
+
## Travis CI
[Travis CI](https://travis-ci.com) service is used as our current [CI/CD](https://en.wikipedia.org/wiki/CI/CD) server. Build and deploy jobs are configured in [.travis.yml](.travis.yml) file. Please refer its [documentation](https://docs.travis-ci.com/) for more details.
@@ -31,7 +34,7 @@ where `[SCOPE]` can be one of: `major`, `minor`, `patch`, and determines which p
5. When Gradle build is finished, developer's work is done and the rest of the release process is automated.
6. After push to the `origin`, Travis detects new tag and triggers a build.
7. Travis [is instructed](.travis.yml#L23) to execute [release stage](https://docs.travis-ci.com/user/build-stages/) when on Git tag.
-8. In this stage [Gradle script](.travis.yml#L21) assembles plugin binaries (with new version) and uploads them to Bintray (credentials are stored as [secure variables](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings) in Travis). Also JavaDoc is published to `gh-pages` branch.
+8. In this stage [Gradle script](.travis.yml#L21) assembles plugin binaries (with new version) and uploads them to Bintray (credentials are stored as [secure variables](https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings) in Travis). Also JavaDoc is published to `gh-pages` branch (GitHub username and [access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) are also stored as secure variables).
# Useful links
* [Semantic Versioning](http://semver.org/)

diff:
@@ -11,7 +11,7 @@ buildscript {
apply plugin: org.ajoberstar.gradle.git.publish.GitPublishPlugin
gitPublish {
- repoUri = 'git@github.com:bmuschko/gradle-docker-plugin.git'
+ repoUri = 'https://github.com/bmuschko/gradle-docker-plugin.git'
branch = 'gh-pages'
contents {

diff:
@@ -11,10 +11,9 @@ buildscript {
}
}
-ext.repo = Grgit.open(rootDir)
+apply plugin: org.ajoberstar.gradle.git.base.GrgitPlugin
release {
- grgit = Grgit.open(rootDir)
versionStrategy Strategies.FINAL
defaultVersionStrategy = Strategies.FINAL
tagStrategy {

original_message: Fix publishing JavaDoc


message: Delete default release.scope and update doc

diff:
@@ -21,15 +21,10 @@ The release process is automated to some extent. The following steps describe th
1. Developer updates `RELEASE_NOTES.md` with new planned version.
2. Developer commits all changes in local working copy.
3. Developer triggers new version release using the following command:
-```
-./gradlew release -Prelease.stage=final
-```
-`release` task will use the default value for `release.scope` parameter which is `patch`. It is defined in the [`gradle.properties`](gradle.properties) file. If necessary, developer can override it by executing the following command:
```
./gradlew release -Prelease.stage=final -Prelease.scope=[SCOPE]
```
-where `[SCOPE]` can be one of: `major`, `minor`, `patch`, and determines which part of the version string `<major>.<minor>.<patch>` will be increased.
-
+where `[SCOPE]` can be one of: `major`, `minor` or `patch`, and determines which part of the version string `<major>.<minor>.<patch>` will be increased.
4. Gradle executes a build on developer's machine which calculates new version string, creates new tag with it and pushes to the `origin`.
5. When Gradle build is finished, developer's work is done and the rest of the release process is automated.
6. After push to the `origin`, Travis detects new tag and triggers a build.

diff:
-
-# Can take the form of either `major`, `minor`, or `patch`
-release.scope=patch
-
-# Defined here for documentation purposes. When ready
-# to do a release the user should pass this along on
-# the command line for a 1 time execution.
-#
-# release.stage=final
-


message: Restore default version strategy to SNAPSHOT

diff:
@@ -15,7 +15,7 @@ apply plugin: org.ajoberstar.gradle.git.base.GrgitPlugin
release {
versionStrategy Strategies.FINAL
- defaultVersionStrategy = Strategies.FINAL
+ defaultVersionStrategy = Strategies.SNAPSHOT
tagStrategy {
generateMessage = { version -> "Version ${project.version}" }
}


message: Fix formatting in RELEASE.md

diff:
@@ -25,6 +25,7 @@ The release process is automated to some extent. The following steps describe th
./gradlew release -Prelease.stage=final -Prelease.scope=[SCOPE]
```
where `[SCOPE]` can be one of: `major`, `minor` or `patch`, and determines which part of the version string `<major>.<minor>.<patch>` will be increased.
+
4. Gradle executes a build on developer's machine which calculates new version string, creates new tag with it and pushes to the `origin`.
5. When Gradle build is finished, developer's work is done and the rest of the release process is automated.
6. After push to the `origin`, Travis detects new tag and triggers a build.

original_message: Fix formatting in RELEASE.md


diff:
@@ -494,6 +494,49 @@ class DockerWorkflowFunctionalTest extends AbstractFunctionalTest {
result.output.contains("Devices : [/dev/sda:/dev/xvda:rwm]")
}
+ def "Can build an image, create a container and set /dev/shm size"() {
+ File imageDir = temporaryFolder.newFolder('images', 'minimal')
+ File dockerFile = createDockerfile(imageDir)
+
+ String uniqueContainerName = createUniqueContainerName()
+
+ buildFile << """
+ import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
+ import com.bmuschko.gradle.docker.tasks.container.DockerCreateContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerInspectContainer
+
+ task buildImage(type: DockerBuildImage) {
+ inputDir = file("${dockerFile.parentFile.path}")
+ tag = "${createUniqueImageId()}"
+ }
+
+ task createContainer(type: DockerCreateContainer) {
+ dependsOn buildImage
+ targetImageId { buildImage.getImageId() }
+ containerName = "${uniqueContainerName}"
+ shmSize = 128000
+ cmd = ['/bin/sh']
+ }
+
+ task inspectContainer(type: DockerInspectContainer) {
+ dependsOn createContainer
+ targetContainerId { createContainer.getContainerId() }
+ onNext { c ->
+ if(c.hostConfig.shmSize != 128000) {
+ throw new GradleException("Invalid ShmSize value!")
+ }
+ }
+ }
+
+ task workflow {
+ dependsOn inspectContainer
+ }
+ """
+
+ expect:
+ build('workflow')
+ }
+
def "Can build an image, create a container and assign a network and alias"() {
File imageDir = temporaryFolder.newFolder('images', 'minimal')
File dockerFile = createDockerfile(imageDir)

diff:
@@ -65,6 +65,32 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
}
+ def "can set /dev/shm size for image build"() {
+ buildFile << buildImageWithShmSize()
+
+ when:
+ build("buildWithShmSize")
+
+ then:
+ noExceptionThrown()
+ }
+
+ private String buildImageWithShmSize() {
+ """
+ import com.bmuschko.gradle.docker.tasks.image.Dockerfile
+ import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
+
+ task dockerfile(type: Dockerfile) {
+ from 'alpine'
+ }
+
+ task buildWithShmSize(type: DockerBuildImage) {
+ dependsOn dockerfile
+ inputDir = file("build/docker")
+ shmSize = 128000
+ }
+ """
+ }
private String imageCreation() {
"""

diff:
@@ -151,6 +151,15 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
@Optional
List<String> devices
+ /**
+ * Size of <code>/dev/shm</code> in bytes.
+ * The size must be greater than 0.
+ * If omitted the system uses 64MB.
+ */
+ @Input
+ @Optional
+ Long shmSize
+
@Internal
String containerId
@@ -326,6 +335,11 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
if(getTty()) {
containerCommand.withTty(getTty())
}
+
+ if(getShmSize() != null) { // 0 is valid input
+ def hostConfig = threadContextClassLoader.createHostConfig([shmSize: getShmSize()])
+ containerCommand.withHostConfig(hostConfig)
+ }
}
static class LogConfig {

diff:
@@ -76,6 +76,15 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
@Optional
Map<String, String> buildArgs = [:]
+ /**
+ * Size of <code>/dev/shm</code> in bytes.
+ * The size must be greater than 0.
+ * If omitted the system uses 64MB.
+ */
+ @Input
+ @Optional
+ Long shmSize
+
/**
* The target Docker registry credentials for building image.
*/
@@ -133,6 +142,10 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
buildImageCmd.withLabels(getLabels())
}
+ if(getShmSize() != null) { // 0 is valid input
+ buildImageCmd.withShmsize(getShmSize())
+ }
+
if (getRegistryCredentials()) {
def authConfig = threadContextClassLoader.createAuthConfig(getRegistryCredentials())
def authConfigurations = threadContextClassLoader.createAuthConfigurations([authConfig])
@@ -140,7 +153,7 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
}
buildArgs.each { arg, value ->
- buildImageCmd = buildImageCmd.withBuildArg(arg, value);
+ buildImageCmd = buildImageCmd.withBuildArg(arg, value)
}
def callback = onNext ? threadContextClassLoader.createBuildImageResultCallback(this.onNext)

original_message: Allow setting /dev/shm size in DockerBuildImage and DockerCreateContainer (#413)


message: Fix Dockerfile task up-to-date logic

diff:
@@ -4,6 +4,7 @@ import com.bmuschko.gradle.docker.AbstractFunctionalTest
import com.bmuschko.gradle.docker.TestConfiguration
import org.gradle.api.GradleException
import org.gradle.testkit.runner.BuildResult
+import org.gradle.testkit.runner.TaskOutcome
import java.nio.file.Files
import java.nio.file.Paths
@@ -194,4 +195,36 @@ task ${DOCKERFILE_TASK_NAME}(type: Dockerfile) {
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com
"""
}
+
+ def "Dockerfile task can be up-to-date"() {
+ given:
+ buildFile << """
+import com.bmuschko.gradle.docker.tasks.image.Dockerfile
+
+ext.version = project.properties.getOrDefault('version', '1.0')
+
+task ${DOCKERFILE_TASK_NAME}(type: Dockerfile) {
+ instruction 'FROM $TEST_IMAGE_WITH_TAG'
+ instruction { 'MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"' }
+ label([ver: version])
+}
+"""
+ when:
+ def result = build(DOCKERFILE_TASK_NAME)
+
+ then:
+ TaskOutcome.SUCCESS == result.tasks.first().outcome
+
+ when:
+ result = build(DOCKERFILE_TASK_NAME)
+
+ then:
+ TaskOutcome.UP_TO_DATE == result.tasks.first().outcome
+
+ when:
+ result = build(DOCKERFILE_TASK_NAME, "-Pversion=1.1")
+
+ then:
+ TaskOutcome.SUCCESS == result.tasks.first().outcome
+ }
}

diff:
@@ -28,7 +28,7 @@ class Dockerfile extends DefaultTask {
File destFile = project.file("$project.buildDir/docker/Dockerfile")
Dockerfile() {
-
+ inputs.property("instructionsCache", { instructions.collect { it.build() }.join('\n') })
}
@TaskAction

original_message: Fix Dockerfile task up-to-date logic (#433)


diff:
@@ -501,7 +501,7 @@ The plugin provides a set of tasks for your project and preconfigures them with
[options="header"]
|=======
|Task name |Depends On |Type |Description
-|`dockerCopyDistResources` |`distTar` |link:http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/Copy.html[Copy] |Copies the resource files (like the Java application's TAR file) to a temporary directory for image creation.
+|`dockerCopyDistResources` |`distTar` |link:http://www.gradle.org/docs/current/javadoc/org/gradle/api/tasks/Sync.html[Sync] |Copies the resource files (like the Java application's TAR file) to a temporary directory for image creation.
|`dockerDistTar` |`dockerCopyDistResources` |Dockerfile |Creates the Docker image for the Java application.
|`dockerBuildImage` |`dockerDistTar` |DockerBuildImage |Builds the Docker image for the Java application.
|`dockerPushImage` |`dockerBuildImage` |DockerPushImage |Pushes created Docker image to the repository.

diff:
@@ -21,8 +21,9 @@ class DockerJavaApplicationPluginFunctionalTest extends AbstractFunctionalTest {
dockerfile.text ==
"""FROM java
MAINTAINER ${System.getProperty('user.name')}
-ADD ${projectName}-1.0.tar /
-ENTRYPOINT ["/${projectName}-1.0/bin/${projectName}"]
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
+ENTRYPOINT ["/${projectName}/bin/${projectName}"]
EXPOSE 8080
"""
result.output.contains('Author : ')
@@ -54,8 +55,9 @@ EXPOSE 8080
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
-ADD ${projectName}-1.0.tar /
-ENTRYPOINT ["/${projectName}-1.0/bin/${projectName}"]
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
+ENTRYPOINT ["/${projectName}/bin/${projectName}"]
EXPOSE 9090
"""
result.output.contains('Author : Benjamin Muschko "benjamin.muschko@gmail.com"')
@@ -87,8 +89,9 @@ EXPOSE 9090
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
-ADD ${projectName}-1.0.tar /
-ENTRYPOINT ["/${projectName}-1.0/bin/${projectName}"]
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
+ENTRYPOINT ["/${projectName}/bin/${projectName}"]
EXPOSE 9090 8080
"""
result.output.contains('Author : Benjamin Muschko "benjamin.muschko@gmail.com"')
@@ -124,7 +127,8 @@ EXPOSE 9090 8080
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
-ADD ${projectName}-1.0.tar /
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
CMD ["arg1"]
ENTRYPOINT ["bin/run.sh"]
EXPOSE 9090
@@ -158,7 +162,8 @@ EXPOSE 9090
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
-ADD ${projectName}-1.0.tar /
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
EXPOSE 9090
"""
@@ -206,8 +211,9 @@ EXPOSE 9090
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
-ADD ${projectName}-1.0.tar /
-ENTRYPOINT ["/${projectName}-1.0/bin/${projectName}"]
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
+ENTRYPOINT ["/${projectName}/bin/${projectName}"]
EXPOSE 9090
ADD file1.txt /some/dir/file1.txt
ADD file2.txt /other/dir/file2.txt
@@ -219,6 +225,7 @@ ADD file2.txt /other/dir/file2.txt
@Requires({ TestPrecondition.DOCKERHUB_CREDENTIALS_AVAILABLE })
def "Can create image for Java application and push to DockerHub"() {
+ String projectName = temporaryFolder.root.name
createJettyMainClass()
writeBasicSetupToBuildFile()
Properties gradleProperties = TestPrecondition.readDockerHubCredentials()
@@ -253,13 +260,15 @@ ADD file2.txt /other/dir/file2.txt
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER ${System.getProperty('user.name')}
-ADD javaapp-1.0.tar /
-ENTRYPOINT ["/javaapp-1.0/bin/javaapp"]
+ADD javaapp /javaapp
+ADD app-lib/${projectName}-1.0.jar /javaapp/lib/$projectName-1.0.jar
+ENTRYPOINT ["/javaapp/bin/javaapp"]
EXPOSE 8080
"""
}
def "Can create image for Java application and push to private registry"() {
+ String projectName = temporaryFolder.root.name
createJettyMainClass()
writeBasicSetupToBuildFile()
buildFile << """
@@ -282,8 +291,9 @@ EXPOSE 8080
dockerfile.text ==
"""FROM $CUSTOM_BASE_IMAGE
MAINTAINER ${System.getProperty('user.name')}
-ADD javaapp-1.0.tar /
-ENTRYPOINT ["/javaapp-1.0/bin/javaapp"]
+ADD javaapp /javaapp
+ADD app-lib/${projectName}-1.0.jar /javaapp/lib/$projectName-1.0.jar
+ENTRYPOINT ["/javaapp/bin/javaapp"]
EXPOSE 8080
"""
noExceptionThrown()

diff:
@@ -20,10 +20,11 @@ import com.bmuschko.gradle.docker.tasks.image.DockerPushImage
import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import org.gradle.api.Plugin
import org.gradle.api.Project
+import org.gradle.api.distribution.plugins.DistributionPlugin
import org.gradle.api.plugins.ApplicationPlugin
-import org.gradle.api.tasks.Copy
-import org.gradle.api.tasks.bundling.Tar
-import org.gradle.util.ConfigureUtil
+import org.gradle.api.plugins.JavaPlugin
+import org.gradle.api.tasks.Sync
+import org.gradle.jvm.tasks.Jar
/**
* Opinionated Gradle plugin for creating and pushing a Docker image for a Java application.
@@ -43,12 +44,13 @@ class DockerJavaApplicationPlugin implements Plugin<Project> {
DockerJavaApplication dockerJavaApplication = configureExtension(dockerExtension)
project.plugins.withType(ApplicationPlugin) {
- Tar tarTask = project.tasks.getByName(ApplicationPlugin.TASK_DIST_TAR_NAME)
+ Sync installTask = project.tasks.getByName(DistributionPlugin.TASK_INSTALL_NAME)
+ Jar jarTask = project.tasks.getByName(JavaPlugin.JAR_TASK_NAME)
dockerJavaApplication.exec {
- entryPoint { determineEntryPoint(project, tarTask) }
+ entryPoint { determineEntryPoint(project, installTask) }
}
- Dockerfile createDockerfileTask = createDockerfileTask(project, tarTask, dockerJavaApplication)
- Copy copyTarTask = createDistCopyResourcesTask(project, tarTask, createDockerfileTask)
+ Dockerfile createDockerfileTask = createDockerfileTask(project, installTask, jarTask, dockerJavaApplication)
+ Sync copyTarTask = createDistCopyResourcesTask(project, installTask, jarTask, createDockerfileTask)
createDockerfileTask.dependsOn copyTarTask
DockerBuildImage dockerBuildImageTask = createBuildImageTask(project, createDockerfileTask, dockerJavaApplication)
createPushImageTask(project, dockerBuildImageTask)
@@ -66,31 +68,35 @@ class DockerJavaApplicationPlugin implements Plugin<Project> {
return dockerExtension.javaApplication
}
- private Dockerfile createDockerfileTask(Project project, Tar tarTask, DockerJavaApplication dockerJavaApplication) {
+ private Dockerfile createDockerfileTask(Project project, Sync installTask, Jar jarTask, DockerJavaApplication dockerJavaApplication) {
project.task(DOCKERFILE_TASK_NAME, type: Dockerfile) {
description = 'Creates the Docker image for the Java application.'
- dependsOn tarTask
+ dependsOn jarTask
from { dockerJavaApplication.baseImage }
maintainer { dockerJavaApplication.maintainer }
- addFile({ tarTask.archivePath.name }, { '/' })
+ addFile({ installTask.destinationDir.name }, { "/${installTask.destinationDir.name}" })
+ addFile({ "app-lib/${jarTask.archiveName}" }, { "/${installTask.destinationDir.name}/lib/${jarTask.archiveName}" })
instructions << dockerJavaApplication.exec
exposePort { dockerJavaApplication.getPorts() }
- }
+ } as Dockerfile
}
- private Copy createDistCopyResourcesTask(Project project, Tar tarTask, Dockerfile createDockerfileTask) {
- project.task(COPY_DIST_RESOURCES_TASK_NAME, type: Copy) {
+ private Sync createDistCopyResourcesTask(Project project, Sync installTask, Jar jarTask, Dockerfile createDockerfileTask) {
+ project.task(COPY_DIST_RESOURCES_TASK_NAME, type: Sync) {
group = DockerRemoteApiPlugin.DEFAULT_TASK_GROUP
description = "Copies the distribution resources to a temporary directory for image creation."
- dependsOn tarTask
- from { tarTask.archivePath }
+ dependsOn installTask
+ from { installTask.destinationDir.parentFile }
into { createDockerfileTask.destFile.parentFile }
+ exclude "**/lib/${jarTask.archiveName}"
+ into("app-lib") {
+ from jarTask
}
+ } as Sync
}
- private String determineEntryPoint(Project project, Tar tarTask) {
- String installDir = tarTask.archiveName - ".${tarTask.extension}"
- "/$installDir/bin/$project.applicationName".toString()
+ private String determineEntryPoint(Project project, Sync installTask) {
+ "/${installTask.destinationDir.name}/bin/${project.applicationName}".toString()
}
private DockerBuildImage createBuildImageTask(Project project, Dockerfile createDockerfileTask, DockerJavaApplication dockerJavaApplication) {
@@ -99,7 +105,7 @@ class DockerJavaApplicationPlugin implements Plugin<Project> {
dependsOn createDockerfileTask
conventionMapping.inputDir = { createDockerfileTask.destFile.parentFile }
conventionMapping.tag = { determineImageTag(project, dockerJavaApplication) }
- }
+ } as DockerBuildImage
}
private String determineImageTag(Project project, DockerJavaApplication dockerJavaApplication) {

original_message: Allow Docker to cache app libraries dir in DockerJavaApplication plugin (#459)


diff:
@@ -514,6 +514,7 @@ class DockerWorkflowFunctionalTest extends AbstractFunctionalTest {
dependsOn buildImage
targetImageId { buildImage.getImageId() }
containerName = "${uniqueContainerName}"
+ portBindings = ['8080:8080']
shmSize = 128000
cmd = ['/bin/sh']
}
@@ -525,6 +526,9 @@ class DockerWorkflowFunctionalTest extends AbstractFunctionalTest {
if(c.hostConfig.shmSize != 128000) {
throw new GradleException("Invalid ShmSize value!")
}
+ if(!c.hostConfig.portBindings) {
+ throw new GradleException("Invalid port bindings!")
+ }
}
}

diff:
@@ -345,8 +345,7 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
}
if(getShmSize() != null) { // 0 is valid input
- def hostConfig = threadContextClassLoader.createHostConfig([shmSize: getShmSize()])
- containerCommand.withHostConfig(hostConfig)
+ containerCommand.hostConfig.withShmSize(getShmSize())
}
}

original_message: Fix setting shm size when creating container (#480)


diff:
@@ -461,7 +461,7 @@ The plugin defines the following extension properties in the `javaApplication` c
[options="header"]
|=======
|Property name |Type |Default value |Description
-|`baseImage` |String |java |The Docker base image used for Java application.
+|`baseImage` |String |openjdk |The Docker base image used for Java application.
|`exec` |Closure |Create ENTRYPOINT using script from Application plugin |The ENTRYPOINT and/or CMD Dockerfile instructions
|`maintainer` |String |Value of `user.home` system property |The name and email address of the image maintainer.
|`port` |Integer |8080 |The Docker image entry point port used for the Java application (Deprecated)

diff:
@@ -4,7 +4,7 @@ import org.gradle.testkit.runner.BuildResult
import spock.lang.Requires
class DockerJavaApplicationPluginFunctionalTest extends AbstractFunctionalTest {
- public static final CUSTOM_BASE_IMAGE = 'yaronr/openjdk-7-jre'
+ public static final CUSTOM_BASE_IMAGE = 'openjdk:7-alpine'
def "Can create image for Java application with default configuration"() {
String projectName = temporaryFolder.root.name
@@ -19,7 +19,7 @@ class DockerJavaApplicationPluginFunctionalTest extends AbstractFunctionalTest {
File dockerfile = new File(projectDir, 'build/docker/Dockerfile')
dockerfile.exists()
dockerfile.text ==
-"""FROM java
+"""FROM openjdk
MAINTAINER ${System.getProperty('user.name')}
ADD ${projectName} /${projectName}
ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar

diff:
*/
package com.bmuschko.gradle.docker
-import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import com.bmuschko.gradle.docker.tasks.image.Dockerfile.CompositeExecInstruction
class DockerJavaApplication {
- String baseImage = 'java'
+ String baseImage = 'openjdk'
final CompositeExecInstruction exec = new CompositeExecInstruction()
String maintainer = System.getProperty('user.name')
@Deprecated

original_message: Use openjdk as a default image in DockerJavaApplicationPlugin (#528)


message:
Add skipMaintainer to DockerJavaApplication
* Add skipMaintainer to DockerJavaApplication
This change allows user to build an image without MAINTAINER when using DockerJavaApplicationPlugin.
* Update README
Add note about maintainer deprecation.

diff:
@@ -463,7 +463,7 @@ The plugin defines the following extension properties in the `javaApplication` c
|Property name |Type |Default value |Description
|`baseImage` |String |openjdk |The Docker base image used for Java application.
|`exec` |Closure |Create ENTRYPOINT using script from Application plugin |The ENTRYPOINT and/or CMD Dockerfile instructions
-|`maintainer` |String |Value of `user.home` system property |The name and email address of the image maintainer.
+|`maintainer` |String |Value of `user.home` system property |The name and email address of the image maintainer (Deprecated).
|`port` |Integer |8080 |The Docker image entry point port used for the Java application (Deprecated)
|`ports` |Integer[] |[] |The Docker image exposed ports (if provided, this values will override `port` configuration property)
|`tag` |String |<project.group>/<applicationName>:<project.version> |The tag used for the Docker image.

diff:
@@ -332,6 +332,34 @@ EXPOSE 8080
noExceptionThrown()
}
+ def "Can create image without MAINTAINER"() {
+ String projectName = temporaryFolder.root.name
+ writeBasicSetupToBuildFile()
+ writeCustomTasksToBuildFile()
+
+ buildFile << """
+ docker {
+ javaApplication {
+ skipMaintainer = true
+ }
+ }
+ """
+
+ when:
+ build('inspectImage', '-s')
+
+ then:
+ File dockerfile = new File(projectDir, 'build/docker/Dockerfile')
+ dockerfile.exists()
+ dockerfile.text ==
+ """FROM openjdk
+ADD ${projectName} /${projectName}
+ADD app-lib/${projectName}-1.0.jar /$projectName/lib/$projectName-1.0.jar
+ENTRYPOINT ["/${projectName}/bin/${projectName}"]
+EXPOSE 8080
+"""
+ }
+
private void createJettyMainClass() {
File packageDir = temporaryFolder.newFolder('src', 'main', 'java', 'com', 'bmuschko', 'gradle', 'docker', 'application')
File jettyMainClass = new File(packageDir, 'JettyMain.java')

diff:
@@ -20,7 +20,21 @@ import com.bmuschko.gradle.docker.tasks.image.Dockerfile.CompositeExecInstructio
class DockerJavaApplication {
String baseImage = 'openjdk'
final CompositeExecInstruction exec = new CompositeExecInstruction()
+
+ /**
+ * Deprecated as per https://docs.docker.com/engine/deprecated/#maintainer-in-dockerfile
+ * Will be removed in 4.x release.
+ */
+ @Deprecated
String maintainer = System.getProperty('user.name')
+
+ /**
+ * Temporary solution to be able to create image without
+ * MAINTAINER and preserve backward compatibility.
+ * Will be removed in 4.x release.
+ */
+ boolean skipMaintainer
+
@Deprecated
Integer port = 8080
Set<Integer> ports

diff:
@@ -81,6 +81,9 @@ class DockerJavaApplicationPlugin implements Plugin<Project> {
if (dockerJavaApplication.getPorts().length > 0) {
exposePort { dockerJavaApplication.getPorts() }
}
+ if(dockerJavaApplication.skipMaintainer) {
+ instructions.removeAll { it instanceof Dockerfile.MaintainerInstruction }
+ }
}
} as Dockerfile
}

original_message:
Add skipMaintainer to DockerJavaApplication (#529)

* Add skipMaintainer to DockerJavaApplication

This change allows user to build an image without MAINTAINER when using DockerJavaApplicationPlugin.

* Update README

Add note about maintainer deprecation.


diff:
@@ -644,6 +644,48 @@ class DockerWorkflowFunctionalTest extends AbstractFunctionalTest {
result.output.contains('[env]')
}
+ def "Can build an image and create a container with labels"() {
+ File imageDir = temporaryFolder.newFolder('images', 'minimal')
+ File dockerFile = createDockerfile(imageDir)
+
+ String uniqueContainerName = createUniqueContainerName()
+
+ buildFile << """
+ import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
+ import com.bmuschko.gradle.docker.tasks.container.DockerCreateContainer
+ import com.bmuschko.gradle.docker.tasks.container.DockerInspectContainer
+
+ task buildImage(type: DockerBuildImage) {
+ inputDir = file("${dockerFile.parentFile.path}")
+ tag = "${createUniqueImageId()}"
+ }
+
+ task createContainer(type: DockerCreateContainer) {
+ dependsOn buildImage
+ targetImageId { buildImage.getImageId() }
+ containerName = "${uniqueContainerName}"
+ labels = ["test.label1":"aaa", "test.label2": "bbb"]
+ }
+
+ task inspectContainer(type: DockerInspectContainer) {
+ dependsOn createContainer
+ targetContainerId { createContainer.getContainerId() }
+ onNext { c ->
+ if(c.config.labels.size() != 2) {
+ throw new GradleException("Invalid labels size!")
+ }
+ }
+ }
+
+ task workflow {
+ dependsOn inspectContainer
+ }
+ """
+
+ expect:
+ build('workflow')
+ }
+
private File createDockerfile(File imageDir) {
File dockerFile = new File(imageDir, 'Dockerfile')
dockerFile << """

diff:
@@ -167,6 +167,10 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
@Optional
Long shmSize
+ @Input
+ @Optional
+ Map<String, String> labels = [:]
+
@Internal
String containerId
@@ -355,6 +359,10 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
if(getShmSize() != null) { // 0 is valid input
containerCommand.hostConfig.withShmSize(getShmSize())
}
+
+ if(getLabels()) {
+ containerCommand.withLabels(getLabels())
+ }
}
static class LogConfig {

original_message: Allow to define labels in DockerCreateContainer task (#530)


diff:
@@ -37,6 +37,7 @@ class DockerCreateContainerFunctionalTest extends AbstractFunctionalTest {
cmd = ['ifconfig']
macAddress = '02:03:04:05:06:07'
cpuset = '1'
+ labels = ["project.name": "\$project.name"]
}
task startContainer(type: DockerStartContainer) {

diff:
@@ -126,7 +126,7 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
task buildImage(type: DockerBuildImage) {
dependsOn dockerfile
inputDir = file("build/docker")
- buildArgs = ['arg1':'test1', 'arg2':'test2']
+ buildArgs = ['arg1':'test1', 'arg2':'test2', 'arg3': "\$project.name"]
}
task inspectImage(type: DockerInspectImage) {
@@ -149,7 +149,7 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
task buildImage(type: DockerBuildImage) {
dependsOn dockerfile
inputDir = file("build/docker")
- labels = ['label1':'test1', 'label2':'test2']
+ labels = ['label1':'test1', 'label2':'test2', 'label3':"\$project.name"]
}
task inspectImage(type: DockerInspectImage) {
@@ -164,6 +164,8 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
+ project.version = "123"
+
task dockerfile(type: Dockerfile) {
from '$TEST_IMAGE_WITH_TAG'
}
@@ -171,7 +173,7 @@ class DockerBuildImageFunctionalTest extends AbstractFunctionalTest {
task buildImageWithTags(type: DockerBuildImage) {
dependsOn dockerfile
inputDir = file("build/docker")
- tags = ['test/image:123', 'registry.com:5000/test/image:123']
+ tags = ['test/image:123', "registry.com:5000/test/image:\$project.version"]
}
task buildImageWithTag(type: DockerBuildImage) {

diff:
@@ -55,9 +55,11 @@ class DockerListImagesFunctionalTest extends AbstractFunctionalTest {
buildFile << """
import com.bmuschko.gradle.docker.tasks.image.DockerListImages
+ String imageId = "${ImageId}"
+
task listImages(type: DockerListImages) {
showAll = true
- labels = ["setup":"${ImageId}"]
+ labels = ["setup":"\$imageId"]
onNext { images ->
if(!images.every { image -> image.repoTags.contains("${ImageId}:latest") }) {

diff:
@@ -374,7 +374,7 @@ class DockerCreateContainer extends AbstractDockerRemoteApiTask {
}
if(getLabels()) {
- containerCommand.withLabels(getLabels())
+ containerCommand.withLabels(getLabels().collectEntries { [it.key, it.value.toString()] })
}
if(getMacAddress()) {

diff:
@@ -119,7 +119,7 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
} else if (getTags()) {
def tagListString = getTags().collect {"'${it}'"}.join(", ")
logger.quiet "Using tags ${tagListString} for image."
- buildImageCmd.withTags(getTags())
+ buildImageCmd.withTags(getTags().collect { it.toString() }.toSet() )
}
if (getNoCache()) {
@@ -139,7 +139,7 @@ class DockerBuildImage extends AbstractDockerRemoteApiTask implements RegistryCr
}
if (getLabels()) {
- buildImageCmd.withLabels(getLabels())
+ buildImageCmd.withLabels(getLabels().collectEntries { [it.key, it.value.toString()] })
}
if(getShmSize() != null) { // 0 is valid input

diff:
@@ -53,7 +53,7 @@ class DockerListImages extends AbstractDockerRemoteApiTask {
}
if (getLabels()) {
- listImagesCmd.withLabelFilter(getLabels())
+ listImagesCmd.withLabelFilter(getLabels().collectEntries { [it.key, it.value.toString()] })
}
if (getImageName()) {

original_message: Explicitly call toString() on values in maps (#595)


message: Support multi-stage builds in Dockerfile task

diff:
@@ -387,4 +387,64 @@ task ${DOCKERFILE_TASK_NAME}(type: Dockerfile) {
and:
buildAndFail('dockerFile2')
}
+
+ def "supports multi-stage builds"() {
+ buildFile << """
+import com.bmuschko.gradle.docker.tasks.image.Dockerfile
+
+task ${DOCKERFILE_TASK_NAME}(type: Dockerfile) {
+ from '$TEST_IMAGE_WITH_TAG', 'builder'
+ maintainer 'Benjamin Muschko "benjamin.muschko@gmail.com"'
+ copyFile 'http://hsql.sourceforge.net/m2-repo/com/h2database/h2/1.4.184/h2-1.4.184.jar', '/opt/h2.jar'
+ from({'$TEST_IMAGE_WITH_TAG'}, {'prod'})
+ copyFile '/opt/h2.jar', '/opt/h2.jar', 'builder'
+}
+"""
+ when:
+ build(DOCKERFILE_TASK_NAME)
+
+ then:
+ File dockerfile = new File(projectDir, 'build/docker/Dockerfile')
+ dockerfile.exists()
+ dockerfile.text ==
+ """FROM $TEST_IMAGE_WITH_TAG AS builder
+MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
+COPY http://hsql.sourceforge.net/m2-repo/com/h2database/h2/1.4.184/h2-1.4.184.jar /opt/h2.jar
+FROM alpine:3.4 AS prod
+COPY --from=builder /opt/h2.jar /opt/h2.jar
+"""
+ }
+
+
+ def "supports multi-stage builds (lazy evaluation)"() {
+ buildFile << """
+import com.bmuschko.gradle.docker.tasks.image.Dockerfile
+
+ext.buildStageName = ''
+
+task ${DOCKERFILE_TASK_NAME}(type: Dockerfile) {
+ from({'$TEST_IMAGE_WITH_TAG'}, {buildStageName})
+ maintainer 'Benjamin Muschko "benjamin.muschko@gmail.com"'
+ copyFile 'http://hsql.sourceforge.net/m2-repo/com/h2database/h2/1.4.184/h2-1.4.184.jar', '/opt/h2.jar'
+ from({'$TEST_IMAGE_WITH_TAG'}, {'prod'})
+ copyFile({'/opt/h2.jar'}, {'/opt/h2.jar'}, {buildStageName})
+ doFirst {
+ buildStageName = 'builder'
+ }
+}
+"""
+ when:
+ build(DOCKERFILE_TASK_NAME)
+
+ then:
+ File dockerfile = new File(projectDir, 'build/docker/Dockerfile')
+ dockerfile.exists()
+ dockerfile.text ==
+ """FROM $TEST_IMAGE_WITH_TAG AS builder
+MAINTAINER Benjamin Muschko "benjamin.muschko@gmail.com"
+COPY http://hsql.sourceforge.net/m2-repo/com/h2database/h2/1.4.184/h2-1.4.184.jar /opt/h2.jar
+FROM alpine:3.4 AS prod
+COPY --from=builder /opt/h2.jar /opt/h2.jar
+"""
+ }
}

diff:
@@ -112,9 +112,10 @@ class Dockerfile extends DefaultTask {
* subsequent instructions.
*
* @param image Base image name
+ * @param stageName stage name in case of multi-stage builds (default null)
*/
- void from(String image) {
- instructions << new FromInstruction(image)
+ void from(String image, String stageName = null) {
+ instructions << new FromInstruction(image, stageName)
}
/**
@@ -132,9 +133,10 @@ class Dockerfile extends DefaultTask {
* subsequent instructions.
*
* @param image Base image name
+ * @param stageName closure returning stage name in case of multi-stage builds (default null)
*/
- void from(Closure image) {
- instructions << new FromInstruction(image)
+ void from(Closure image, Closure stageName = null) {
+ instructions << new FromInstruction(image, stageName)
}
/**
@@ -276,9 +278,10 @@ class Dockerfile extends DefaultTask {
*
* @param src Source file
* @param dest Destination path
+ * @param stageName stage name in case of multi stage build
*/
- void copyFile(String src, String dest) {
- instructions << new CopyFileInstruction(src, dest)
+ void copyFile(String src, String dest, String stageName = null) {
+ instructions << new CopyFileInstruction(src, dest, stageName)
}
/**
@@ -287,9 +290,10 @@ class Dockerfile extends DefaultTask {
*
* @param src Source file
* @param dest Destination path
+ * @param stageName closure returning stage name in case of multi stage build
*/
- void copyFile(Closure src, Closure dest) {
- instructions << new CopyFileInstruction(src, dest)
+ void copyFile(Closure src, Closure dest, Closure stageName) {
+ instructions << new CopyFileInstruction(src, dest, stageName)
}
/**
@@ -430,8 +434,7 @@ class Dockerfile extends DefaultTask {
String getKeyword() {
if (instruction instanceof String) {
parseKeyword(instruction)
- }
- else if(instruction instanceof Closure) {
+ } else if (instruction instanceof Closure) {
parseKeyword(instruction())
}
}
@@ -444,8 +447,7 @@ class Dockerfile extends DefaultTask {
String build() {
if (instruction instanceof String) {
instruction
- }
- else if(instruction instanceof Closure) {
+ } else if (instruction instanceof Closure) {
instruction()
}
}
@@ -466,8 +468,7 @@ class Dockerfile extends DefaultTask {
String build() {
if (command instanceof String) {
"$keyword $command"
- }
- else if(command instanceof Closure) {
+ } else if (command instanceof Closure) {
"$keyword ${command()}"
}
}
@@ -488,14 +489,12 @@ class Dockerfile extends DefaultTask {
String build() {
if (command instanceof String[]) {
keyword + ' ["' + command.join('", "') + '"]'
- }
- else if(command instanceof Closure) {
+ } else if (command instanceof Closure) {
def evaluatedCommand = command()
if (evaluatedCommand instanceof String) {
keyword + ' ["' + evaluatedCommand + '"]'
- }
- else {
+ } else {
keyword + ' ["' + command().join('", "') + '"]'
}
}
@@ -590,41 +589,70 @@ class Dockerfile extends DefaultTask {
static abstract class FileInstruction implements Instruction {
final Object src
final Object dest
+ final Object flags
- FileInstruction(String src, String dest) {
+ FileInstruction(String src, String dest, String flags = null) {
this.src = src
this.dest = dest
+ this.flags = flags
}
- FileInstruction(Closure src, Closure dest) {
+ FileInstruction(Closure src, Closure dest, Closure flags = null) {
this.src = src
this.dest = dest
+ this.flags = flags
}
@Override
String build() {
+ String keyword = getKeyword()
+ if (flags) {
+ if (flags instanceof String) {
+ keyword += " $flags"
+ } else if (flags instanceof Closure) {
+ keyword += " ${flags()}"
+ }
+ }
if (src instanceof String && dest instanceof String) {
"$keyword $src $dest"
- }
- else if(src instanceof Closure && dest instanceof Closure) {
+ } else if (src instanceof Closure && dest instanceof Closure) {
"$keyword ${src()} ${dest()}"
}
}
}
static class FromInstruction extends StringCommandInstruction {
- FromInstruction(String image) {
+ final Object stageName
+
+ FromInstruction(String image, String stageName = null) {
super(image)
+ this.stageName = stageName
}
- FromInstruction(Closure image) {
+ FromInstruction(Closure image, Closure stageName = null) {
super(image)
+ this.stageName = stageName
}
@Override
String getKeyword() {
"FROM"
}
+
+ @Override
+ String build() {
+ String result = super.build()
+
+ if (stageName) {
+ if (stageName instanceof String) {
+ result += " AS $stageName"
+ } else if (stageName instanceof Closure) {
+ result += " AS ${stageName()}"
+ }
+ }
+
+ result
+ }
}
static class ArgInstruction extends StringCommandInstruction {
@@ -707,14 +735,12 @@ class Dockerfile extends DefaultTask {
String build() {
if (ports instanceof Integer[]) {
"$keyword ${ports.join(' ')}"
- }
- else if(ports instanceof Closure) {
+ } else if (ports instanceof Closure) {
def evaluatedPorts = ports()
if (evaluatedPorts instanceof String || evaluatedPorts instanceof Integer) {
"$keyword ${evaluatedPorts}"
- }
- else {
+ } else {
"$keyword ${evaluatedPorts.join(' ')}"
}
}
@@ -756,12 +782,12 @@ class Dockerfile extends DefaultTask {
}
static class CopyFileInstruction extends FileInstruction {
- CopyFileInstruction(String src, String dest) {
- super(src, dest)
+ CopyFileInstruction(String src, String dest, String stageName = null) {
+ super(src, dest, stageName ? "--from=$stageName" : null)
}
- CopyFileInstruction(Closure src, Closure dest) {
- super(src, dest)
+ CopyFileInstruction(Closure src, Closure dest, Closure stageName = null) {
+ super(src, dest, stageName ? { "--from=${stageName()}" } : null)
}
@Override
@@ -887,6 +913,7 @@ class Dockerfile extends DefaultTask {
void defaultCommand(String... command) {
instructions << new DefaultCommandInstruction(command)
}
+
void defaultCommand(Closure command) {
instructions << new DefaultCommandInstruction(command)
}
@@ -894,6 +921,7 @@ class Dockerfile extends DefaultTask {
void entryPoint(String... entryPoint) {
instructions << new EntryPointInstruction(entryPoint)
}
+
void entryPoint(Closure entryPoint) {
instructions << new EntryPointInstruction(entryPoint)
}


message: Allow to specify additional groups in DockerCreateContainer

diff:
@@ -21,6 +21,27 @@ import org.gradle.testkit.runner.TaskOutcome
class DockerCreateContainerFunctionalTest extends AbstractGroovyDslFunctionalTest {
+ def "can setup additional user groups"() {
+ given:
+ String containerCreationTask = """
+ task createContainer(type: DockerCreateContainer) {
+ dependsOn pullImage
+ targetImageId pullImage.getImageId()
+ cmd = ['groups']
+ groups = ['postgres']
+ }
+ """
+ buildFile <<
+ containerStart(containerCreationTask) <<
+ containerLogAndRemove()
+
+ when:
+ BuildResult result = build('logContainer')
+
+ then:
+ result.output.contains("postgres")
+ }
+
def "can override default MAC address"() {
given:
String containerCreationTask = """

diff:
@@ -48,6 +48,13 @@ class DockerCreateContainer extends DockerExistingImage {
@Optional
final Property<String> user = project.objects.property(String)
+ /**
+ * A list of additional groups that the container process will run as.
+ */
+ @Input
+ @Optional
+ final ListProperty<String> groups = project.objects.listProperty(String)
+
@Input
@Optional
final Property<Boolean> stdinOpen = project.objects.property(Boolean)
@@ -215,6 +222,7 @@ class DockerCreateContainer extends DockerExistingImage {
tty.set(false)
devices.set([])
autoRemove.set(false)
+ groups.set([])
}
@Override
@@ -271,6 +279,10 @@ class DockerCreateContainer extends DockerExistingImage {
containerCommand.withUser(user.get())
}
+ if(groups.getOrNull()) {
+ containerCommand.hostConfig.withGroupAdd(groups.get())
+ }
+
if(stdinOpen.getOrNull()) {
containerCommand.withStdinOpen(stdinOpen.get())
}


message:
Refactor test for additional groups
Inspect created container and check if additional group is returned.

diff:
@@ -33,13 +33,26 @@ class DockerCreateContainerFunctionalTest extends AbstractGroovyDslFunctionalTes
"""
buildFile <<
containerStart(containerCreationTask) <<
- containerLogAndRemove()
+ """
+ ${containerRemove()}
+
+ task inspectContainer(type: DockerInspectContainer) {
+ dependsOn startContainer
+ finalizedBy removeContainer
+ targetContainerId startContainer.getContainerId()
+ onNext { info ->
+ if (info.hostConfig.groupAdd != ['postgres']) {
+ throw new GradleException("Additional group not found!")
+ }
+ }
+ }
+ """
when:
- BuildResult result = build('logContainer')
+ build('inspectContainer')
then:
- result.output.contains("postgres")
+ noExceptionThrown()
}
def "can override default MAC address"() {


message: Document changes related to groups property in DockerCreateContainer

diff:
[discrete]
=== v4.4.0 (TBA)
* Bump **docker-java-shaded** to latest and greatest and decrease wait time for **DockerLiveness** test. - {uri-github}/pull/729[PR 729]
+* Task `DockerCreateContainer` gained option `groups` - {uri-github}/pull/731[Pull Request 731]
[discrete]
=== v4.3.0 (January 12, 2019)

diff:
@@ -50,6 +50,8 @@ class DockerCreateContainer extends DockerExistingImage {
/**
* A list of additional groups that the container process will run as.
+ *
+ * @since 4.4.0
*/
@Input
@Optional


message: Use expect in DockerCreateContainerFunctionalTest

diff:
@@ -48,11 +48,8 @@ class DockerCreateContainerFunctionalTest extends AbstractGroovyDslFunctionalTes
}
"""
- when:
+ expect:
build('inspectContainer')
-
- then:
- noExceptionThrown()
}
def "can override default MAC address"() {