We train encoder-decoder baselines that are available on HuggingFace. We warm-start the model with pretrained BERT checkpoints and finetune it for the task of Text Summarization.