In recent years many new text generation models have been developed while evaluation of text generation remains a considerable challenge.  Currently, the only metric that is able to fully capture the quality of a generated text is human evaluation, which is expensive and time consuming. One of the most used intrinsic evaluation metrics is perplexity. This paper researched the correspondence between perplexity scores and human evaluation of scripts for the TV-show \textit{Friends} generated using OpenAI's GPT-2 model.  This was done by conducting a survey taken by 226 participants that evaluated selected scripts on creativity, realism and coherence.  The survey results revealed that generations with a perplexity value close to that of an actual Friends script perform best on creativity, but score low on realism and coherence.  The most realistic and coherent generations were those with a lower perplexity value, while the worst in all fields were the generations with the highest perplexity value.  The research shows that perplexity is not an adequate measure for the quality of generated TV-show scripts.    ... Read more

Authorship identification is often applied to large documents, but less so to short, everyday sentences. The ability of identifying who said a short line could provide help to chatbots or personal assistants. This research compares performance of TF-IDF and fastText when identifying authorship of short sentences, by applying these feature extraction techniques to the television series Friends' transcripts. TF-IDF outperforms fastText in every measurement, but its performance is only marginally better than randomly guessing the original character, reaching an accuracy of 28 percent when making a distinction between 6 characters. Accuracy increases linearly at the same rate for both techniques as the minimum word count per sentence set on the test data increases. TF-IDF's confidence remains constant as this limit is set on either the test or training data, whereas fastText's confidence decreases and increases, respectively. Cross-entropy loss, however, remains constant for fastText and decreases for TF-IDF as the minimum word count set on the test data increases. ... Read more

Large text corpora used for creating word embeddings (vectors which represent word meanings) often contain a stereotypical gender bias. This unwanted bias is then also present in the word embeddings and in downstream applications in the field of natural language processing. To prevent and reduce this, more knowledge about the gender bias is necessary. This paper will contribute to this by showing how gender bias in word embeddings from Wikipedia develops over time. Quantifying the gender bias over time shows that words in Science and Arts have become more female biased. Family and Career have stereotypical biases towards respectively female and male words, which have steadily decreased since 2006. This provides new insight in what should be done to make Wikipedia more gender neutral and how important the time of writing can be when considering biases in training word embeddings from Wikipedia or from other text corpora. ... Read more