What 300-dimensional Fridges can Tell Us about Language

A lecture by Prof. Dr. Dirk Hovy

Social media has to do with language, that is beyond a doubt. But can we find new insights on German dialects through it? And what has it to do with 300-dimensional fridges? This is what Dirk Hovy, professor at the University of Bocconi in Milan discussed in his lecture in the lecture series “Einblicke in die Digital Humanties”. This blog post will try to summarise the lecture and give prospect how it might be used for theatre studies.

The Digital Turn

People can locate others geographically by the way they are talking. Only by listening to the way people pronounce or use certain words or based on their grammar we can determine where they have grown up or where they live. Therefore, we associate certain dialects[1] with certain areas. Since the end of the 18^th century variational linguists have tried to map the German dialects[2] and the lines or linguistical borders they have drawn are called isoglosses[3]. Traditionally, to draw these maps researchers gathered the linguistic data by talking to people who have lived in an area all their life and therefore are “ortsfeste”[4] speaker. But to find these people is very hard, if they even still exist in our globalized world. So, Dirk Hovy and his colleague Christoph Purschke thought because of social media there already is a lot of data out there. The beauty of it is that it is already published and transcribed in abundance, it is free and one can test hypotheses with it. By automating the dialectological research and combining qualitative and quantitative methods, Hovy and Purschke hoped to compare the similarities of words in the German speaking world (Germany, Austria and Switzerland) and how cities or regions relate to each other in accordance of these similarities.

The Method

By using embeddings, the assumption in the research was that words that have similar meanings will occur in similar contexts. By representing the words as vectors, similar words will have similar vectors. A vector is a point in space, which has coordinates. Transforming the words into vectors is necessary because only then can the words be quantitatively analysed by the computer. But how does this work? That is where the titular fridge comes into play. Hovy uses the metaphor of the two-dimensional front of a fridge and the words as fridge magnets. Imagine that you have a magnet for each word on the front of the fridge. Now, every time a pair of words occur in the same sentence or context, we move the words closer together. If they do not occur in the same sentence of our corpus, we move them further apart. Eventually, the words will have a stable configuration through this quite simple algorithm. Because of all of this happened on the two-dimensional space, each word will have x- and y-coordinates and therefore a vector. Because we looked at the contexts in which a word occurs, similar words which occur in similar contexts will have similar vectors. Unfortunately, only two dimensions do not represent the words well enough. Every time we compare two words, they will either move closer or further apart which will take far too long to become a stable configuration if it ever will. But it is always possible to add another dimension. In their research, Hove and Purschke ended up with 300 dimensions. They evaluated this number of dimensions to be ideal by trial and error – it simply seems to work the best.[5] So, each word is represented in vector with 300 coordinates. This is quite hard to grasp as human because above three dimensions there seems to be no other real-world appliance. But with 300 dimensions says Hovy, not only semantic fields can be properly distinguished but also other representations such as geographical location can be somehow accommodated. To come back to the dialectological research question, the researchers changed the algorithm. Now, if words occur in the same city, they will be moved closer together and respectively, if not apart from each other. The product is capturing the regional distribution of words and can help to compare different linguistic questions through vectors. For example, cities can be compared to each other, so linguistic similarities between regions can be established. Also, words can be compared to each other with a view on the cities they have been written in, which results in local variants. As well, words can be compared to each other to look at meaning distribution. And finally, labels can be attached to word, such as standard language or dialect.

The research

Cities to cities

To find out how similar cities linguistic profiles are Hovy and Purschke analysed 16.8 million posts in 2.3 million threads of 408 German-speaking cities of the social media platform jodel. The advantage of this social media is that people can post fully anonymously, and the posts are only shown in radius of approximately 10 kilometres. Each word was transformed into a vector with 300 dimensions. Because the word-vectors include the information about the city where the words were posted, it is possible to emit the cities as embeddings as well. Therefore the cities are themselves 300-dimensional vectors too. But before they could analyse the data it had to be pre-processed: When you live in a city you talk much more about entities (such as places, people, or organisations) in it. All these entities had to be thrown out of the corpus because they are not really part of the dialect. By using statistical models and machine learning this process is automated but unfortunately it is not 100% accurate. Now the data of the city can be visualized. By reducing the vectors from 300 to 3 through linear algebra and assigning the remaining three coordinates to a RGB-triple each city becomes a colour. These colour representations then again were put on the geographical maps. The size of the dots simply represents the amount of data.

In this case, similar colours represent similar dialects. Interestingly, this visualisation shows the about the same classification of German Dialects as variational linguists did with conventional methods. Now, Hovy and Purschke tried to see if their output aligns with linguistic theory. By running a clustering algorithm each city started out as its own cluster and was then clustered in multiple groups again. These larger clusters were coloured and put on the dialect map of Alfred Lameli. By evaluating the clusters on their homogeneity (are the clusters separated by region as on Lameli’s map) and completeness (are the clusters in a region pure) Hovy and Purschke found out that 16 clusters align the best with Lameli’s map with 77% with retrofitting it with the additional information of the distances between the cities. This is quite astonishing because the researchers took 17 million social media data points and ran it through algorithms to achieve such a high percentage.

Because big cities are always more divers and have more people from different regions and therefore would always be more like other big cities than to the surrounding region, Hovy and Purschke fitted in geographical distance between the data points to accommodate this. Still, sometimes University cities are clustered with other regions because of the majority students come from the same region, such as Würzburg being clustered with Munich because many students in Würzburg come from the Munich/Bavarian area.

Words to cities

By comparing words to cities, you can see the enactment of the local dialect. But having every word and all the cities in the corpus in vectors they can be added up into aggregates which again are vectors. Therefore, it is possible to see the enactment of language on regional, national, or international level. For example, Hovy’s and Purschke’s research showed that in the northern part of Germany the regional enactment of language seems to be standard German. Especially interesting is the fact that the essential words for this enactment seem to be filler words such as “ja gut”, “erstmals”, or “vielleicht”[6].

As well, these aggregates allow to look at the spelling of lexical variations which again mark regionality.

Word properties

Simply focussing on the words themselves, the researchers labelled the words into different categories by first manually labelling the words and then training a machine learning classifier. This model then can classify the labels of the words and predict how certain it is about the assigned labels. Now, this model can be applied to all of the corpus or just certain parts of it such as region or clusters. For example, by country:

This again can then be plotted:

Together with collleagues from the University of Melbourne, Hovy tried to do the same on European Tweets. But the main difficulty was how they could compare words from different languages. For example, “Welcome to Hamburg” in Danish and Swedish sound quite similar but are written quite differently. So, they split the words up in n-grams of three letters or trigrams. This showed the similarities much better between the two greetings. By applying an even grid across Europe, they extracted the geolocated tweets from each cell, chopped them into n-grams and transformed them into vectors. For each of the cells the vectors were aggregated and the edges between the cells smoothed out because if they did not do that, a cell with few tweets only by foreigners for example would have biased the entire regional representation. The vectors then again were turned into RGB-triples and put on the map.

This map represents quite well for example the Germanic and Roman languages in European Countries as well as the mix of it in Great Britain. But this map also has a bias, Eastern European countries which use the Cyrillian Alphabet all have the same colour, because the model seems to be trained primarily for the Roman Alphabet.

Conclusion

Through embeddings dialect continua can be represented but also new insights can be found, as the research have shown. Resources and Styles as well as geolocation and clusters can be researched and are the first step for translating the usually qualitative variational linguistics into a mathematical, quantitative approach. By leaning on the already done qualitative research, this method simultaneously can be controlled and deliver new findings, therefore bridging the existing gap between qualitative and quantitative research.

But how can this approach be used for theatre studies? The German university subject originally has been founded in the beginning of the 20^th century to research the historical aspect of theatre. It has been understood as theatre historiography until the 1960ies when the first theatre scholars started to focus on the analysis of performances. It seems that this has little to do with linguistics. But it only seems like that. If we look for example on how theatre in the German Enlightenment has been re-coined as verbal or literate art or that German theatre studies itself evolved out of German literature studies, we see the intricate connection of language and theatre (studies). So, for example it would be a possibility to analyse German dramas according to Hovy’s method. We would have to vectorise the words in Drama and could compare them to each other according to the locations they originated. We might perhaps find out which regions in which ages had their own style of writing drama. Then, we could look at these styles diachronically. Also, it would be possible to look at how “Bühnendeutsch”[7] is connected to the German Standard as well as how certain playwrights’ dialects influence their way of writing drama. The possibilities seem endless. This seems to be the beauty of DH: Because humanists have to translate their research into a way that the machine understands new approaches and interdisciplinary methods seem to be more tangible to our own subjects.

---

[1] See: Unknown: dialect. merriam-webster.com. https://www.merriam-webster.com/dictionary/dialect, 07.12.2022.

[2] See: Niebaum, Hermann u. Macha, Jürgen: Einführung in die Dialektologie des Deutschen. Tübingen 2006 (= Germanistische Arbeitshefte, Bd. 37), P. 51.

[3] This word stems from ancient Greek: ἴσος ísos = same, equal, identical and γλῶσσα glossa = tongue, language

[4] Hovy, Dirk: What 300-dimensional Fridges can Tell Us about Language. Lecture. In: Ringvorlesung. Einblicke in die Digital Humanities HS 2022. University of Bern, 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022, 00:12:38. It roughly translates to “locally fixed”.

[5] Also, because 300 for vector dimensionality was established in the paper by Tomas Mikolov. (See: Mikolov, Tomas et al: Efficient Estimation of Word Representations in Vector Space. In: arxiv.org, 07.09.2016, https://arxiv.org/pdf/1301.3781.pdf, 27.11.2022.)

[6] This translates into “ok, well”, “firstly”, or “maybe”.

[7] The German Standard for speech on stage.

---

Sources:

Bibliography:

Hovy, Dirk: What 300-dimensional Fridges can Tell Us about Language. Lecture. In: Ringvorlesung. Einblicke in die Digital Humanities HS 2022. University of Bern, 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

Mikolov, Tomas et al: Efficient Estimation of Word Representations in Vector Space. In: arxiv.org, 07.09.2016, https://arxiv.org/pdf/1301.3781.pdf, 27.11.2022.

Niebaum, Hermann u. Macha, Jürgen: Einführung in die Dialektologie des Deutschen. Tübingen 2006 (= Germanistische Arbeitshefte, Bd. 37).

Unknown: dialect. merriam-webster.com. https://www.merriam-webster.com/dictionary/dialect, 07.12.2022.

Illustrations:

illustration 1: Hovy, Dirk u. Purschke, Christian: Visualisation (Folio 22). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

illustration 2: Hovy, Dirk u. Purschke, Christian: 16 Clusters (Folio 31). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

illustration 3: Hovy, Dirk u. Purschke, Christian: Aggregate by Cluster (Folio 36). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

illustration 4: Hovy, Dirk u. Purschke, Christian: Resources by County (Folio 42). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

illustration 5: Hovy, Dirk u. Purschke, Christian: Aggregating Prototypes (Folio 45). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

illustration 6: Hovy, Dirk, Afshin, Rahimi, Baldwin, Tim u. Brooke, Julian: Map It (Folio 53). In: dh.unibe.ch. 14.11.2022, https://www.dh.unibe.ch/studium/vergangene_lehrveranstaltungen/videoaufzeichnungen_vergangener_ringvorlesungen_hs2022/index_ger.html, 07.12.2022.

---

---