How do people introduce you?<\/strong><\/p>\n\n\n\n They introduce me as a researcher at the IJS institute. I\u2019m in the last year of my Ph.D. thesis research. I\u2019m mostly working on Natural Language Processing (NLP). NLP is a big field and I\u2019m currently exploring several different areas.<\/p>\n\n\n\n I initially started by automatically profiling text authors by their style of writing – we can detect their age, gender, and psychological properties. I also worked on automatic identification of text readability. We\u2019ve also created a system to detect Alzheimer’s patients based on their writing.<\/p>\n\n\n\n Lately, I\u2019ve been working on automatic keyword extraction and detecting political bias in word usage in media articles. I\u2019m also contributing to research on semantic change – how word usage changes through time.<\/p>\n\n\n\n References to research that Matej is referencing throughout this interview. I encourage you to read them as they\u2019re written in a very clear language.<\/em><\/p>\n\n\n\n Scalable and Interpretable Semantic Change Detection<\/strong><\/a><\/p>\n\n\n\n [..] We propose a novel scalable method for word usage change detection that offers large gains in processing time and significant memory savings while offering the same interpretability and better performance than unscalable methods. We demonstrate the applicability of the proposed method by analyzing a large corpus of news articles about COVID-19<\/em><\/p>\n\n\n\n Zero-Shot Learning for Cross-Lingual News Sentiment Classification<\/strong><\/a><\/p>\n\n\n\n In this paper, we address the task of zero-shot cross-lingual news sentiment classification. Given the annotated dataset of positive, neutral, and negative news in Slovene, the aim is to develop a news classification system that assigns the sentiment category not only to Slovene news, but to news in another language without any training data required. [..]<\/em><\/p>\n\n\n\n Automatic sentiment and viewpoint analysis of Slovenian news corpus on the topic of LGBTIQ+<\/strong><\/a><\/p>\n\n\n\n We conduct automatic sentiment and viewpoint analysis of the newly created Slovenian news corpus containing articles related to the topic of LGBTIQ+ by employing the state-of the-art news sentiment classifier and a system for semantic change detection. The focus is on the differences in reporting between quality news media with long tradition and news media with financial and political connections to SDS, a Slovene right-wing political party. The results suggest that political affiliation of the media can affect the sentiment distribution of articles and the framing of specific LGBTIQ+ specific topics, such as same-sex marriage.<\/em><\/p>\n<\/div><\/div>\n<\/div><\/div>\n<\/div><\/div>\n\n\n\n Can you start by explaining some background about NLP (Natural Language Processing) to start with?<\/strong><\/p>\n\n\n\n As a first step, it\u2019s good to consider how SVM (support vector machine) classifiers and decision tree techniques used for classification work. Very broadly speaking, they operate on a set of manually crafted features extracted from the dataset that you train your model on. Examples of that type of features would be: “number of words in a document” or a “bag of words model” where you put all the words into \u201ca bag\u201d and a classifier learns which words from this bag appear in different documents. If you have a dataset of documents, for which you know into which class they belong to (e.g., a class can be a gender of the author that wrote a specific document), you can train your model on this dataset and then use this model to classify new documents based on how similar these documents are to the ones in the dataset on which the model was trained. The limitation of this approach is that these statistical features do not really take semantic relation between words into account, since they are based on simple frequency-based statistics.<\/p>\n\n\n\n About 10 years ago a different approach was invented using neural networks. What neural networks allow you to do is to work with unstructured datasets because you don\u2019t need to define these features (i.e., classification rules) in advance. You train them by inputing sequences of words and the network learns on itself how often a given word appears closer to another word in a sequence. The information on each word is gathered in a special layer of this neural network, called an embedding layer<\/em> that is basically a vector representation that encodes how a specific word relates to other words. <\/p>\n\n\n\n What\u2019s interesting is that synonyms have a very similar vector representation. This allows you to extract relations between words. <\/p>\n\n\n\n An example of that would be trying to answer: \u201cParis in relation to France\u201d is the same as \u201cBerlin in relation to (what?)\u201d. To solve this question you can take the embedding of Paris, subtract the embedding of France and add embedding of Berlin and you\u2019ll get an embedding as an answer – Germany. This was a big revolution in the field as it allows us to operationalize relations in the context of languages. The second revolution came when they invented transfer learning, a procedure employed for example in the BERT neural network<\/a> that was trained on BookCorpus with 800 million words and English Wikipedia with 2500 million words. <\/p>\n\n\n\n In this procedure, the first thing you want to do is to train a language model. You want the model to predict the next word in a given sequence of words. You can also mask words in a given text and train the neural network to fill the gaps with the correct words. What implicitly happens in such training is that the neural network will learn about semantic relations between words. So if you\u2019re doing this on a large corpus of texts (like billions of words in BERT) you get a model that you can use on a wide variety of general tasks. Because nobody had to label the data to do the training it means that it\u2019s an unsupervised model.<\/p>\n\n\n\n Are you working with special pre-trained datasets?<\/strong><\/p>\n\n\n\n I\u2019m now mostly working with unsupervised methods similar to the BERT model. So what we do is to take that kind of model and do additional fine-tuning on a smaller training set that makes it better suited for that specific research. This approach allowed us to do all of the research that I\u2019m referencing here.<\/p>\n\n\n\n A different research area that doesn\u2019t require additional training is to employ clustering on the embeddings of these neural networks. You can take a corpus of text from the 1960s and another one from the 2000s. We can then compare how usage of specific embeddings (words) compare between these two collections of texts. That\u2019s essentially how we can study how the semantic meaning of words changed in our culture.<\/p>\n\n\n\n Modern neural networks can also produce embedding for each usage of a word, meaning that words with more than one meaning have more than one embedding. This allows you to differentiate between Apple (software company) and apple (fruit). We used this approach when studying how different words connected to COVID changed through time. We generated embeddings for each word appearance in the corpus of news about COVID and clustered these word occurrences into distinct word usages. Two interesting terms that we identified were diamond<\/em> and strain<\/em>. For strain<\/em>, you can see the shift from using it in epidemiological terms (strain virus) to a more economic usage in later months (financial strain).<\/p>\n\n\n\n What we showed with our research is that you can detect changes even across short (monthly) time periods. There\u2019s a limit to how accurately we can identify the difference. It\u2019s often hard even for humans to decide how to label such data. We can usually get close to humane performance by using our unsupervised methods.<\/p>\n\n\n\n Does this work for Non-English languages?<\/strong><\/p>\n\n\n\n You can use the same technology with a non-English language and we\u2019re successfully using it with Slovenian language. In the case of viewpoint analysis of Slovenian news reporting, we\u2019ve discovered a difference in how the word deep<\/em> is used in different context. Mostly because of the deep state<\/em> that became a popular topic in certain publications.<\/p>\n\n\n\n For our LGBTIQ+ research, we can show that certain media avoids using the word marriage<\/em> in the context of LGBTIQ+ reporting and replaces it with terms like domestic partnership<\/em>. They\u2019re also not discussing LGBTIQ+ relationship within the context of terms such as family<\/em>. We can detect the political leaning of the media based on how they write about these topics.<\/p>\n\n\n\n We just started with this research on the Slovenian language so we expect that we\u2019ll have much more to show later in the year.<\/p>\n\n\n\n So far you\u2019ve talked about analysis and understanding of texts. What other research are you doing?<\/strong><\/p>\n\n\n\n We\u2019re working on models for generating texts as part of the Embeddia project<\/a>. The output of this research also works with the Slovenian language.<\/p>\n\n\n\n We\u2019re also investigating if we can transfer embeddings between languages. We have a special version of the BERT neural network that has been trained on 100+ different language Wikipedias. What we\u2019ve found out is that you can take a corpus of texts in the English language, train the model on it to, for example, detect the gender of the author, and then use that same model to predict the gender of the author of some Slovenian text. This approach is called a zero-shot transfer.<\/p>\n\n\n\n How approachable is all this research and knowledge? Do I need a Ph.D. to be able to understand and use your research?<\/strong><\/p>\n\n\n\n It takes students of our graduate school about a year to become productive in this field. The biggest initial hurdle is that you need to learn how to work with neural networks.<\/p>\n\n\n\n Good thing is that we now have very approachable libraries in this field. I\u2019m a big fan of PyTorch<\/a> as it\u2019s well integrated with the Python ecosystem. There\u2019s also TensorFlow<\/a> that\u2019s more popular in the industry and less in research. I found it harder to use for the type of work we\u2019re doing and harder to debug. With PyTorch it takes about a month or two for our students to understand the basics.<\/p>\n\n\n\n In our context, it\u2019s not just about using the existing neural networks and methods. Understanding the science part of our field and how to contribute via independent paper writing and publishing it\u2019s usually about 2 years.<\/p>\n\n\n\n How easy is it to use your research in \u2018real-world\u2019 applications?<\/strong><\/p>\n\n\n\n We have some international media companies that are using our research in the area of automatic keyword extraction from text. We\u2019re helping them with additional tweaking of our models.<\/p>\n\n\n\n Overall we try to publish everything that we do under open access licenses with code and datasets publicly available.<\/p>\n\n\n\n What we don\u2019t do is maintain our work in terms of production code. It\u2019s beyond the scope of research and we don\u2019t have funding to do it. It\u2019s also very time-consuming and it doesn\u2019t help us with our future research. That\u2019s also what I like about scientific research. We get to invent things and we don\u2019t need to maintain and integrate them. We can shift our focus to the next research question.<\/p>\n\n\n\n So in practice, all of our research is available to you but you\u2019ll need to do the engineering work to integrate it with your product.<\/p>\n\n\n\n Let\u2019s shift a bit to your story and how you got into this research. How did you get here?<\/strong><\/p>\n\n\n\n I first graduated in philosophy and sociology in 2011, at the time when Slovenia was still recovering from the financial crisis. While I considered Ph.D. in philosophy I decided that there are not many jobs for philosophers. That\u2019s why I\u2019ve enrolled in a Computer Science degree that offered better job prospects.<\/p>\n\n\n\n During my Computer Science studies, I was also working in different IT startups. I quickly realized that you don\u2019t have a lot of freedom in such an environment. Software engineering was too constrained for me in terms of what kind of work I could do.<\/p>\n\n\n\n After I graduated I took the opportunity to do Erasmus Exchange and I went to University in Spain. In that academic environment, I found the opposite approach. I received a dataset, a very loose description of a problem, and complete freedom to decide on how I\u2019m going to approach and solve the problem.<\/p>\n\n\n\n When I returned to Slovenia I decided to apply to a few different laboratories inside IJS to see if I could continue with academic research. I\u2019ve got a few offers and accepted the offer from the laboratory where I\u2019m working today. <\/p>\n\n\n\n I also decided to focus on NLP and language technologies as I\u2019m still interested in doing philosophical and sociological research. Currently, I have the freedom to explore these topics in my research field without too many constraints. I\u2019m also really enjoying all the conferences and travel that comes with it. Due to the fast-changing nature of my field, all the cutting-edge research is presented at conferences, and publishing in journals is just too slow. It takes over a year to publish a paper but there’s groundbreaking research almost monthly.<\/p>\n\n\n\n How do you see research done at FAANG (Facebook, Amazon, Apple, Netflix, Google) companies? We know that they\u2019re investing a large amount of money into this field and have large research teams.<\/strong><\/p>\n\n\n\n They\u2019re doing a lot of good research. At the same time, they\u2019re also often relying more on having access to a large number of hardware resources that we don\u2019t. This can be both a blessing and a curse. At the moment I don\u2019t see their research being that much better from the findings from universities. Universities are also more incentivized to develop new optimization techniques as they can’t use brute hardware force for their research.<\/p>\n\n\n\n Are you considering working for a FAANG company after your Ph.D.?<\/strong><\/p>\n\n\n\n Not really. I already have a lot of freedom in my research and I can get funding to explore the areas that interest me. If I would work inside a FAANG company I would need to start at the bottom of the hierarchy and also be limited by their research agenda.<\/p>\n\n\n\n I also really like living in Slovenia and I don\u2019t want to relocate to another country. At the same time, I’m excited about potential study\/researchexchanges as I enjoy collaborating with researchers at foreign institutions.<\/p>\n\n\n\n What are some good resources to follow in your field?<\/strong><\/p>\n\n\n\n You can follow the current state of the art at:<\/p>\n\n\n\n Papers describing paradigm shifts in the field of NLP:<\/p>\n\n\n\n![\"\"](\"https:\/\/www.jurecuhalev.com\/blog\/wp-content\/uploads\/2021\/06\/image-1024x426.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.jurecuhalev.com\/blog\/wp-content\/uploads\/2021\/06\/image-1-1024x419.png\")
![\"\"](\"https:\/\/www.jurecuhalev.com\/blog\/wp-content\/uploads\/2021\/06\/image-2-1024x465.png\")