The Relevance of ‘Shared Tasks’ in NLP

Even after centuries of studies, we still have very little hard scientific knowledge about natural languages (NLs). Unlike in other branches of engineering, we don’t know the exact physical or mathematical laws which NLs follows, or even whether they do. So, at least for the time being, we can only rely on empirical techniques for solving practical problems in Natural Language Processing (NLP). Even after some general approach seems to hold promise for solving a problem, a lot of practical work remains to be done in refining the methods and in tuning the systems for the best possible performance. This is why once some initial breakthrough has been made, a lot of people have to try the techniques under different conditions to figure out what is the best setup, i.e., the best selection of parameter values, features, etc. What has come to be called a ‘shared task’ is one way of ensuring that this gets done.

Shared tasks are contest like events where many researchers or even developers working on a particular problem or a set of similar problems try to come up with the best systems. All the systems are evaluated on the same data to provide a fair, competition like setting. All the participants also have to submit papers describing their systems. The major goals of a shared task are:

  • To find out what is the state of the art in a specific area
  • To simultaneously advance the state of the art, even if slightly
  • To bring together researchers so that they can interact and perhaps argue and discuss
  • To act as an incentive for the researchers to build proper systems, some of which may become available for use by others

It was in view of this that the NLP Association of India (NLPAI) started conducting an annual event called the NLPAI Machine Learning Contest in which researchers, including students, are invited to participate and compete in solving a specific problem which is considered relevant. Last year the topic of the shared task was Shallow Parsing for South Asian languages. A workshop was also organized as an extension of this event as part of the IJCAI conference, which was held in Hyderabad, India. The topic this year was Named Entity Recognition for South and South East Asian languages. This year’s event will also have an extended version in the form of a workshop as part of the IJCNLP conference, which is also going to be held in Hyderabad, India.

In the context of South Asian languages, conducting a shared task has its own problems. This is because funding for them is usually unlikely. Without funding it is difficult to prepare the reference data which is usually essential for a shared task. Those who have annotated data are often unwilling to share it with others. IIIT has taken a lead in preparing annotated data for various purposes and also sharing it with others. Since the data is prepared under difficult conditions, sometimes there are problems with the data, but let’s hope things will improve. In any case, data with some errors is better than no data.

Another problem is that the number of full time researchers in NLP is quite small in South Asia, which affects the quality of submissions, but the shared tasks are meant to get over this situation by creating awareness and interest.

It needs to be emphasized that the goal is not just to show good performace on the data provided but also to build practically usable systems that perform well in general. This implies that the participants are supposed to go beyong being mere competitors in a contest. And the idea is to go further than just being the first in the race. Participation in a shared task should be a milestone, not the final destination.

I feel compelled to end this write up by saying that shared tasks with focus on South Asia can only succeed if there is collaboration and sharing of resources by researchers working in South Asia. We are still far from that situation.

The IJCNLP NER workshop site is located here.

(This write up was originally written for the NLPAI newsletter called Spandan, but it was taking a lo…ng time, I became impatient and so you find it here)

(Note on 11th March, 2021: Google had started around that time another platform of their own which was supposed to be like their version of Wikipedia, just as they had started their own social media platform called Orkut or something like that. That was the time social media and other large scale digital platforms were coming up everywhere and they seemed to be the future (as they were, but in a different way). So, as an enthusiast, I used to try out many of these platforms at least by creating an account and just ‘checking out’ the look and design and functionality of the platform. I never really ended up using any of them, with the sole exception of this blog. The above writeup may actually have been written very originally to ‘test out’ the Google platform for crowdsourced encyclopaedia, which consisted of what they called (if I remember correctly) ‘knoles’, as in stand alone units of knowledge, just like Wikipedia articles, where also I had created an account, but didn’t contribute much. In other words, it was written like one writes a ‘Hello World’ program in a new programming language. However, since I did and still do believe in the importance of ‘shared tasks’ from the point of view of replicability, the crisis of which has become very evident now in science, I decided to submit it to the NLPAI newsletter and also post it here. What I am trying to say is that I wouldn’t have written it in the same way if it was not just for testing a platform, and would have tried to write it better.)

Author: anileklavya

मैं सांगणिक भाषाविज्ञान (Computational Linguistics) में एक शोधकर्ता हूँ। इसके अलावा मैं पढ़ता हूँ, पढ़ता हूँ, पढ़ता हूँ, और कुछ लिखने की कोशिश भी करता हूँ। हाल ही मैं मैने ज़ेडनेट का हिन्दी संस्करण ( भी शुरू किया है। एक छोटी सी शुरुआत है। उम्मीद करता हूँ और लोग भी इसमें भाग लेंगे और ज़ेडनेट/ज़ेडमैग के सर्वोत्तम लेखों का हिन्दी (जो कि अपने दूसरे रूप उर्दू के साथ करोड़ों लोगों की भाषा है) में अनुवाद किया जा सकेगा।

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.