Towards Building ASR Systems for the Next Billion Users

Tahir Javed1,2 Sumanth Doddapaneni2,4 Abhigyan Raman2

Kaushal Santosh Bhogale2 Gowtham Ramesh2,4 Anoop Kunchukuttan2,3

Pratyush Kumar2,3 Mitesh M. Khapra1,2,4∗

1IIT Madras, 2AI4Bharat, 3Microsoft, 4RBCDSAI

Abstract

Recent methods in speech and language tech-nology pretrain very LARGE models whichare fine-tuned for specific tasks. However, thebenefits of such LARGE models are often lim-ited to a few resource rich languages of theworld. In this work, we make multiple con-tributions towards building ASR systems forlow resource languages from the Indian sub-continent. First, we curate 17,000 hours ofraw speech data for 40 Indian languages froma wide variety of domains including education,news, technology, and finance. Second, usingthis raw speech data we pretrain several vari-ants of wav2vec style models for 40 Indianlanguages. Third, we analyze the pretrainedmodels to find key features: codebook vec-tors of similar sounding phonemes are sharedacross languages, representations across lay-ers are discriminative of the language family,and attention heads often pay attention withinsmall local windows. Fourth, we fine-tune thismodel for downstream ASR for 9 languagesand obtain state-of-the-art results on 3 publicdatasets, including on very low-resource lan-guages such as Sinhala and Nepali. Our workestablishes that multilingual pretraining is aneffective strategy for building ASR systemsfor the linguistically diverse speakers of the In-dian subcontinent. Our code, data and modelsare available publicly at IndicWav2Vec and wehope they will help advance research in ASRfor Indic languages.

1 Introduction

The Indian subcontinent is one of the most linguis-tically diverse regions in the world as well as one ofthe most populous regions in the world - little won-der that it is called a ‘subcontinent’. It is home toaround 2 billion people from 7 countries who speak100+ major languages (and thousands of minoritylanguages and dialects) belonging to four major

∗Corresponding author: miteshk@cse.iitm.ac.in

language families1,2. Of these, the Indo-Aryan andDravidian languages are spoken by a large sectionof the population. These language families haveintermingled over a large period of time giving riseto the Indian linguistic area/sprachbund where lan-guages across these families share many features(Emeneau, 1956; Subbarao, 2012; Kunchukuttanand Bhattacharyya, 2020).

Building high-quality ASR models for such alarge and diverse pool of languages is challenging,even if we restrict ourselves to 30 odd languageswhich have more than a million speakers. Manymodern ASR models rely on large amounts of la-beled data for each language to build high-qualityASR systems. Such approaches are expensive andnot scalable, thus limiting the reach of ASR tech-nologies to some languages and a section of thepopulation. In addition to these challenges on avail-ability of labeled data, Indian languages also face aset of common challenges that need to be addressed.Most Indic scripts have a larger character set thanEnglish. The complex inflectional systems of In-dian languages make modelling phonotactics morechallenging. The rich inflectional/agglutinative na-ture of Indian languages results in larger vocabu-lary sizes, presenting challenges to incorporatinglanguage models. At the same time there are oppor-tunities from a unified perspective. Collection ofunlabeled data for pretraining can be undertaken asa joint effort since many sources might be sharedamongst these languages. A largely overlappingphoneme inventory, logically overlapping charactersets and syntactic similarity can help utilize linguis-tic similarity at various levels to build multilingualmodels where transfer learning can be effective.

In this context, it is important to take note ofrecent work that has demonstrated the benefitsof unsupervised pretraining and multilingual fine-tuning to significantly improve ASR quality for

1https://en.wikipedia.org/wiki/Languages_of_South_Asia2https://en.wikipedia.org/wiki/Languages_of_India

arX

iv:2

111.

0394

5v3

[cs

.CL

] 2

2 D

ec 2

021

low-resource languages (Baevski et al., 2020b;Conneau et al., 2020a). In particular, the wav2vecmodel has established two key results for EnglishASR. One, an end-to-end DNN architecture bor-rowing the popular Transformer architecture fromNLP establishes SOTA results. And two, pre-training on a large corpus of data reduces the re-quirement for labeled fine-tuning from hundredsof hours to few hours and to even tens of min-utes. It is worthwhile to explore if these findingsfrom English ASR transfer to Indic ASR, espe-cially given the diversity and above mentionedchallenges with Indic languages. More specifi-cally, would a wav2vec-like model establish SOTAresults on available benchmarks across Indic lan-guages? Further, would large pretraining precludethe need for collecting large amounts of labeleddata? And finally, would pretraining a multilingualmodel across Indian languages provide positivebenefits across these related languages? In orderto answer these questions, we make the followingcontributions:• We curate 17,314 hours of raw audio data forpretraining across 40 languages from 4 languagefamilies, making it one of the largest and mostdiverse collections of Indic language data. Thisdata has been curated in a short time frame frompublic sources which have a permissive license.It indicates that the feasibility of collecting largeamount of pretraining data and further efforts canbe made to significantly expand this collection.• Starting from the wav2vec 2.0 model, we performextensive ablation studies on architecture choices,pretraining and fine-tuning strategies, languagemodels and choice of lexicon to arrive at a trainingand decoding regimen that works well for Indianlanguages.• Our ASR models achieve SOTA performance on9 Indian languages on 3 publicly available bench-marks with small fine-tuning datasets. These re-sults indicate that end-to-end ASR systems basedon multilingual pretraining with the wav2vecmodel hold promise for Indic languages.• Our ablation studies reveal that the accuracy ofthe ASR system on Indic languages sensitively de-pends on the size of the pretraining corpus, amountof labelled data for fine-tuning, and access to task-specific lexicon.

In summary, we establish that the recent ad-vances of pretraining wav2vec models transfer toIndic ASR and achieve SOTA results against mod-

els proposed over multiple years. However, unlikein the reported results of English ASR, we observethat the WER reported for Indic ASR is signifi-cantly higher and sensitively depends on availabil-ity of resources: pretraining corpus, fine-tuningdata, and task-specific language information. Thissuggests that the ASR task on Indic languages re-mains far from being solved and requires modelinnovation and continued efforts on curating re-sources. We publicly release all the artifacts of ourwork to spur further work in the area of Indic ASR.This includes: (a) sources of pretraining data alongwith scripts for their collection and pre-processing,(b) pretraining, fine-tuning and decoding scripts,(c) language models, and (d) our best ASR models.

2 Curating speech data for Indianlanguages

As mentioned earlier, existing wav2vec models forEnglish (Baevski et al., 2020b) and some other lan-guages (Conneau et al., 2020a; Wang et al., 2021a)have been trained using 1000s of hours of curatedraw audio data in these languages. However, forIndian languages there is no such publicly availablerepository of audio data. Hence, as our first task,we curate audio data from online sources such asYouTube.

We hired native speakers in each of the 22 con-stitutionally recognised Indian languages to helpus with this task. These workers were employ-ees of a data collection agency which specializesin creating and curating text and speech data inIndian languages with prior experience of work-ing on similar tasks. The workers were asked todiscover channels, playlists or individual videoson YouTube by using a wide variety of keywordsfrom different domains such as education, news,technology, sports, and finance. Once the contentwas discovered, they were asked to ensure that theaudio was clean, i.e., it did not contain any back-ground music and the content was predominantlyin the target language. Note that some amount ofcode-mixing with English is unavoidable in Indianspeech content but the workers manually checkedthat the audio was largely in the target language.The workers were also instructed to ensure speakerdiversity by not taking a large amount of contentfrom a single channel. Lastly, the workers wereasked to ensure that the video was available undera Creative Commons License. This was importantto ensure that we can freely share the URLs with

Language NoA YT Language NoA YT

assamese 667 176 ladakhi 121 0balti 7 0 lepcha 52 0bengali 740 295 maithili 37 1bhojpuri 62 0 malayalam 625 232bhutia 42 0 manipuri 374 90bodo 53 11 marathi 387 667chhatisgar 73 0 mizo 304 0dogri 606 8 nagamese 210 0garo 63 0 nepali 374 333gojri 197 0 odia 716 302gujarati 375 686 pahari 24 0hindi 353 722 punjabi 562 301english 408 0 rajasthani 134 0jaintia 57 0 sambalpuri 192 0kannada 425 587 sanskrit 152 348karbi 56 0 santali 0 9kashmiri 417 19 sindhi 77 30khasi 85 0 tamil 815 197kokborok 185 0 telugu 395 657konkani 471 28 urdu 359 362

Table 1: Number of hours audio data per language afterapplying the preprocessing steps. Here NoA refers tonewsonair data, YT refers to YouTube data.

the research community, ensuring reproducibilityof our work, although this significantly limited theamount of data we could collect, especially in thelow resource Indian languages. For each videowhich cleared the above checks, the workers wereasked to copy the URL in an excel sheet and men-tion the target language.

Apart from YouTube, we also curated contentfrom newsonair3 which is a radio news channelrun by the Govt. of India and broadcasts news inmultiple Indian languages. We did not need anyof the above manual checks for newsonair asbeing a large, professional news channel, the con-tent there was already clean (e.g., the language ofthe audio was clearly mentioned and the audio fileswere largely free of any background music). Wecrawled their website to get the URLs of audio filesin 40 Indian languages.

Once the URLs were available, we used theyoutube-dl library4 to download the videos fromYouTube. We then used the FFmpeg library5

to extract the audio data from the videos. Fornewsonair, these two steps were not needed aswe could simply download the audio files from theURLs. We noted that existing speech datasets typ-ically contain mono channel audio data sampledat a sampling frequency of 16Khz. However, in

3http://newsonair.com/4https://github.com/tpikonen/youtube-dl5https://ffmpeg.org/

our case, since the data was curated from diversesources, some of the content was not mono channeland the sampling frequency varied from 8kHz to44 kHz. To standardise the data, we once againused the FFmpeg library to (i) upsample (down-sample) the data which was originally sampled ata frequency lesser (greater) than 16 kHz and (ii)reduce the number of audio channels to 1. Wefurther refined the data by removing long silencesin the audio clips using the py-webrtcvad6 librarywhich is a python interface to the popular WebRTCVAD (Voice Activity Detection) module developedby Google7. The VAD algorithm filters out non-speech content and allows us to set an aggressive-ness parameter, which is an integer between 0 and3 (0 is the least aggressive about filtering out non-speech, 3 is the most aggressive). We found thatsetting this parameter to 2 worked best for our data.Next, to avoid including highly noisy content inour data, we used Waveform Amplitude Distribu-tion Analysis (WADA-SNR) (Kim and Stern, 2008)and filtered out audio clips which had a signal-to-noise-ratio (SNR) less than 15 dB. This thresholdwas chosen after experimenting with a subset ofthe audio data. Finally, as is standard practice, wechunked audio files so that the maximum durationof any audio file was only 25 secs.

The final statistics of the data thus obtained ineach language are summarised in Table 1. Theuncompressed size of the data is 1.5 TB. The URLsof all the audio files as well as the scripts used fordownloading, standardising and cleaning the datawill be made publicly available.

3 IndicWav2Vec: A multilingual ASRmodel for Indian Languages

In this section, we describe the methodology fol-lowed for pretraining a wav2vec ASR model for In-dian languages using the raw audio data describedin the previous section. In particular, we describe(i) the model architecture, (ii) the pretraining proce-dure, (iii) the fine-tuning and decoding procedureand (iv) the procedure for rescoring using an exter-nal transformer based language model (LM). Notethat, for all of the above, we closely follow theprocedure described in (Baevski et al., 2020b) forbuilding an English ASR system. However, we pro-vide the details here for the sake of completenessand reproducibility.

6https://github.com/wiseman/py-webrtcvad7https://webrtc.org/

3.1 Model architecture

We use the same architecture as wav2vec 2.0 whichconsists of (i) a feature encoder for encoding rawaudio into a sequence of T latent representations,(ii) a transformer for learning contextualised rep-resentations for each of the T units, and (iii) aquantizer for discretizing the representations learntby the feature encoder to obtain the targets for self-supervised learning. The feature encoder is a mul-tilayered convolutional neural network which takesraw audio signal as input and outputs a sequenceof frames Z = {z1, z2, . . . , zT } where T is thenumber of timesteps and zi ∈ Rd.

The output Z of the encoder is fed to a trans-former based context network which computes acontextualised representation for each of the Tunits in the input as C = {c1, c2, . . . , cT }, whereci is the contextual representation for the i-th input(i.e., zi).

Following standard practice, we augment the zi’swith positional information before feeding them tothe context network using a convolutional layersimilar to that used in (Baevski et al., 2020b; Mo-hamed et al., 2020; Baevski et al., 2020a; Wu et al.,2019).

The final component of the architecture is aquantizer which takes Z as input and discretizesit to a finite set of representations using prod-uct quantization (Jégou et al., 2011). The outputof the quantizer is a sequence of representationsQ = {q1,q2, . . . ,qT } where qi is the quantizedrepresentation for the i-th input (i.e., zi)..

3.2 Pretraining objectives and data sampling

To train the model, we mask a certain fraction ofthe T input representations before feeding themto the context network. However the inputs to thequantizer are not masked since they serve as thetarget for the self-supervised objective. This ob-jective is implemented as minimizing a contrastiveloss (Graves et al., 2006) that involves maximizingthe similarity of the context network’s output to thequantized representation for masked input at timet, while minimizing the similarity with quantizedrepresentations of other masked inputs. In additionto the above contrastive loss, we also use a diversityloss (Baevski et al., 2020b) which ensures betterutilisation of the codebook.Data sampling. While pretraining multilingualmodels involving a large number of languages withdifferent amounts of training data in each language,

it is important to ensure that all languages get afair representation in the training. In particular, lan-guages which account for a larger share of the train-ing data should not dominate the training. We usetemperature based sampling (Arivazhagan et al.,2019; Conneau et al., 2020b; Wang et al., 2021b) toform multilingual batches (Devlin et al., 2019; Con-neau and Lample, 2019) during pretraining. Let Lbe the set of all languages used for pretraining. Foreach language l ∈ L, we draw audio samples froma multinomial distribution pl ∼

(nlN

)α where nl isthe number of hours of pretraining data in languagel, N is the total number of hours of pretraining dataacross all languages, and α ∈ [0, 1]. Smaller thevalue of α, more is the upsampling for the lowerresource languages.

3.3 Fine-tuningOnce the model is pretrained, we fine-tune it forASR using task-specific supervised data. To do so,we add a randomly initialised projection layer ontop of the context network. This layer maps eachd dimensional output of the context network, intoa C dimensional output where C is the size of thevocabulary. We then use the softmax function tocompute a distribution over the vocabulary. In ourcase, the vocabulary contains all the unique char-acters in the language. The model can either bejointly fine-tuned using data from all the languagesor individually fine-tuned for a specific language.We tried both and for multilingual fine-tuning, weconverted the text data into a single script, i.e., theDevanagari script. Though each of these scriptshave their own Unicode codepoint range, it is pos-sible to get a 1-1 mapping between characters inthese different scripts since the Unicode standardaccounts for the similarities between these scripts(Kunchukuttan, 2020). This results in a smaller,compact output vocabulary while enabling bettertransfer across languages (Shetty and Umesh, 2021;Ramesh et al., 2021). We fine-tune the model usingthe standard CTC loss function (Graves et al., 2006;Baevski et al., 2020b,a). Following (Baevski et al.,2020b), we augment data with a modified versionof SpecAugment (Park et al., 2019).

3.4 DecodingTo decode the emissions from the softmax layer, weuse a lexicon and a separately trained word-level n-gram language model. Let y = {y1, y2, . . . , yM}be a candidate sequence. Further, let pAM (y)and pLM (y) be the probability assigned to the se-

quence y by the fine-tuned network and the lan-guage model respectively. We select the optimalword sequence y∗ as follows:

y∗ = argmaxy log pAM (y) + α log pLM (y) + β|y|(1)

where |y| is the length of the sequence and α andβ are hyperparameters. We use an efficient beamsearch decoder (Pratap et al., 2019; Liptchinskyet al., 2019) to search candidates while combin-ing network scores, LM scores and word insertionbonuses.

3.5 RescoringNote that in the above decoding process, we use ann-gram KenLM language model (Heafield, 2011).However, recent advances in language modelinghave shown that transformer based language mod-els perform well. To get the best of both worlds,we optionally use an external transformer basedlanguage model to rescore the n-best hypothesisproduced above (Synnaeve et al., 2019). Specifi-cally, for each output in the n-best list, we com-pute a new score by combining the decoder log-probability, with the weighted log-probability ofan external LM, pELM (y). We then select the bestsequence using the following equation:

y∗ext = argmaxy log pAM (y) + α1 log pLM (y)

+α2 log pELM (y) + β|y|(2)

where |y| is the length of the sequence and α1, α2

and β are hyperparameters.

4 Experimental Setup

In this section we describe (i) the datasets used forour experiments, (ii) the hyperparameters used forpretraining and fine-tuning, and (iii) the corporaand hyperparameters used for training the languagemodels.Pretraining Dataset From the 17k hours of rawaudio data, we use 3% data from each languageas validation data for tracking the per languagevalidation loss during pretraining.Fine-tuning Datasets We experiment with 3 ASRdatasets covering 9 Indian languages. These in-clude the MSR (Microsoft Research) dataset whichwas released as a part of the Low Resource SpeechRecognition Challenge for Indian Languages (Sri-vastava et al., 2018), the MUCS2021 dataset which

was released as a part of the Multilingual and code-switching ASR challenges for low resource Indianlanguages (Diwan et al., 2021), and a subset ofthe OpenSLR dataset (Kjartansson et al., 2018)obtained from the authors of Shetty and Umesh(2021). The train-val-test splits across datasets aresummarised in Table 2.

MSR MUCS OpenSLR

Train Valid Test Train Valid Test Train Valid Test

bn - - - - - - 70.4 4.8 4.8gu 40.0 5.0 5.0 40.0 5.0 5.26 - - -hi - - - 95.05 5.55 5.49 - - -mr - - - 93.89 5 0.67 - - -ne - - - - - - 70.2 5.0 5.0or - - - 94.54 5.49 4.66 - - -si - - - - - - 70.5 4.8 4.8ta 40.0 5.0 4.2 40.0 5.0 4.41 - - -te 40.0 5.0 4.2 40.0 5.0 4.39 - - -

Table 2: Size of train-val-test splits across datasets (inhours)

Pretraining setup We pretrain two variants of themodel, viz., BASE and LARGE. Similar to theBASE model in Baevski et al. (2020b), our BASEmodel contains 7 convolutional layers each with512 channels, strides of (5,2,2,2,2,2,2) and kernelwidths of (10,3,3,3,3,2,2). The BASE model has 12transformer blocks with model dimension 768, andFFN dimension 3072 with 8 attention heads. Simi-larly, same as Baevski et al. (2020b), our LARGEmodel has the same CNN encoder as the BASEmodel and differs only in the transformer archi-tecture. The LARGE model has 24 transformerblocks with model dimension 1024 and FFN di-mension 4096 with 16 attention heads. For boththe architectures, we use G = 2 (codebooks) withV = 320 entries per codebook in the quantizationmodule. Instead of pretraining the models fromscratch we start with the pretrained checkpoint ofthe corresponding (base or large) English wav2vec2.0 model8. We then further pretrain the modelusing the data we curated for Indian languages.As mentioned earlier, to account for the skew inthe amount of data across different languages, weused temperature based sampling. We experimentwith three different values of α ∈ {0.5, 0.7, 1} andfound that 0.7 works best in terms of model accu-racy in distinguishing the correct masked unit fromthe distractor units.

For the BASE model, we crop audio segments to250k samples or 15.6 seconds of audio. We restrict

8https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#pre-trained-models

the max tokens per GPU to 3M and train the modelon 24 A100 GPUs with gradient accumulation of2 steps with the effective batch size of 3.2 hours.We used Adam optimizer with learning rate set to0.0005 and decayed the learning rate polynomiallyafter a warmup for 32k steps. We trained the modelfor 160k steps. For the LARGE model we cropaudio segments to 320k samples or 20 secondsof audio. We restrict the max tokens per GPUto 1.2M and train the model on 24 A100 GPUswith gradient accumulation of 6 steps making theeffective batch size as 3 hours. We again usedthe Adam optimizer but with learning rate set to0.005 and decayed the learning rate polynomiallywithout any warmup. We trained the model for110k steps. We set the random seed to 1 for allexperiments. All other hyper-parameters are setto the default values from the original wav2vec2.0 code base. Our BASE and LARGE modelshave 95M and 317M parameters respectively. Ourmodels are implemented in fairseq (Ott et al., 2019)and we use fp169 operations to speed up trainingand reduce memory usage.Fine-tuning setup During fine-tuning, we updateall the parameters of the network except the param-eters of the convolutional feature encoder. For both,the BASE and the LARGE model, we used Adamoptimiser with a learning rate of 1e-4 and a tri-stagelearning rate schedule: the learning rate is warmedup for first 10% of the steps, then held constantfor the next 40% steps and finally exponentiallydecayed for the remaining steps.

We set the maximum number of frames per GPUto 1M and fine-tune the models on 8 A100 GPUswithout any gradient accumulation, making oureffective batch size as 8M samples or 500 secs.We trained the BASE model for 80k steps andthe LARGE model for 120k steps. For the first200 steps, we only update the parameters in thefinal layer and then update all the parameters in themodel (except those in the feature encoder). As adata augmentation, strategy, we mask the outputsof the feature encoder, similar to SpecAugment(Park et al., 2019). We set the masking probabil-ity to 0.05 and the LayerDrop rate as 0.1 for boththe models. For the other hyper-parameters, de-fault values were taken from the wav2vec 2.0 coderepository. Further we use early stopping with thepatience set to 12 epochs. We used Fairseq (Ottet al., 2019) for our fine-tuning experiments.

9https://github.com/NVIDIA/apex

Language model setup As mentioned earlier, weuse a lexicon-based beam search decoder as im-plemented in Flashlight10 library along with a lan-guage model to obtain predictions. For each lan-guage, we train 6-gram KenLM language model(s)(Heafield, 2011) with pruning for singletons of or-der 5 and doubletons of order 6. We consider threeoptions for the corpus used for training the lan-guage model: (i) the IndicCorp corpus (Kakwaniet al., 2020) which contains 836 M, 1,860 M, 551M, 719 M, 582 M, 674 M, and 107 M tokens in Ben-gali, Hindi, Marathi, Gujarati, Tamil, Telugu andOdia, respectively, (ii) the transcriptions availablein the respective ASR datasets for each language,and (iii) a combination of the earlier two options.Similarly, we considered three choices for the lex-icon: (i) the top 180K most frequent words fromIndicCorp, (ii) the task-specific lexicon providedby the challenge organizers and (iii) combinationof above two. Later, in Section 3, we will showthat the choice of lexicon and training corpus havea significant impact on the performance of the ASRsystem.

We tune hyperparameters α and β in Equation 1on the validation set, using grid search with valuesranging from -4 to +4 and 0 to 5 for α and β, re-spectively. During this grid search, we use a beamsize of 64. Note that α and β are tuned indepen-dently for each language. Once the best value of αand β are found, we use these for the test data andincrease the beam size to 1024.Rescoring setup As mentioned earlier, we alsoconsider using an external Transformer LM forrescoring the top-k hypotheses obtained from thedecoder. For this, we train a BPE-tokenized Trans-former LM using Fairseq toolkit11 with 12 decoderlayers, 16 attention heads, embedding dimensionsize of 1024 and FFN dimension size of 4096. Thedataset used for training Transformer LM was ex-actly the same as that of KenLM with IndicCorpcorpus included, while the vocabulary size wasrestricted to top 50k sub-words. The model wastrained for 160k steps with the Adam optimizer anda tri-stage learning rate scheduler, comprising of16k linearly increasing warm-up steps and a hold-up until 80k steps at learning-rate of 0.0001 afterwhich the learning rate was decreased exponen-tially for the last 80k steps. Using this languagemodel, we rescore the beams obtained from the

10https://github.com/flashlight/flashlight11https://github.com/pytorch/fairseq

MSR MUCS OpenSLR

gu ta te gu hi mr or ta te bn ne si

M0: No pretraining 46.0 37.5 35.5 53.2 48.1 87.1 73.4 44.8 44.7 36.0 78.8 37.0M1: IndicW2Vb (EkStep data) 23.4 24.0 25.8 30.3 18.0 26.5 28.7 29.9 33.2 19.7 14.4 31.0M2: IndicW2Vb (our data) 22.8 23.7 24.9 29.4 17.8 24.3 27.2 29.3 31.9 18.1 13.8 24.3M3: IndicW2Vl (our data) 20.5 22.1 22.9 26.2 16.0 19.3 25.6 27.3 29.3 16.6 11.9 24.8M4: + LMsmall 16.6 14.9 14.4 18.0 16.3 14.8 19.0 25.4 22.4 14.3 13.0 18.6M5: + LMlarge 11.7 13.6 11.0 17.2 14.7 13.8 17.2 25.0 20.5 13.6 13.6 -M6: + augmented lexicon 12.3 15.1 12.4 14.8 10.5 12.2 21.9 20.0 15.2 10.6 9.7 -M7: + Rescoring 11.9 14.8 12.0 14.3 9.5 11.7 20.6 19.5 15.1 10.5 9.4 -

Table 3: Comparison of different choices for pretraining, fine-tuning, and decoding. IndicW2Vb and IndicW2Vl

refer to our base and LARGE models respectively. LMsmall refers to the language model trained using transcriptsfrom the training and validation data and LMlarge refers to the one trained using IndicCorp in addition to thetranscripts.

decoder according to Equation 2. For selecting op-timal values of α and β, we perform grid search inthe range of 0 to 5 and -4 to +4 respectively.

5 Results and Discussions

5.1 Ablation studies on pretrainingWe first discuss the effect of pretraining as well asvarious choices made during pretraining.No pretraining In row 1 of Table 3, we present theresults for an ASR model which has the same archi-tecture as our BASE model but was trained fromscratch (i.e., the weights were not initialised usingpretraining). Comparing this to row M2, which is amodel with the same architecture but is fine-tunedafter pretraining, we see that pretraining signifi-cantly improves the performance. These resultsare consistent with similar results reported for En-glish models in speech (Baevski et al., 2020b) aswell as multilingual models in NLP (Conneau andLample, 2019). In a separate set of experiments(not included in Table 3) we found that pretrain-ing is even more important when the model sizeis large. For example, training the LARGE modelfrom scratch was highly unstable and even after180K steps, the WER on the validation set was sig-nificantly higher as compared to a LARGE modelwhich was initialised using pretraining (82.18 v/s24.62).Role of Pretraining Corpus Gupta et al. (2021)released a pretrained wave2vec model for 23 In-dian languages using 9800 hours of speech data.In contrast, our model uses over 17,000 hours ofpretraining data for 40 languages, with at least 500hours of data in 15 languages. Comparing rowsM2 and M3 of Table 3, we observe that our modeltrained on larger and more diverse data consistentlyimproves over the model reported in Gupta et al.

(2021). In particular, we get an average improve-ment of 1.44 WER across all languages with a max-imum improvement of 6.75 WER for Sinhala anda minimum improvement of 0.09 WER for Hindi.The improvements on Sinhala are noteworthy giventhat Sinhala is not present in the pretraining dataof both the models. We believe that since our pre-training data has more diversity (40 languages asopposed to 23) and a better distribution of pretrain-ing data across languages, it generalises better forlanguages not seen during pretraining. This is alsoevident given that our model performs better onHindi despite a much smaller amount of Hindi pre-training data in our model (1075 v/s 4564 hours).These results establish importance of the larger pre-training dataset released as a part of this work.

Effect of model size Several works (Conneau et al.,2020c; Baevski et al., 2020b) have shown that largemodels pretrained on large amount of pretrainingdata outperform smaller models in downstreamtasks. However, in the context of speech mod-els for Indian languages, the right thresholds fordata size and model size are not known. For exam-ple, with 17,000 hours of pretraining data woulda LARGE model (317M parameters) outperforma BASE model (95M parameters). To answer thisquestion we compare rows M3 and M4 of Table 3which only differ in the model size. We observethat the LARGE model consistently outperformsthe BASE model with an average improvementof 2.08 WER across all languages, a maximumimprovement of 7.22 WER for Marathi and a mini-mum improvement of 1.9 WER for Hindi. Again, aconsistent reduction in WER across languages, in-cluding Hindi which has the largest representationestablishes the value of larger model.

Effect of model initialization As mentioned ear-

MUCS OpenSLR

gu hi mr or ta te bn ne si

Best Models from Table 3 14.3 9.5 11.7 17.2 19.5 15.1 10.5 9.4 18.6M4a: M4 + test lexicon (Oracle) 14.5 10.8 8.6 16.5 15.5 14.9 8.0 5.6 12.1M5a: M5 + test lexicon (Oracle) 13.0 8.6 7.5 14.8 14.2 11.3 8.0 5.6 -

Table 4: Comparison of model performance when the test lexicon is used (oracle setting).

Fine-tuning data ta gu te

1 Hour 38.8 35.9 42.6+LMlarge+aug. lex. 20.6 19.9 20.3

10 Hours 26.5 25.4 28.1+LMlarge+aug. lex. 16.9 15.1 15.2

20 Hours 24.0 23.0 25.4+LMlarge+aug. lex. 16.2 13.8 13.8

40 Hour 22.1 20.9 22.9+LMlarge+aug. lex. 15.5 13.1 12.9

Table 5: Comparison of model performance acrossvarying amounts of fine-tuning data. We used a beamsize of 128 keeping all other hyperparameters same asM6.

lier, for pretraining our models we started with thepretrained English models released by (Baevskiet al., 2020b). This is in line with similar effortsin multilingual NLP where models pretrained for alarger set of languages are initialised with weightsfrom a model pretrained for a smaller set of lan-guages (Tang et al., 2020). To assess if this isneeded for Indian speech models, we pretraineda LARGE model from scratch instead of initialis-ing with the weights from the English model. Wefound this to be very expensive as even after 120ksteps (which corresponds to 95 hours of trainingon 24 GPUs) this model had a loss of 2.17 on thevalidation data as opposed to the English-initialisedmodel which had a loss of 1.82 after just 110k steps.This is an important finding as it suggests that aswe expand to more low resource languages (beyondthe 40 covered in our model), we should be able toleverage the pretrained checkpoints of our modeland save costs on compute for training.

5.2 Ablation studies on fine-tuning anddecoding

The previous section focused on discovering theright choices for pretraining. In this section, wefocus on different choices for fine-tuning and de-coding.Choice of language model As mentioned earlier,language model (LM) and an accompanying lexi-con play a crucial role in decoding the emissions

of the acoustic model. Keeping the lexicon fixed(as extracted from the transcripts for the trainingand validation set), we consider two choices for thelanguage model: (i) a LM trained using only thetranscripts for the training and validation data, and(ii) a LM trained on much larger data from Indic-Corp in addition to the transcripts for the trainingand validation data. Comparing rows M5 and M6of Table 3 we observe that integrating a LM trainedon larger generic data outperforms a LM trained onsmaller task specific data. We also observe the sig-nificant improvement in the WER by integrating aLM (using small data or large data) as compared tothe WER of the acoustic model (row M4). Finally,rescoring the beams with an external TransformerLM, improves WER by an average of 0.6 WER asshown by row M7 of Table 3.

To further investigate the role of language modelin ASR, in Table 5, we experiment with differentsizes of fine-tuning data (1 hour, 10 hours, 20 hours,40 hours). For the 1h dataset, the avg. WER camedown from 39.1 WER to 20.3 WER, a reduction of48.2%, while for the 40h dataset, it reduced from22.0 WER to 13.8 WER, a decrease of 21.4%. Thisshows that the LM plays an important role whenlimited training data is available (which is typicallythe case for Indian languages). These observationslead to the following insight for developing ASRsystems for low resource languages: build betterLMs using (relatively) easily available raw textcorpora in these languages.

Effect of lexicon Having established the impor-tance of using LMs trained on larger data, we nowfix the LM and change the lexicon. In particular, weaugment the lexicon provided by task organizerswith top 180K most frequent words from IndicCorp(Kakwani et al., 2020). The hope is that this aug-mented lexicon may reduce the number of OOVsduring testing. Comparing rows M6 and M7 of Ta-ble 3, we observe that results are a bit mixed with adrop in WER for the MUCS and OpenSLR datasetbut an increase in WER for the MSR dataset. Oneprobable reason for worse WER in case of MSR

MSR MUCS OpenSLR

gu ta te gu hi mr or ta te bn ne

M6: 12.3 15.1 12.4 14.8 10.5 12.2 21.9 20.0 15.2 10.6 9.7M8: + Multi. FT (Single Softmax) 12.8 16.0 12.5 15.5 10.1 30.3 26.8 20.8 15.9 11.0 11.9M9: + Multi. FT (Multi-Softmax) (Joshi et al., 2021) 12.4 15.8 12.5 14.4 10.5 11.7 22.3 21.0 16.0 12.0 13.4

Table 6: Comparison of multilingual fine-tuning strategies

MSR MUCS OpenSLR

gu ta te gu hi mr or ta te bn ne

M6: 12.3 15.1 12.4 14.8 10.5 12.2 21.9 20.0 15.2 10.6 9.7M10: IPL with M6 checkpoint 13.8 21.3 16.5 15.1 14.1 24.2 34.2 21.9 17.1 13.2 13.8

Table 7: Comparison of different Iterative Pseudo-Labeling (IPL) strategies

dataset could be that the lexicon provided by taskorganisers for MSR challenge already containedtokens from test dataset and hence appending itwith extra tokens degrades its quality. We believethat this is an error on the part of the organisers astest data should be excluded while building the lex-icon (but we use it nonetheless to make our resultscomparable with other participants). To investigatethis further, we did an oracle experiment where wefurther augmented the lexicon with words from thetest set for the other datasets. We refer to this as anoracle experiment, because in practice we wouldnot have access to the lexicon from the test set.

We present the oracle experiment results in rowM4a, M5a of Table 4. With the new best models,we see an average improvement of 3.8 WER acrosslanguages, a maximum improvement of 6.5 WERin Sinhala and minimum improvement of 1.9 WERin Hindi. We do not report these numbers in theTable 3 as it would be unfair to compare this withthe other results. This leads to another insight: theright lexicon for the target domain can significantlyhelp in low resource scenarios. Of course, thelexicon should not be taken from the test set butindependently curated.

Monolingual vs multilingual fine-tuning Someworks (Conneau et al., 2020c; Joshi et al., 2021)have shown that jointly fine-tuning a multilingualmodel using the task specific data in all languagesleads to better performance. Motivated by theseresults, we jointly fine-tune the pretrained model bycombining the training data from all the languages.

During fine-tuning, we treat character units fromeach language as a unique token. We call thismodel as Single Softmax, as shown in row M8of Table 6 (since the characters across all the lan-guages are competing with each other in a single

softmax computation). Comparing rows M6 andM8, which only differ in the fine-tuning strategy(monolingual v/s multilingual), we observe thatthe multilingual model gives a similar performanceacross all the languages. This is consistent withthe results in multilingual NLP, where we typicallysee good gains due to a shared encoder (e.g. many-to-one translation) and less gains due to shareddecoder (e.g. one-to-many translation) (Arivazha-gan et al., 2019). One observation is that error rateson Marathi are significantly higher as for many sen-tences the model outputs Odia characters insteadof Devanagari characters. To address this issue, wefollow (Joshi et al., 2021) and use a multi-softmaxlayer, with each language having a separate soft-max layer. The output vocabulary contains uniquetokens from the respective language. During train-ing, each mini-batch consists of data randomly sam-pled from a particular language. We show the re-sults in row M9 of Table 6. We observe that theresults for Marathi are improved significantly com-pared to row M8 while the performance remainsconsistent for other languages. These results areencouraging as they show that a single joint model(with a separate output layer for each language)gives performance which is comparable to that ofindividual models trained for specific languages.Such joint models are preferred as they are easierto maintain and deploy.

Impact of fine-tuning dataset size A significantresult reported in Baevski et al. (2020b) was thatpretraining a model using a large amount of un-labeled data significantly reduces the amount offine-tuning data needed for building ASR models.In particular, they showed that starting with a pre-trained model, using just 10 mins of fine-tuningdata, led to single digit WERs for English. This is

hi gu mr or ta te

Baseline 27.45 25.98 20.41 31.28 35.82 29.35

CSTR 14.33 20.59 15.79 25.34 23.16 21.88BSA 16.59 21.30 15.65 17.81 28.59 25.37EM 17.54 20.11 20.15 19.99 28.52 26.08EkStep 12.24 30.65 39.74 27.10 27.20 22.43Uniphore 22.79 22.79 14.9 29.55 18.8 28.69Lottery 17.81 23.62 58.78 17.74 30.69 27.67IIITH 31.11 26.94 33.8 37.19 35.03 17.00

M5: 14.7 17.2 13.8 17.2 25.0 20.5M6: 10.5 14.8 12.2 21.9 20.0 15.2

Table 8: Comparison of our best models (M5, M6) withthe the top performers from the MUCS 2021 leader-board. Individual best models from the leaderboard areunderlined.

a significant result in the context of low resourcelanguages. To check if the same holds true forIndian languages, we vary the amount of train-ing data provided to the model from 1 to 10 to40 hours. Quite disappointingly, our results as re-ported in Table 5, show that using just 1 hour offine-tuning data leads to much worse performance.Further, we see gains in increasing the data from20 hours to 40 hours, suggesting that adding morefine-tuning data would be beneficial. Lastly, notethat we do not get a single digit WER for any ofthe languages compared to the results on EnglishASR in (Baevski et al., 2020b). This indicates thatdeveloping ASR systems for Indic languages withtheir richer phoneme/character sets and vocabularyis more challenging than English.

gu ta te

Baseline (Srivastava et al., 2018) 19.8 19.4 22.6

Jilebi (Pulugundla et al., 2018) 14.0 13.9 14.7Cogknit (Fathima et al., 2018) 17.7 16.0 17.1CSALT-LEAP(Srivastava et al., 2018) - 16.3 17.6

ISI-Billa (Billa, 2018) 19.3 19.6 20.9MTL-SOL (Sailor and Hain, 2020) 18.4 16.3 18.6Reed (Sen et al., 2021) 16.1 19.9 20.2CNN + Context temporal features(Sen et al., 2020) 18.4 24.3 25.2

EkStep model∗ 19.5 22.1 21.9

M5: 11.7 13.6 11.0M6: 12.3 15.1 12.4

Table 9: Comparison of our best models (M5, M6) withthe the top performers from the MSR 2018 leaderboardas well as other recent state of the art methods.

5.3 Iterative Pseudo Labelling

We also explore augmenting the labelled datasetswith Iterative Pseudo-Labeling (IPL) (Xu et al.,

bn ne si

Baseline (Shetty and Umesh, 2021) 17.9 12.9 21.8

Ekstep model∗ 15.2 13.8 20.0

M4: 14.3 13.0 18.6M6: 10.6 9.7 -

Table 10: Comparison of our best models (M4, M6)with state-of-the-art results reported in the literature. *The Ekstep model was fine-tuned by us.

2020) which is a semi-supervised algorithm for end-to-end ASR applied on larger unlabelled datasets.Motivated by the success of this approach (Xu et al.,2020; Baevski et al., 2020b), we explore the use ofIPL on our large unlabelled pretraining corpus. Werandomly sample 500 hours of pretraining data foreach language. Then, using our monolingual mod-els (M6) we generate pseudo-labels for the data.Similar to the fine-tuning setup, we use a lexicon-based beam search decoder using LMlarge + aug-mented lexicon with beam size of 128. To eliminatenoise from the data, we remove transcriptions hav-ing less than 3 words. We observe that∼10% of thetranscriptions are filtered out. Next, we perform asecond round of fine-tuning combining the pseudo-labeled data along with the fine-tuning datasets.We present the results of iterative pseudo-labelingin Table 7. We initialize the models from the fine-tuned checkpoints (M6) and perform IPL, whichis shown in row M10. We observe that the bestperformance is retained by models that do not useIPL, i.e., M6. This is a surprising finding as IPLhas been shown to work very well for English (Xuet al., 2020). We believe that this is not the case forIndian languages because the performance of theIndian models used for pseudo labeling the data ispoorer as compared to English ASR models (WERs> 10 as opposed to < 5 for English). Thus, thegenerated pseudo-labels would be more noisy inour case. Furthermore, the unlabeled audio datathat we used does not contain sentence-wise seg-ments unlike the LibriSpeech dataset (Panayotovet al., 2015). This causes the transcriptions to benoisy at the boundaries of the audio sample. Theseissues need to be resolved before establishing thevalue of IPL for Indic ASR.

5.4 Comparison with existing baselines

We now compare with existing models on the threedatasets. For each dataset, we compare with thefollowing models:

• Baseline. This is the baseline model as reportedby authors of the dataset (OpenSLR) or the organ-isers of the challenge (MUCS/MSR). Often, thisis a Time delay neural network (TDNN) model(Peddinti et al., 2015).• Top3 entries from the leaderboard. For theMUCS12 and MSR13 challenge, we include resultsfor the Top 3 entries from the leaderboard of thechallenge. Note that the MUCS challenge has onlyrecently concluded and the system papers writtenby the participating teams are not yet available.• Best per language. For the MUCS challenge,we observed that some systems did poorly whenconsidering the average performance across all lan-guages, but did very well on a specific language(and hence did not appear in the top3 entries onthe leaderboard). For example, one participant,Uniphore, got an average WER of 22.92 ranking4th on the leaderboard but had the best performancefor Tamil with a WER of 18.8. For a comprehen-sive comparison, for every language we report thebest individual system for that language even if itsperformance was very poor on other languages.• Other SOTA systems. Given that the MSR chal-lenge was launched in 2018, there are some follow-up works (Sailor and Hain, 2020; Sen et al., 2021,2020) which have reported results on this dataset.We report the numbers as it is from these works.Similarly, for OpenSLR, we compare with the re-sults reported in Shetty and Umesh (2021).• EkStep. As mentioned earlier, this is the recentlyreleased pretrained model covering 23 Indian lan-guages (Gupta et al., 2021).

The above results are summarised in Table 8, 9,10. Across all the datasets and all the languages(except MUCS-Tamil), we establish new state ofthe art results. Even for MUCS-Tamil, we outper-form the top3 models on the leaderboard. Theseresults clearly establish the recipe for developingASR systems for Indian languages: multilingualpretraining followed by multilingual fine-tuningcombined with language models trained on largecorpora with good lexicons.

6 Analysis of the pretrained model

IndicWav2Vec has been trained on 40 languagesand represents the largest diversity of Indian lan-

12https://navana-tech.github.io/MUCS2021/leaderboard.html13https://www.microsoft.com/en-

us/research/event/interspeech-2018-special-session-low-resource-speech-recognition-challenge-indian-languages/#!leaderboard

Figure 1: The plot shows the distribution of codebookentries across languages.

guages in any such multilingual model. In thissection, we analyze this pretrained model to under-stand (a) the multilingual representations learnt bythe model, and (b) the role of the attention headsthat are characteristic of Transformer layers in themodel.

6.0.1 Multilingual representations

As discussed, the first part of the IndicWav2Vecmodel computes representations of frames of audiofrom a learnt codebook. In the model, we use acodebook of 102.4K (320× 320) vectors which areshared by all 40 languages. One may ask two ques-tions about these code-books: (a) Are codebookentries disjointed or common across languages?,and (b) Is the sharing of codebook entries strongerbetween more related languages? To answer this,we visualize the codebook entries (columns) acrossall languages (rows) in the heatmap shown in 1. Werandomly sample 5 hours of training data samplesfrom each language, i.e. a total of 200 hours of data.We generate codebook vectors by passing the audiosequence through the feature encoder followed bythe quantizer. The color of each cell denotes thefrequency of that code-book entry for that language- wherein red denotes higher frequency and bluelower-frequencies. Here the columns are sortedbased on decreasing order of frequency of occur-rence of code-book entries in Hindi audio. Therows are sorted in decreasing order of similarity of

Figure 2: t-SNE plot showing language representationslearnt at the 5-th layer of the context network.

the frequency pattern of that language with Hindi.As can be seen from the first few columns whichhave red cells in many rows, a large number of thecode-book entries common in Hindi are also usedin other languages. This indicates the value in mul-tilingual pretraining. However, it is also clear thatas we move to languages of other families, such asMizo, there are large numbers of codebook entriesthat are unique to these languages. For instance, avery few of the early red columns for Hindi havesimilarly high frequency counts for Mizo.

While the above visualization is from the code-book entries, we may also study the representationslearnt by the model across the layers of the Trans-former. We use the same 200 hours of sampleddata, and generate the transformer layer outputs foreach sequence.We take an average of all framesin a sequence to compute a single representation.Next, we compute the centroids of all representa-tions for a given language. We plot the centroidslanguage-wise after a t-SNE projection in 2. Asthe plots shows several related languages (suchas Kannada and Telugu; Hindi and Urdu; Khasi,Garo and Jaintia; Bodo and Karbi; Balti, Pahariand Gojri; Bhojpuri and Maithili) are close to eachother, while some languages (Bhutia and Lepcha)are quite distinct from others. This plot again reaf-firms that multilingual pretraining has the potentialto generalize representation learning across similarlanguages.

6.0.2 Role of Attention Heads in TransformerThe attention heads which characterize Transform-ers are crucial for sharing information between therepresentations of multiple frames in a sequence.In the NLP literature, extensive studies have beenperformed to analyze and interpret the role of theseattention heads (Pande et al., 2021). To the best of

Figure 3: Attention heads pay attention to nextphoneme, previous phoneme, and silence tokens. ’bcl’indicates the silence token.

our knowledge, no similar study exists for Trans-former architectures for ASR. For an attention headin a Transformer layer, let aij denote the attentionpaid in updating the representation of frame i usingthe representation of frame j. We make use of ourmodel fine-tuned on the Hindi MUCS dataset togenerate phoneme annotations for each frame. Weuse the predictions computed by the CTC (Graveset al., 2006) loss to extract this information. Giventhat frames are relatively short and a phonemespans multiple frames, we compute an aggrega-tion of a given as apq =

∑i∈P

∑j∈Q aij where p

and q are phonemes and P and Q are the framesrespectively that comprise the two phonemes. Wecompute a for pairs of phonemes for a few sampleinput sentences and plot them in 3. We make twokey observations from these plots. First, attentionheads seem to play some distinctive roles - a majorfraction of them either attend to nearby phonemes(next or previous) or to the nearest silence. Thisis similar to findings in NLP where a majority ofthe attention heads encode local and a few heads

encode syntactic information (Pande et al., 2021).Second, this clear pattern in the attention headsmotivates building efficient models that regularizeattention heads to attend to neighboring phonemesand silence. We note this a potential future work.

7 Discussion and Recommendations

Based on our work, we make the following recom-mendations for developing ASR systems for Indianlanguages. Informing ASR model choice Our re-sults comparing IndicWav2Veclarge models withseveral model architectures proposed over the lastseveral years suggests that a Wav2Vec like modelachieves state-of-the-art results as also evidencedin English ASR. We also establish that larger mod-els which can be pretrained with larger corpora ofmultilingual data are beneficial.Informing ASR metrics for evaluation In Table3, we observed that the Language Model and lexi-con play a significant role in improving the WER.We further report experiments which show that, thelarge gap in the WER of two acoustic models, sig-nificantly reduces when the same LM is integratedwith both the models. Hence, when evaluatingarchitectural choices made by different acousticmodels (such as a better feature encoder, betterquantizer, better vocabulary, etc) it is importantto ensure that the LMs and lexicons used by themodels being compared are exactly the same.Informing ASR benchmark creation Given therole of LM and lexicon, it is important to carefullydesign benchmarks for evaluating ASR systems.In particular, the distinction between performanceon test sets with a large number of OOVs v/s testsets with mainly in-domain vocabulary should beclearly made. This requires the creation of bench-marks which contain data from different domainsallowing a more nuanced evaluation of ASR mod-els.Informing ASR resource creation The findingfrom English ASR that fine-tuning can be per-formed with just tens of minutes of labelled datadoes not transfer to Indic languages. In particu-lar, we find that both pretraining and fine-tuningdatasets sensitively affect accuracy. Thus, modelor training choices do not seem to substitute thecontinued need for collecting resources for Indiclanguages.Extending to the long tail of languages Our re-sults show that pretraining with a diverse set of lan-guages, also improves the performance on related

languages which our not present in the pretrainingdata (e.g., Sinhala). This creates hope for develop-ing ASR systems for the long tail of languages inIndia. However, to enable this, we still need effortsto create evaluation benchmarks and fine-tuningdata (at least 40 hours) for these languages. Thisis a call to the community to support us in this ef-fort of developing ASR systems for the next billionusers.

8 Conclusion

We report results of applying two recent and suc-cessful ideas from English ASR to Indic ASR: useof wav2vec like model architecture and use of un-labelled data to pretrain the model. We implementthis with a curated dataset on Indic languages anda range of ablation studies on architecture, pretrain-ing, fine-tuning, and decoding choices. Throughthis, we obtain state-of-the-art results on 9 Indiclanguages across 3 datasets. While advancing ASRsystems for the next billion users from the sub-continent, our results highlight the need for largerresources and benchmarks across more languages.

All the models developed as a part of this work,viz., the pretrained model, the language-specificfine-tuned models and the language models alongwith the Fairseq and KenLM scripts and configu-ration files used for building them will be publiclyreleased. We hope that these models will help inadvancing the state of the art for Indian SpeechTechnology.

Acknowledgments

We would like to thank the Ministry of Electron-ics and Information Technology (MeitY14) of theGovernment of India and the Centre for Develop-ment of Advanced Computing (C-DAC15), Punefor generously supporting this work and providingus access to multiple GPU nodes on the Param Sid-dhi Supercomputer. We would like to thank theEkStep Foundation for their generous grant whichwent into hiring human resources as well as cloudresources needed for this work. We would also liketo thank the Robert Bosch Center for Data Scienceand Artificial Intelligence for supporting Sumanthand Gowtham through their Post Baccalaureate Fel-lowship Program.

14https://www.meity.gov.in/15https://www.cdac.in/index.aspx?id=pune

ReferencesNaveen Arivazhagan, Ankur Bapna, Orhan Firat,

Dmitry Lepikhin, Melvin Johnson, Maxim Krikun,Mia Xu Chen, Yuan Cao, George F. Foster, ColinCherry, Wolfgang Macherey, Zhifeng Chen, andYonghui Wu. 2019. Massively multilingual neuralmachine translation in the wild: Findings and chal-lenges. CoRR, abs/1907.05019.

Alexei Baevski, Michael Auli, and Abdelrahman Mo-hamed. 2020a. Effectiveness of self-supervised pre-training for speech recognition.

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed,and Michael Auli. 2020b. wav2vec 2.0: A frame-work for self-supervised learning of speech represen-tations.

J. Billa. 2018. Isi asr system for the low resourcespeech recognition challenge for indian languages.In INTERSPEECH.

Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab-delrahman Mohamed, and Michael Auli. 2020a. Un-supervised cross-lingual representation learning forspeech recognition.

Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab-del rahman Mohamed, and Michael Auli. 2020b.Unsupervised cross-lingual representation learningfor speech recognition. ArXiv, abs/2006.13979.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal,Vishrav Chaudhary, Guillaume Wenzek, FranciscoGuzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020c. Unsupervisedcross-lingual representation learning at scale. InProceedings of the 58th Annual Meeting of the As-sociation for Computational Linguistics, ACL 2020,Online, July 5-10, 2020, pages 8440–8451. Associa-tion for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advancesin Neural Information Processing Systems 32: An-nual Conference on Neural Information ProcessingSystems 2019, NeurIPS 2019, December 8-14, 2019,Vancouver, BC, Canada, pages 7057–7067.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah,Ankita Singh, Srinivasa Raghavan, Shreya Khare,Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chi-ranjeevi Yarra, Ashish Mittal, Prasanta KumarGhosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri,

Sunayana Sitaram, Samarth Bharadwaj, Jai Nana-vati, Raoul Nanavati, Karthik Sankaranarayanan, Te-jaswi Seeram, and Basil Abraham. 2021. Multi-lingual and code-switching asr challenges for lowresource indian languages. Proceedings of Inter-speech.

Murray B Emeneau. 1956. India as a lingustic area.Language.

Noor Fathima, Tanvina Patel, Mahima C, and AnuroopIyengar. 2018. Tdnn-based multilingual speechrecognition system for low resource indian lan-guages. In Interspeech 2018, 19th Annual Confer-ence of the International Speech Communication As-sociation, Hyderabad, India, 2-6 September 2018,pages 3197–3201. ISCA.

Alex Graves, Santiago Fernández, Faustino Gomez,and Jürgen Schmidhuber. 2006. Connectionisttemporal classification: Labelling unsegmented se-quence data with recurrent neural networks. In Pro-ceedings of the 23rd International Conference onMachine Learning, ICML ’06, page 369–376, NewYork, NY, USA. Association for Computing Machin-ery.

Anirudh Gupta, Harveen Singh Chadha, PriyanshiShah, Neeraj Chimmwal, Ankur Dhuriya, RishabhGaur, and Vivek Raghavan. 2021. CLSRIL-23:cross lingual speech representations for indic lan-guages. CoRR, abs/2107.07402.

Kenneth Heafield. 2011. KenLM: Faster and smallerlanguage model queries. In Proceedings of theSixth Workshop on Statistical Machine Translation,pages 187–197, Edinburgh, Scotland. Associationfor Computational Linguistics.

Vikas Joshi, Amit Das, Eric Sun, Rupesh R Mehta,Jinyu Li, and Yifan Gong. 2021. Multiple softmaxarchitecture for streaming multilingual end-to-endasr systems. Proc. Interspeech 2021, pages 1767–1771.

Herve Jégou, Matthijs Douze, and Cordelia Schmid.2011. Product quantization for nearest neighborsearch. IEEE Transactions on Pattern Analysis andMachine Intelligence, 33(1):117–128.

Divyanshu Kakwani, Anoop Kunchukuttan, SatishGolla, Gokul N.C., Avik Bhattacharyya, Mitesh M.Khapra, and Pratyush Kumar. 2020. IndicNLPSuite:Monolingual corpora, evaluation benchmarks andpre-trained multilingual language models for Indianlanguages. In Findings of the Association for Com-putational Linguistics: EMNLP 2020, pages 4948–4961, Online. Association for Computational Lin-guistics.

Chanwoo Kim and Richard M. Stern. 2008. Ro-bust signal-to-noise ratio estimation based on wave-form amplitude distribution analysis. In Proc. Inter-speech 2008, pages 2598–2601.

Oddur Kjartansson, Supheakmungkol Sarin, Knot Pi-patsrisawat, Martin Jansche, and Linne Ha. 2018.Crowd-Sourced Speech Corpora for Javanese, Sun-danese, Sinhala, Nepali, and Bangladeshi Ben-gali. In Proc. The 6th Intl. Workshop on SpokenLanguage Technologies for Under-Resourced Lan-guages (SLTU), pages 52–55, Gurugram, India.

Anoop Kunchukuttan. 2020. The Indic-NLP Library. https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.

Anoop Kunchukuttan and Pushpak Bhattacharyya.2020. Utilizing Language Relatedness to im-prove Machine Translation: A Case Study on Lan-guages of the Indian Subcontinent. arXiv preprintarXiv:2003.08925.

Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Col-lobert. 2019. Letter-based speech recognition withgated convnets.

Abdelrahman Mohamed, Dmytro Okhonko, and LukeZettlemoyer. 2020. Transformers with convolu-tional context for asr.

Myle Ott, Sergey Edunov, Alexei Baevski, AngelaFan, Sam Gross, Nathan Ng, David Grangier, andMichael Auli. 2019. fairseq: A fast, extensibletoolkit for sequence modeling. In Proceedings ofNAACL-HLT 2019: Demonstrations.

Vassil Panayotov, Guoguo Chen, Daniel Povey, andSanjeev Khudanpur. 2015. Librispeech: an asr cor-pus based on public domain audio books. In 2015IEEE international conference on acoustics, speechand signal processing (ICASSP), pages 5206–5210.IEEE.

Madhura Pande, Aakriti Budhraja, Preksha Nema,Pratyush Kumar, and Mitesh M Khapra. 2021. Theheads hypothesis: A unifying statistical approach to-wards understanding multi-headed attention in bert.In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 35, pages 13613–13621.

Daniel S Park, William Chan, Yu Zhang, Chung-ChengChiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.2019. Specaugment: A simple data augmentationmethod for automatic speech recognition. arXivpreprint arXiv:1904.08779.

Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khu-danpur. 2015. A time delay neural network architec-ture for efficient modeling of long temporal contexts.In INTERSPEECH 2015, 16th Annual Conferenceof the International Speech Communication Asso-ciation, Dresden, Germany, September 6-10, 2015,pages 3214–3218. ISCA.

Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai,Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky,and Ronan Collobert. 2019. Wav2letter++: A fastopen-source speech recognition system. ICASSP

2019 - 2019 IEEE International Conference onAcoustics, Speech and Signal Processing (ICASSP).

Bhargav Pulugundla, K. Murali Baskar, Santosh Kesir-aju, Ekaterina Egorova, Martin Karafiát, Lukáš Bur-get, and Jan Cernocký. 2018. But system for lowresource indian language asr. In Proceedings of In-terspeech 2018, volume 2018, pages 3182–3186. In-ternational Speech Communication Association.

Gowtham Ramesh, Sumanth Doddapaneni, AravinthBheemaraj, Mayank Jobanputra, Raghavan AK,Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Ma-halakshmi J, Divyanshu Kakwani, Navneet Ku-mar, Aswin Pradeep, Kumar Deepak, Vivek Ragha-van, Anoop Kunchukuttan, Pratyush Kumar, andMitesh Shantadevi Khapra. 2021. Samanantar: Thelargest publicly available parallel corpora collectionfor 11 indic languages. CoRR, abs/2104.05596.

Hardik B. Sailor and Thomas Hain. 2020. Multi-lingual speech recognition using language-specificphoneme recognition as auxiliary task for indian lan-guages. In INTERSPEECH.

Bipasha Sen, Aditya Agarwal, Mirishkar Sai Ganesh,and A. K. Vuppala. 2020. Short term context basedfast multilingual acoustic model for low resourcelanguages.

Bipasha Sen, Aditya Agarwal, Mirishkar Sai Ganesh,and Anil Kumar Vuppala. 2021. Reed: An approachtowards quickly bootstrapping multilingual acousticmodels. In 2021 IEEE Spoken Language Technol-ogy Workshop (SLT), pages 272–279.

Vishwas M. Shetty and S. Umesh. 2021. Exploringthe use of common label set to improve speechrecognition of low resource indian languages. InICASSP 2021 - 2021 IEEE International Confer-ence on Acoustics, Speech and Signal Processing(ICASSP), pages 7228–7232.

Brij Mohan Lal Srivastava, Sunayana Sitaram, RupeshKumar Mehta, Krishna Doss Mohan, Pallavi Matani,Sandeepkumar Satpal, Kalika Bali, RadhakrishnanSrikanth, and Niranjan Nayak. 2018. Interspeech2018 Low Resource Automatic Speech RecognitionChallenge for Indian Languages. In Proc. 6th Work-shop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), pages 11–14.

Karumuri V Subbarao. 2012. South Asian languages:A syntactic typology. Cambridge University Press.

Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Ta-tiana Likhomanenko, Edouard Grave, Vineel Pratap,Anuroop Sriram, Vitaliy Liptchinsky, and RonanCollobert. 2019. End-to-end asr: from supervised tosemi-supervised learning with modern architectures.arXiv preprint arXiv:1911.08460.

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na-man Goyal, Vishrav Chaudhary, Jiatao Gu, and An-gela Fan. 2020. Multilingual translation with exten-sible multilingual pretraining and finetuning. CoRR,abs/2008.00401.

Changhan Wang, Morgane Rivière, Ann Lee, AnneWu, Chaitanya Talnikar, Daniel Haziza, MaryWilliamson, Juan Pino, and Emmanuel Dupoux.2021a. Voxpopuli: A large-scale multilingualspeech corpus for representation learning, semi-supervised learning and interpretation.

Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani,Shujie Liu, Furu Wei, Michael Zeng, and XuedongHuang. 2021b. Unispeech: Unified speech repre-sentation learning with labeled and unlabeled data.In Proceedings of the 38th International Conferenceon Machine Learning, volume 139 of Proceedingsof Machine Learning Research, pages 10937–10947.PMLR.

Felix Wu, Angela Fan, Alexei Baevski, Yann N.Dauphin, and Michael Auli. 2019. Pay less attentionwith lightweight and dynamic convolutions.

Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn,Awni Hannun, Gabriel Synnaeve, and Ronan Col-lobert. 2020. Iterative pseudo-labeling for speechrecognition. arXiv preprint arXiv:2005.09267.