Interspeech 2024

• About The Repository
• Getting Started
  • Prerequisites and Installation
  • Training
• Roadmap
• Dataset Statistics
• Contributing
• Contributors
• Citation
• License

This repository hosts the artefacts pertaining to our paper Improving Self-supervised Pre-training using Accent-Specific Codebooks accepted to the main conference of Interspeech 2024. This work is an extension of our previous work Accented Speech Recognition With Accent-specific Codebooks which was accepted at the main conference of EMNLP 2023.

The main contribution of this paper 🔎 is to extend the accent adaptation technique that uses a set of learnable codebooks and a modified beam-search decoding algorithm to both self-supervised pre-training and ASR finetuning.

The repository contains two folders:

• fairseq code 📁 - Contains code to run our SSL experiments on Fairseq toolkit. Detailed instruction on how to run our experiments can be found here.
• espnet_code 📁 - Contains code to run our experiments on ESPnet toolkit. Detailed instruction on how to run ASR experiments can be found here.

• For fairseq related installation, follow the instructions here.
• FOr ESPnet related installation, follow the instructions here.
• Finally, clone the repository containing our code and dataset.

git clone https://github.com/csalt-research/accented-codebooks-asr.git

git checkout accented-pretraining 

• Additionally, to run the dataset creation script, run the following:

pip install -r accented-codebooks-asr/data/requirements.txt

1. Extract the csvs from the tar file in data folder

tar  -xvzf accented-codebooks-asr/data/dataset.tar.gz 

2. For experiments related to Fairseq, please refer to these instructions.

3. Instructions related to ESPnet training:

   • Copy the files from espnet_code into ESPnet egs

   cp -r accented-codebooks-asr/espnet_code/* <espnet_root_folder>/egs/commonvoice/asr1

   • Enter the path to the the directory hosting our splits in run.sh

   csvdir=  # Path to the directory hosting all our csvs.

   • Run the script

   ./run.sh

The statistics of train, dev and test splits used in our experiments are as follows:

See the open issues for a list of proposed features (and known issues) relevant to this work. For ESPnet related features/issues, checkout their github repository.

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

• If you have suggestions for adding or removing projects, feel free to open an issue to discuss it, or directly create a pull request after you edit the README.md file with necessary changes.
• Please open an individual PR for each suggestion.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/NewFeature)
3. Commit your Changes (git commit -m 'Add appropriate commit message'). The correct way to write your commit message can be found here
4. Push to the Branch (git push origin feature/NewFeature)
5. Open a Pull Request

• Darshan Prabhu - M.Tech, CSE, IIT Bombay - Darshan Prabhu
• Abhishek Kumar Gupta - M.Tech, CSE, IIT Bombay - Abhishek Kumar Gupta
• Omkar Nitsure - B.Tech, EE, IIT Bombay - Omkar Nitsure
• Preethi Jyothi - Associate Professor, CSE, IIT Bombay - Preethi Jyothi
• Sriram Ganapathy - Associate Professor, EE, IISc Bangalore - Sriram Ganapathy
• Vinit Unni - Ph.D, CSE, IIT Bombay - Vinit Unni

If you use this code for your research, please consider citing our works.

@misc{prabhu2023accented,
      title={Accented Speech Recognition With Accent-specific Codebooks}, 
      author={Darshan Prabhu and Preethi Jyothi and Sriram Ganapathy and Vinit Unni},
      year={2023},
      eprint={2310.15970},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{prabhu2024improvingselfsupervisedpretrainingusing,
      title={Improving Self-supervised Pre-training using Accent-Specific Codebooks}, 
      author={Darshan Prabhu and Abhishek Gupta and Omkar Nitsure and Preethi Jyothi and Sriram Ganapathy},
      year={2024},
      eprint={2407.03734},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.03734}, 
}

Distributed under the MIT License. See LICENSE for more information.