J-MAC: Tarin Kaset na Littafin Sauti na Masu Magana Da Yawa na Jafananci don Haɗa Murya

1. Gabatarwa

Takardar ta gabatar da J-MAC (Tarin Kaset na Littafin Sauti na Masu Magana Da Yawa na Jafananci), wani sabon tarin murya da aka tsara don ci gaba da bincike a cikin haɗa murya mai bayyanawa, mai fahimtar mahallin, musamman don aikace-aikacen littafin sauti. Marubutan suna jayayya cewa, yayin da haɗa murya na salon karatu ya kai matakin kusan na ɗan adam, gaba gaba ya ƙunshi sarrafa mahallin rikitarwa, tsakanin jimloli, bayyanawa ta musamman ga mai magana, da kwararar labari—dukkan suna da mahimmanci don samar da littafin sauti mai jan hankali. Rashin ingantattun tarin kaset na littafin sauti na masu magana da yawa ya kasance babban cikas. J-MAC ta magance wannan ta hanyar samar da hanyar gina irin wannan tarin kaset ta atomatik daga littattafan sauti na kasuwanci da ƙwararrun masu ba da labari suka karanta, wanda ya sa bayanan da aka samu su zama buɗaɗɗe.

2. Gina Tarin Kaset

Tsarin ginin mataki ne uku da aka tsara don sarrafa kansa (atomatik) da zaman kansa daga harshe.

2.1 Tattara Bayanai

Ana zaɓar littattafan sauti bisa manyan ma'auni guda biyu: 1) Kasancewar ingantaccen rubutun tunani (zai fi dacewa littattafan da ba su da haƙƙin mallaka don guje wa kurakuran ASR akan sunaye na musamman), da 2) Kasancewar nau'ikan nau'ikan da ƙwararrun masu magana daban-daban suka ba da labari don ɗaukar nau'ikan salon bayyanawa daban-daban. Wannan yana ba da fifiko ga bambancin mai magana fiye da yawan bayanai daga mai magana guda.

2.2 Tsarkakewa & Daidaitawar Bayanai

Ana aiwatar da sarrafa sautin danyen don cire sassan murya masu tsabta da daidaita su daidai da rubutun da ya dace. Wannan ya ƙunshi rabewar tushe, daidaitawar ƙaƙƙarfan amfani da Rarraba Lokaci na Haɗin Kai (CTC), da gyaran ƙaƙƙarfan amfani da Ganowa Ayyukan Murya (VAD).

3. Hanyar Fasaha

3.1 Rabewar Murya da Kayan Kida

Don ware murya mai tsabta daga yuwuwar kiɗan bango ko tasirin sauti a cikin samar da littafin sauti, ana amfani da ƙirar rabewar tushe (kamar waɗanda suka dogara da Rarraba Mai Zurfi ko Conv-TasNet). Wannan mataki yana da mahimmanci don samun ingantaccen bayanan horo don ƙirar haɗawa.

3.2 Daidaitawa na Tushen CTC

Ƙirar ASR da aka horar da CTC tana ba da farkon daidaitawar ƙaƙƙarfan tsakanin siffar sauti da jerin rubutu. Aikin asarar CTC $\mathcal{L}_{CTC} = -\log P(\mathbf{y}|\mathbf{x})$, inda $\mathbf{x}$ shine jerin shigarwa kuma $\mathbf{y}$ shine jerin lakabin manufa, yana ba da damar daidaitawa ba tare da tilasta rarrabuwa ba.

3.3 Gyara na Tushen VAD

Ana gyara ƙaƙƙarfan daidaitawar CTC ta amfani da tsarin Ganowa Ayyukan Murya. Wannan matakin yana cire sassan da ba na magana ba (dakata, numfashi) da daidaita iyakoki don tabbatar da kowane yanki na sauti ya dace daidai da naúrar rubutu (misali, jimla), yana inganta daidaitar ma'auratan rubutu da sauti.

4. Sakamakon Gwaji & Kimantawa

Marubutan sun gudanar da kimantawar haɗa murya na littafin sauti ta amfani da ƙirar da aka horar a kan J-MAC. Muhimman binciken sun haɗa da:

Ingantacciyar Ƙirar Ƙirar: Ingantattun tsarin haɗawa sun inganta yanayin sautin fitarwa a cikin masu magana daban-daban a cikin tarin kaset.
Abubuwan Haɗe-haɗe: Yanayin da aka gane ya kasance mai tasiri sosai ta hanyar mu'amala mai rikitarwa tsakanin hanyar haɗawa, halayen muryar mai magana, da kuma abun cikin littafin kansa. Rarraba waɗannan abubuwan har yanzu kalubale ne.

Bayanin Jadawali (A ƙa'ida): Jadawali na hasashe zai nuna Matsakaicin Maki Ra'ayi (MOS) don yanayi a cikin tsarin haɗawa daban-daban (misali, Tacotron2, FastSpeech2) da masu magana daban-daban na J-MAC. Jadawalin zai iya nuna bambance-bambance a cikin masu magana don ƙirar iri ɗaya da kuma ci gaba mai daidaitawa don ƙirar ci gaba a cikin duk masu magana, yana tabbatar da gani biyu na mahimman fahimta.

5. Muhimman Fahimta & Tattaunawa

J-MAC ta samar da cikakkiyar hanya mai iya faɗaɗawa, atomatik don ƙirƙirar tarin kaset na murya mai bayyanawa.
Zane na masu magana da yawa, littafi ɗaya ƙarfi ne na musamman don nazarin ainihin mai magana da bayyanawa.
Kimantawa ta nuna cewa ƙirar TTS na littafin sauti na gaba dole ne ta yi la'akari da yanayin haɗe-haɗe na abun ciki, mai magana, da salo.

6. Bincike na Asali: Ra'ayi na Masana'antu

Mahimman Fahimta: Takardar J-MAC ba kawai game da sabon tarin bayanai ba ce; wasa ce ta dabara don canza tsarin TTS daga samar da furuci keɓaɓɓe zuwa hankalin labari. Yayin da ƙirar kamar WaveNet da Tacotron suka ci nasara a cikin amincin su, sun yi watsi da babban tsarin magana. J-MAC, ta hanyar samar da labarai masu kama da juna daga ƙwararrun masu magana da yawa, shine tushen da ake buƙata don ƙirar su koyi ba kawai yadda ake magana ba, amma yadda ake yin labari. Wannan ya yi daidai da babban yanayin masana'antu da aka gani a cikin ayyuka kamar takardar Google AudioLM, wanda ke neman ƙirar sauti ta hanyar fahimtar mahallin, matakai.

Kwararar Hankali: Marubutan sun gano daidai cikas na bayanai. Maganinsu na gaskiya ne: hako abubuwan da ake da su, ingantattun samfuran fasaha (littattafan sauti) maimakon ba da sabbin rikodin. Hanyar fasaha ta wayo—amfani da fasahohi masu girma (CTC, VAD) a cikin sabon haɗin gwiwa don takamaiman manufa mai daraja. Kimantawa sannan tana amfani da wannan sabon albarkatun don nuna mahimman binciken, wanda ba a bayyane ba: a cikin haɗawa mai bayyanawa, ba za ku iya inganta don "mafi kyawun ƙira" marar mai magana ba. Aikin yana da alaƙa da ainihin mai magana.

Ƙarfi & Kurakurai: Babban ƙarfi shine ƙa'idar ƙirar tarin kaset. Zaɓin ƙwararrun masu magana da kwatancen rubutu iri ɗaya yana da kyau don nazarin sarrafawa. Hanyar atomatik gudummawa ce mai mahimmanci ga sake samarwa. Duk da haka, kuskuren takardar shine kimantawarta ta farko. Fahimtar "abubuwan haɗe-haɗe" tana da mahimmanci amma kawai an bayyana shi. Ana buƙatar zurfin bincike, watakila ta amfani da dabarun daga wallafe-wallafen canja salo (kamar tsarin maɓalli a cikin Global Style Tokens ko hanyoyin rarrabuwa da aka bincika a cikin CycleGAN-VC). Nawa ne bambancin ya kasance saboda sautin murya vs. salon sautin murya vs. fassarar ma'ana? Takardar ta buɗe kofa amma ba ta bi ta ciki ba.

Fahimta Mai Aiki: Ga masu bincike: Yi amfani da J-MAC don auna ma'auni na dabarun rarrabuwa. Ga ƙungiyoyin samfur: Wannan aikin yana nuna cewa ƙarni na gaba na AI na murya don podcasts, tallace-tallace, da littattafai ba za su zo daga ƙarin bayanai na salon karatu ba, amma daga bayanan aikin labari. Fara tsara bayanai masu bayyanawa, dogon tsari. Hanyar kanta ana iya fitar da ita—tunani "J-MAC don Podcasts" ko "J-MAC don Trailer na Fim". Babban darasi shine cewa a zamanin ƙirar tushe, ƙimar dabara na tarin bayanai na musamman, ingantaccen tsari kamar J-MAC na iya fiye da kowane tsarin ƙira guda ɗaya da aka buga tare da shi.

7. Cikakkun Bayanai na Fasaha & Tsarin Lissafi

Tsarin daidaitawa ya dogara ne akan algorithm na gaba-baya na CTC. Idan aka ba da jerin shigarwa $\mathbf{x}$ na tsawon $T$ da jerin manufa $\mathbf{l}$ na tsawon $L$, CTC ta ayyana rarraba akan daidaitawa ta hanyar gabatar da alamar fanko ($\epsilon$) da ba da damar maimaitawa. Yiwuwar manufa shine jimlar duk ingantattun daidaitawa $\pi$:

$P(\mathbf{l} | \mathbf{x}) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} P(\pi | \mathbf{x})$

inda $\mathcal{B}$ shine aikin da ke rushe alamomin da aka maimaita da kuma cire fanko. Ana iya tsara gyaran VAD azaman aikin rarrabuwa, gano iyakoki $\{t_i\}$ waɗanda ke haɓaka yuwuwar magana a cikin sassan da ba na magana ba a tsakanin su, sau da yawa ta amfani da fasalin tushen kuzari ko na'urar rarraba da aka horar.

8. Tsarin Bincike: Nazarin Lamari

Yanayi: Kimanta tasirin salon mai magana akan "haɗin kai" da aka gane a cikin haɗa littafin sauti.

Aikace-aikacen Tsarin:

Rarraba Bayanai: Ɗauki ƙwararrun masu magana biyu (A & B) daga J-MAC waɗanda suka ba da labarin babi ɗaya na wani labari.
Cire Fasali: Ga kowane furuci a cikin babin, cire ƙananan masu siffanta (LLDs) kamar siffofin sautin murya, ƙarfin kuzari, da tsawon dakata ta amfani da kayan aiki kamar OpenSMILE ko Praat. Haka nan cire ɗimbin salo masu girma ta amfani da ƙirar da aka riga aka horar kamar HuBERT.
Binciken Kwatankwacinsa: Lissafa bambance-bambancen ƙididdiga (misali, ta amfani da gwajin t ko bambancin KL) tsakanin rarraba LLDs don Mai Magana A da Mai Magana B don abun cikin rubutu iri ɗaya. Wannan yana ƙididdige "hattum hannunsu" na sautin murya na musamman.
Haɗawa & Kimantawa: Horar da ƙirar TTS guda biyu: ɗaya akan bayanan Mai Magana A, ɗaya akan na Mai Magana B. Haɗa wannan nassi na labari ɗaya wanda ba a gani yayin horo ba. Gudanar da gwajin sauraro inda masu kimantawa suka ƙididdige kowane haɗawa don "bayyanawa" da "haɗin kai na labari."
Dangantaka: Haɗa bambance-bambancen salo na haƙiƙa (Mataki na 3) tare da makin ra'ayi na haɗin kai (Mataki na 4). Wannan tsarin, wanda tsarin J-MAC ya ba da damar, zai iya ware wane fasalin sauti ya fi ba da gudummawa ga ingancin aikin da aka gane.

Wannan nazarin lamari yana nuna yadda J-MAC ke sauƙaƙe binciken dalili, ya wuce dangantaka zuwa fahimtar ginshiƙan magana mai bayyanawa.

9. Aikace-aikace na Gaba & Hanyoyin Bincike

Kwafin Murya Mai Bayyanawa & Keɓancewa: Bayanan masu magana da yawa na J-MAC sun dace don haɓaka tsarin daidaitawar murya kaɗan-kaɗan ko sifili wanda zai iya kwaikwayon salon labarin mai magana, ba kawai sautinsu ba.
Koyon Wakilcin Rarrabuwa: Aikin gaba zai iya amfani da J-MAC don horar da ƙirar da ke raba abun ciki, ainihin mai magana, da salon bayyanawa zuwa wurare na ɓoye daban-daban, yana ba da damar sarrafa haɗawa mai ƙaƙƙarfan ƙira.
Haɗa Littafin Sauti na Tsakanin Harsuna: Ana iya amfani da hanyar zuwa wasu harsuna don gina tarin kaset masu kama, yana ba da damar bincike kan kiyaye salon bayyanawa a cikin fassarar ko dubbing.
Ƙirƙirar Abun Ciki Mai Taimakon AI: Haɗawa tare da manyan ƙirar harshe (LLMs) na iya haifar da tsarin da ke rubutu da yin gajerun labarai ko abun cikin sauti na keɓance a cikin takamaiman salon mai ba da labari.
Kayan Aikin Samun dama: Samar da ingantattun littattafan sauti masu bayyanawa akan buƙata don kowane rubutun dijital, yana faɗaɗa damar ga masu amfani da nakasar gani.

10. Nassoshi

J. Shen, da sauransu, "Haɗa TTS na Halitta ta Hanyar Sharadi WaveNet akan Hasashen Siffar Mel," ICASSP, 2018.
A. Vaswani, da sauransu, "Hankali Duk Abinda Kake Bukata," NeurIPS, 2017.
Y. Ren, da sauransu, "FastSpeech: Mai Sauri, Ƙarfi da Sarrafa Rubutu zuwa Magana," NeurIPS, 2019.
A. v. d. Oord, da sauransu, "WaveNet: Ƙirar Samarwa don Sautin Danye," arXiv:1609.03499, 2016.
J.-Y. Zhu, da sauransu, "Fassarar Hoto zuwa Hoto mara Biyu ta amfani da Cibiyoyin Adawa na Zagaye-Mai daidaitawa," ICCV, 2017. (CycleGAN)
Y. Wang, da sauransu, "Alamun Salo: Ƙirar Salo mara Kulawa, Sarrafawa da Canja wuri a cikin Haɗa Magana na Ƙarshe-zuwa-Ƙarshe," ICML, 2018.
Google AI, "AudioLM: Hanyar Ƙirar Harshe zuwa Samar da Sauti," Google Research Blog, 2022.
A. Graves, da sauransu, "Rarraba Lokaci na Haɗin Kai: Lakabin Bayanan Jeri mara Rarrabuwa tare da Cibiyoyin Jijiyoyi masu Maimaitawa," ICML, 2006.