Movie101v2: Ƙarin Ma'auni Mai Kyau don Ƙirƙirar Labarin Fim ta Atomatik

Teburin Abubuwan Ciki

• 1. Gabatarwa
• 2. Ayyukan Da Suka Gabata & Dalili
  • 2.1. Gazawar Bayanan Da Suka Gabata
  • 2.2. Bukatar Movie101v2
• 3. Bayanan Movie101v2
  • 3.1. Muhimman Siffofi da Haɓakawa
  • 3.2. Ƙididdigar Bayanai
• 4. Tsarin Aiki Mai Matakai Uku
  • 4.1. Mataki na 1: Bayanin Gaskiyar Gani
  • 4.2. Mataki na 2: Labari Mai Sanin Halaye
  • 4.3. Mataki na 3: Labari Mai Maida Hankali kan Labari
• 5. Tsarin Gwaji & Ma'auni na Farko
  • 5.1. Samfuran Da Aka Yi Gwajin Su
  • 5.2. Ma'aunin Kimantawa
• 6. Sakamako & Bincike
  • 6.1. Ayyuka akan Matakai Uku
  • 6.2. Muhimman Kalubalen Da Aka Gano
• 7. Cikakkun Bayanai na Fasaha & Tsari
• 8. Aikace-aikace na Gaba & Hanyoyi
• 9. Nassoshi
• 10. Ra'ayin Mai Bincike

1. Gabatarwa

Labarin fim ta atomatik, ko Bayanin Sauti (AD), fasaha ce mai taimako mai mahimmanci da ke samar da bayanan labari da suka dace da abubuwan gani na fim, wanda ke baiwa masu nakasar gani damar jin daɗin fina-finai. Ba kamar bayyana bidiyo na yau da kullun ba, yana buƙatar ba kawai bayyana cikakkun bayanai na gani ba har ma da fayyace labarun da ke faruwa a cikin hotuna da yawa, yana gabatar da ƙalubale na musamman a cikin haɗin kai, bin diddigin halaye, da taƙaita labari. Wannan takarda ta gabatar da Movie101v2, ingantacciyar, babbar ma'auni, bayanan ma'auni mai harsuna biyu da aka ƙera don ci gaba da bincike a wannan fanni. Aikin ya gabatar da tsari mai haske mai matakai uku don aikin kuma ya ba da cikakkun kimantawa na ma'auni ta amfani da samfuran gani-harshe na zamani.

2. Ayyukan Da Suka Gabata & Dalili

Bayanan da suka gabata kamar LSMDC, MAD, da ainihin Movie101 sun kafa tushe amma suna fama da manyan gazawa, suna hana ci gaba zuwa tsarin labarai mai amfani, na ainihin duniya.

2.1. Gazawar Bayanan Da Suka Gabata

• Girma & Iyaka: Bayanan farko (misali, M-VAD, MAD) suna amfani da gajerun faifan bidiyo (akan matsakaita 4-6 seconds), suna hana samfuran koyon samar da labarai masu haɗin kai don sassan da suka fi tsawo, masu alaƙa da labari.
• Harshe & Samuwa: Movie101 na Sinanci kawai ne, yana iyakance amfani da ƙaƙƙarfan samfuran da aka horar da su da Turanci.
• Ingancin Bayanai: Bayanan meta da aka tattara ta atomatik sau da yawa sun ƙunshi kurakurai (halaye da suka ɓace, sunaye masu sabani), suna rage amincin horo da kimantawa.
• Sauƙaƙe Aiki: Wasu ayyuka sun rage aikin zuwa bayyana gaba ɗaya ta hanyar ɓoye halaye (misali, maye gurbin sunaye da "wani").

2.2. Bukatar Movie101v2

Movie101v2 tana magance waɗannan gibi ta hanyar samar da babban bayanai, mai harsuna biyu, mai inganci tare da nau'ikan bidiyo-labari masu tsayi da ingantaccen bayanin halaye, ta kafa ma'auni mai gaskiya da ƙalubale.

3. Bayanan Movie101v2

3.1. Muhimman Siffofi da Haɓakawa

• Labarai Masu Harsuna Biyu: Yana ba da labarai na Sinanci da Turanci a layi ɗaya ga kowane ɗan gajeren bidiyo.
• Haɓaka Girma: An faɗaɗa sama da ainihin fina-finai 101 (ƙididdigar sabuwar an ƙiyasta ta fi girma).
• Ingantaccen Ingancin Bayanai: An tabbatar da hannu da kuma gyara bayanan meta na halaye don tabbatar da daidaito.
• Faifan Bidiyo Masu Tsayi: Yana da sassan bidiyo masu tsayi wanda ya isa ya ƙunshi ci gaban labarai, ba kawai ayyuka keɓaɓɓu ba.

3.2. Ƙididdigar Bayanai

4. Tsarin Aiki Mai Matakai Uku

Babban gudunmawar shine rarraba aikin mai sarƙaƙiya zuwa matakai uku masu ci gaba, kowanne yana da maƙasudai da ma'aunin kimantawa.

4.1. Mataki na 1: Bayanin Gaskiyar Gani

Manufa: Bayyana daidai abubuwan da ake iya gani a cikin harbi ɗaya ko ɗan gajeren faifan bidiyo (fage, abubuwa, ayyuka na asali).
Maida Hankali akan Ma'auni: Daidaito a cikin tushen gani (misali, SPICE, CIDEr).

4.2. Mataki na 2: Labari Mai Sanin Halaye

Manufa: Samar da labarai da suka gano daidai kuma suka ambaci halaye da sunansu, suna haɗa ayyuka zuwa takamaiman ƙungiyoyi.
Maida Hankali akan Ma'auni: Daidaiton gano halaye, daidaiton suna a cikin jimloli.

4.3. Mataki na 3: Labari Mai Maida Hankali kan Labari

Manufa: Samar da taƙaitaccen bayani mai haɗin kai wanda ke haɗa abubuwan da suka faru a cikin hotuna da yawa, fayyace dalilan halaye, da haskaka mahimman abubuwan labari.
Maida Hankali akan Ma'auni: Haɗin kai na labari, alaƙar labari, da tsarin magana (misali, ta amfani da ma'auni da aka daidaita daga taƙaita rubutu).

5. Tsarin Gwaji & Ma'auni na Farko

5.1. Samfuran Da Aka Yi Gwajin Su

Takardar ta yi ma'auni na kewayon samfuran gani-harshe masu girma na zamani (VLMs), gami da amma ba'a iyakance ga GPT-4V(ision) ba. Wannan yana ba da hoton aiki mai mahimmanci na samfuran gabaɗaya na yanzu akan wannan aikin na musamman.

5.2. Ma'aunin Kimantawa

Ma'auni sun dace da tsarin aiki mai matakai uku:

• Mataki na 1: Ma'aunin bayyana na yau da kullun (BLEU, METEOR, CIDEr, SPICE).
• Mataki na 2: Ma'auni na al'ada don tunawa da sunan halayen da daidaito.
• Mataki na 3: Ma'auni masu kimanta kwararar labari da daidaiton labari, mai yuwuwa sun haɗa da kimantawar ɗan adam ko ma'auni da aka koya.

6. Sakamako & Bincike

6.1. Ayyuka akan Matakai Uku

Sakamakon da alama yana nuna babban tazara na aiki a cikin matakai. Yayin da VLMs na zamani na iya yin aiki mai kyau a kan Mataki na 1 (Gaskiyar Gani), ayyukansu yana raguwa sosai akan Mataki na 2 (Sanin Halaye) kuma musamman akan Mataki na 3 (Labari Mai Maida Hankali kan Labari). Wannan yana nuna cewa bayyana "abin da ake gani" ya bambanta da fahimtar "abin da ke faruwa a cikin labarin."

6.2. Muhimman Kalubalen Da Aka Gano

• Samfuran Dogaro Mai Tsayi: Samfuran suna fama da kiyaye mahallin da bin diddigin ƙungiyoyi a cikin jerin bidiyo masu tsayi.
• Bambance-bambancen Halaye: Wahalar gano daidai da sunanta halaye, musamman tare da kamancen gani ko kasancewar a waje da allo.
• Taƙaita Labari: Rashin iya fitar da mahimman abubuwan labari daga jerin ayyuka da tsayawar tattaunawa.
• Bambanci a cikin Horon Farko: VLMs na gabaɗaya ana horar da su akan bayanan yanar gizo (gajerun faifan bidiyo, hotuna) kuma ba su da zurfin fahimtar labari na abun cikin fim.

7. Cikakkun Bayanai na Fasaha & Tsari

Tsarin aiki mai matakai uku da kansa tsari ne na ra'ayi don tsara matsalar. Kimantawa yana buƙatar ƙirƙirar ma'auni na musamman na mataki. Misali, kimantawa mai sanin halaye na iya haɗawa da maki F1 da aka ƙidaya akan ƙungiyoyin sunayen halaye:

$\text{Daidaiton Halaye} = \frac{\text{Ambaton Halayen Da Aka Yi Hasashen Daidai}}{\text{Jimlar Ambaton Halayen Da Aka Yi Hasashen}}$

$\text{Tunawa da Halaye} = \frac{\text{Ambaton Halayen Da Aka Yi Hasashen Daidai}}{\text{Jimlar Ambaton Halayen na Gaskiya}}$

Misalin Tsarin Bincike (Ba Lamba ba): Don gano gazawar samfuri a Mataki na 3, mutum zai iya amfani da kimantawar ɗan adam bisa tsari. Masu kimantawa suna ƙididdige labarun da aka samar akan fannoni kamar:

1. Haɗin Kai: Shin jimloli suna biye da juna bisa hankali?
2. Mahimmancin Labari: Shin labarin yana haskaka mafi mahimmancin bugun labari a cikin ɗan gajeren faifan bidiyo?
3. Haɗin Dalili: Shin yana nuna ko faɗi dalilan ayyukan halaye?
4. Fahimtar Lokaci: Shin yana daidaita abubuwan da suka faru daidai?

Tattara waɗannan maki a kowane samfuri yana bayyana takamaiman rauni a cikin tunanin labari fiye da ma'aunin juzu'i-n-gram mai sauƙi.

8. Aikace-aikace na Gaba & Hanyoyi

• Ƙirƙirar AD na Ainihin Lokaci: Babban manufa shine tsarin da ba shi da jinkiri wanda zai iya ba da labari akan abun cikin da ke gudana, yana buƙatar ingantattun samfuran da suka daidaita sauri da inganci.
• Labari Na Musamman: Daidaita salon labari da matakin cikakkun bayanai bisa ga abin da mai amfani ya fi so ko ilimin da ya gabata.
• Horo na Farko na Tsakanin Nau'i: Haɓaka samfuran da aka horar da su musamman akan nau'ikan bidiyo-rubutu masu tsayi, masu labari (fina-finai tare da rubutun allo/ƙaramin rubutu/AD) maimakon gajerun faifan bidiyo na yanar gizo.
• Haɗawa da Tattaunawa & Sauti: Tsarin na gaba dole ne su haɗa labari cikin sauƙi tare da tattaunawa da kiɗan da ke akwai, gano tsayawa na halitta don saka shi—ƙalubale mai kama da matsalolin raba tushen sauti-gani da aka bincika a cikin ayyuka kamar Conv-TasNet (Luo & Mesgarani, 2019).
• Faɗaɗawa zuwa Sauran Kafofin Watsa Labarai: Yin amfani da irin wannan fasaha akan wasan kwaikwayo na kai tsaye, bidiyoyin ilimi, da wasannin bidiyo.

9. Nassoshi

1. Yue, Z., Zhang, Y., Wang, Z., & Jin, Q. (2024). Movie101v2: Ingantaccen Ma'auni na Labarin Fim. arXiv:2404.13370v2.
2. Yue, Z., et al. (2023). Movie101: Sabon Bayanan Labarin Fim. (Ainihin takardar Movie101).
3. Han, Z., et al. (2023a). AutoAD II: Zuwa Haɗa Bayanin Sauti tare da Bayanan Mahalli. (Yana gabatar da bankin halaye).
4. Han, Z., et al. (2023b). AutoAD: Bayanin Fim a cikin Mahalli. (Yana sake sanya sunayen halaye).
5. Soldan, M., et al. (2022). MAD: Babban Bayanan Bayani don Tushen Harshe a cikin Bidiyoyi daga Bayanin Sauti na Fim. CVPR.
6. Rohrbach, A., et al. (2017). Bayanin Fim. International Journal of Computer Vision.
7. Torabi, A., et al. (2015). Yin Amfani da Sabis na Bayanin Bidiyo don Ƙirƙirar Babban Tushen Bayanai don Binciken Bayanin Bidiyo. arXiv:1503.01070.
8. Luo, Y., & Mesgarani, N. (2019). Conv-TasNet: Wucewa Matsakaicin Matsakaicin Lokaci–Matsakaicin Girman Abin Rufe Fuska don Rabewar Magana. IEEE/ACM Transactions on Audio, Speech, and Language Processing. (An ambata don ƙalubalen sarrafa sauti masu alaƙa).
9. OpenAI. (2023). GPT-4V(ision) Tsarin Kati. (A matsayin wakilin ma'auni na VLM).

10. Ra'ayin Mai Bincike

Babban Fahimta: Movie101v2 ba wani bayanai kawai ba ne; yana da shiri na tsoma baki wanda ke fallasa babban tazarar fahimtar labari a cikin Samfuran Gani-Harshe (VLMs) na yau da ake zaton "gabaɗaya". Takardar ta gano daidai cewa SOTA na yanzu, gami da GPT-4V, a zahiri yana yin ingantaccen daidaita tsari akan pixels na gani da alamun rubutu, ba fahimtar labarin fim ba. Tsarin aiki mai matakai uku shine babban fasalin takardar—yana ba da kayan aikin bincike don gano daidai inda samfuran suka gaza: ba a gani ba, amma a ba da labari.

Kwararar Hankali: Hujja tana da ƙarfi: 1) Bayanan da suka gabata suna da aibi (gajere sosai, harshe ɗaya, hayaniya), suna haifar da ma'auni mara gaskiya. 2) Saboda haka, ci gaban ya kasance na ruɗi, yana inganta ma'auni mara kyau. 3) Magani: Gina ingantaccen bayanai (Movie101v2) kuma, mahimmanci, ingantaccen tsarin kimantawa (matakai 3). 4) Tabbatarwa: Nuna cewa ko da mafi kyawun samfuran suna tuntuɓe akan Matakai na 2 da 3, suna tabbatar da wajibcin tsarin da rashin balaga na fanni. Wannan hankali yayi kama da juyin halitta a wasu fannonin AI, kamar motsi daga rabe-raben ImageNet zuwa ƙarin ma'auni na tunani na gani (misali, VQA, GQA).

Ƙarfi & Aibi: Ƙarfinsa shine bayyanarsa da zargi mai aiki. Rarraba matakai uku yana da kyau don jagorantar bincike na gaba. Duk da haka, aibin takardar, na gama gari ga takardun bayanai, shine alkawari na asali. Gwaji na gaske shine ko al'umma ta karɓe shi. Shin zai zama "COCO" na labarin fim, ko kuwa ya ƙare? Bugu da ƙari, yayin da bayanan harsuna biyu ke da fa'ida, rinjayen Turanci/Sinanci na iya iyakance al'adu da bambancin harshe a cikin salon labari—wani batu mara sauƙi ga aikin da ke da alaƙa da al'ada.

Fahimta Mai Aiki: Ga masu bincike: Dakatar da bin riba kaɗan akan ma'auni mara kyau. Yi amfani da matakan Movie101v2 don tsara sabbin samfura. Wannan yana nuna motsi daga samfuran bayyana ƙarshe-zuwa-ƙarshe zuwa tsarin sassa tare da takamaiman sassan bin diddigin halaye da injinan taƙaita labari, watakila an yi wahayi daga ka'idar labari na gargajiya. Ga masu saka hannun jari & ƙungiyoyin samfur: Ka rage tsammanin. Gaskiya, ingantaccen, atomatik AD don fina-finai na sabani manufa ce "mai ban sha'awa" wacce ta kasance mai nisa. Aikace-aikace na ɗan gajeren lokaci za su iyakance ga abun ciki mai tsari ko tsarin ɗan adam-a-cikin-madauki. Takardar a ɓoye tana jayayya cewa ci gaba na gaba ba zai zo daga haɓaka sigogi kaɗai ba, amma daga ƙirƙira a cikin tsarin samfuri da bayanan horo da aka ƙera musamman don hankalin labari.