Menene Fayil na Robots.txt? Duk abin da kuke Buƙatar Rubutu, ƙaddamarwa, da Sake jan Fayil ɗin Robots don SEO
Mun rubuta cikakken labarin akan yadda injunan bincike suke ganowa, rarrafe, da baje kolin gidajen yanar gizonku. Mataki na tushe a cikin wannan tsari shine robots.txt
fayil, ƙofa don injin bincike don rarrafe rukunin yanar gizon ku. Fahimtar yadda ake gina fayil ɗin robots.txt da kyau yana da mahimmanci a haɓaka injin bincike (SEO).
Wannan kayan aiki mai sauƙi amma mai ƙarfi yana taimakawa masu kula da gidan yanar gizon sarrafa yadda injunan bincike ke hulɗa da gidajen yanar gizon su. Fahimtar da ingantaccen amfani da fayil na robots.txt yana da mahimmanci don tabbatar da ingantaccen firikwensin gidan yanar gizo da mafi kyawun gani a sakamakon injin bincike.
Menene Fayil na Robots.txt?
Fayil na robots.txt fayil ne na rubutu da ke cikin tushen adireshin gidan yanar gizon. Babban manufarsa ita ce jagorar injunan bincike game da waɗanne sassan rukunin yanar gizon ya kamata ko bai kamata a rarrafe da ƙididdige su ba. Fayil ɗin yana amfani da ƙa'idar keɓancewar Robots (wakili), daidaitattun gidajen yanar gizo suna amfani da su don sadarwa tare da masu rarrafe yanar gizo da sauran robots na yanar gizo.
REP ba ma'auni ba ne na Intanet amma ana karɓar ko'ina kuma ana samun goyan bayan manyan injunan bincike. Mafi kusa ga ma'auni da aka karɓa shine takaddun bayanai daga manyan injunan bincike kamar Google, Bing, da Yandex. Don ƙarin bayani, ziyartar Bayanan Bayani na Robots.txt na Google an bada shawarar.
Me yasa Robots.txt Mahimmanci ga SEO?
- Sarrafa Rarrafe: Robots.txt yana ba masu gidan yanar gizon damar hana injunan bincike shiga takamaiman sassan rukunin yanar gizon su. Wannan yana da amfani musamman don keɓance kwafin abun ciki, wurare masu zaman kansu, ko sassan da ke da mahimman bayanai.
- Ingantattun Kasafin Kuɗi: Injin bincike suna ware kasafin kuɗi don kowane gidan yanar gizo, adadin shafukan da injin binciken bot zai yi rarrafe akan rukunin yanar gizo. Ta hanyar hana ɓangarori marasa mahimmanci ko ƙasa da mahimmanci, robots.txt yana taimakawa haɓaka wannan kasafin kuɗaɗen rarrafe, yana tabbatar da cewa an zazzage wasu mahimman shafuka da ƙididdiga.
- Ingantattun Lokacin Loda Gidan Yanar Gizo: Ta hanyar hana bots daga samun damar albarkatun da ba su da mahimmanci, robots.txt na iya rage nauyin uwar garken, mai yuwuwar inganta lokacin lodin shafin, muhimmin mahimmanci a cikin SEO.
- Hana Fitar da Shafukan da Ba Jama'a ba: Yana taimakawa wajen kiyaye wuraren da ba na jama'a ba (kamar wuraren tsarawa ko wuraren ci gaba) daga yin lissafin da bayyana a sakamakon bincike.
Robots.txt Muhimman Dokokin da Amfaninsu
- Izinin: Ana amfani da wannan umarnin don tantance shafuka ko sassan rukunin yanar gizon da ya kamata masu rarrafe su shiga. Misali, idan gidan yanar gizon yana da sashe na musamman don SEO, umarnin 'Ba da izini' na iya tabbatar da rarrafe.
Allow: /public/
- Ba da: Akasin 'Bada', wannan umarni yana ba da umarni injunan bincike kar su rarrafe wasu sassan gidan yanar gizon. Wannan yana da amfani ga shafukan da ba su da darajar SEO, kamar shafukan shiga ko fayilolin rubutun.
Disallow: /private/
- Katunan daji: Ana amfani da kati don daidaita tsarin. Alamar alama (*) tana wakiltar kowane jerin haruffa, kuma alamar dala ($) tana nufin ƙarshen URL. Waɗannan suna da amfani don tantance kewayon URLs.
Disallow: /*.pdf$
- Taswirorin yanar gizo: Haɗe da wurin taswirar rukunin yanar gizo a cikin robots.txt yana taimakawa injunan bincike su nemo da zazzage duk mahimman shafuka akan rukunin yanar gizo. Wannan yana da mahimmanci ga SEO kamar yadda yake taimakawa cikin sauri da cikakkiyar fihirisar rukunin yanar gizo.
Sitemap: https://martech.zone/sitemap_index.xml
Karin Dokokin Robots.txt da Amfaninsu
- Mai amfani: Ƙayyade wanne mai rarrafe dokar ta shafi. 'Wakilin mai amfani: *' yana amfani da ƙa'idar ga duk masu rarrafe. Misali:
User-agent: Googlebot
- Noindex: Duk da yake ba wani ɓangare na ƙa'idar robots.txt ba, wasu injunan bincike sun fahimci a Noindex umarni a cikin robots.txt a matsayin umarni don kada a fidda takamaiman URL.
Noindex: /non-public-page/
- Jan hankali-jinkiri: Wannan umarnin yana buƙatar masu rarrafe su jira takamaiman adadin lokaci tsakanin hits zuwa uwar garken ku, mai amfani ga shafuka masu matsalar lodin uwar garken.
Crawl-delay: 10
Yadda Ake Gwaji Fayil na Robots.txt
Ko da yake an binne shi a ciki Shafin Farko na Google, search console yana ba da gwajin fayil na robots.txt.
Hakanan zaka iya sake shigar da Fayil na Robots.txt ta danna dige guda uku a hannun dama kuma zaɓi Nemi Sake Zama.
Gwada ko Sake Shigar da Fayil na Robots.txt
Za a iya Amfani da Fayil na Robots.txt Don Sarrafa Bots na AI?
Ana iya amfani da fayil ɗin robots.txt don ayyana ko AI bots, gami da crawlers na yanar gizo da sauran bots masu sarrafa kansu, na iya yin rarrafe ko amfani da abun ciki akan rukunin yanar gizon ku. Fayil ɗin yana jagorantar waɗannan bots, yana nuna waɗanne sassan gidan yanar gizon da aka ba su izini ko aka hana su shiga. Tasirin robots.txt sarrafa halayen bots na AI ya dogara da dalilai da yawa:
- Riko da Protocol: Yawancin mashahuran injin bincike da yawa da sauran bots na AI suna mutunta dokokin da aka saita a ciki
robots.txt
. Koyaya, yana da mahimmanci a lura cewa fayil ɗin ya fi buƙata fiye da ƙuntatawa mai tilastawa. Bots na iya yin watsi da waɗannan buƙatun, musamman waɗanda ƙananan hukumomi ke sarrafa su. - Ƙayyadaddun Umarni: Kuna iya ƙayyade umarni daban-daban don bots daban-daban. Misali, kuna iya ƙyale ƙayyadaddun bots na AI su rarrafe rukunin yanar gizon ku yayin hana wasu. Ana yin wannan ta amfani da
User-agent
umarni a cikinrobots.txt
misali fayil a sama. Misali,User-agent: Googlebot
zai ƙayyade umarnin don rarrafe na Google, alhaliUser-agent: *
zai shafi duk bots. - gazawar: Duk da yake
robots.txt
zai iya hana bots daga rarrafe takamaiman abun ciki; ba ya ɓoye musu abin da ke ciki idan sun riga sun san URL. Bugu da ƙari, baya bayar da wata hanya don taƙaita amfanin abun ciki da zarar an ja shi. Idan ana buƙatar kariyar abun ciki ko takamaiman ƙuntatawa na amfani, wasu hanyoyin kamar kariyar kalmar sirri ko ƙarin ingantattun hanyoyin sarrafa damar shiga na iya zama dole. - Nau'in Bots: Ba duk bots na AI suna da alaƙa da injunan bincike ba. Ana amfani da bots iri-iri don dalilai daban-daban (misali, tara bayanai, nazari, goge-goge). Hakanan za'a iya amfani da fayil ɗin robots.txt don sarrafa samun dama ga waɗannan nau'ikan bots daban-daban, muddin sun bi REP.
The robots.txt
fayil na iya zama ingantaccen kayan aiki don sigina abubuwan da kuke so game da rarrafe da amfani da abun cikin rukunin yanar gizo ta bots AI. Koyaya, ikonsa yana iyakance ga samar da jagorori maimakon aiwatar da tsauraran matakan samun dama, kuma tasirin sa ya dogara da bin ƙa'idodin keɓancewa na Robots.
Fayil ɗin robots.txt ƙarami ne amma babban kayan aiki a cikin arsenal na SEO. Yana iya yin tasiri sosai ga ganin gidan yanar gizon da aikin injin bincike idan aka yi amfani da shi daidai. Ta hanyar sarrafa waɗanne ɓangarori na rukunin yanar gizon da aka rarrafe da ƙididdige su, masu kula da gidan yanar gizo na iya tabbatar da cewa an ba da fifikon abubuwan su mafi mahimmanci, haɓaka ƙoƙarin SEO da aikin gidan yanar gizo.