Ifayela le-robots.txt elondolozwe empandeni yewebhusayithi yakho izotshela ama-robot ewebhu njengeziphequluli zenjini yokusesha ukuthi yiziphi izinkomba namafayela avunyelwe ukukhasa. Kulula ukusebenzisa ifayela le robots.txt, kodwa kunezinto okufanele uzikhumbule:
- Ama robot webhabhu amnyama azozinakwa ifayela lakho le-robots.txt. Izinhlobo ezivame kakhulu zingama-bots we-malware nama-robot afuna amakheli e-imeyili ukuze avune.
- Abanye abahleli abasha bazobhala ama-robot azinaki ifayela le-robots.txt. Lokhu ngokuvamile kwenziwa ngephutha.
- Noma ubani angabona ifayela lakho le-robots.txt. Bahlale bebizwa ngokuthi i-robots.txt futhi bahlale begcinwe empini yewebhusayithi.
- Okokugcina, uma othile ehlanganisa ifayela noma isiqondisi esingekho ifayela lakho le-robots.txt eliphuma ekhasini elingafaki ifayela layo le-robots.txt, izinjini zokusesha zingayithola noma kunjalo.
Ungasebenzisi amafayela we robots.txt ukufihla noma yini ebalulekile. Esikhundleni salokho, kufanele ufake ulwazi olubalulekile ngemuva kwamaphasiwedi aphephile noma uwashiye ngokuphelele kuwebhu.
Indlela Yokusebenzisa Lezi Amafayela Esibonelo
Kopisha umbhalo kusuka kwisampula esiseduze nalokho ofuna ukukwenza, bese unamathisele kufayela lakho le-robots.txt. Shintsha i-robot, isiqondisi, namagama wefayela ukufanisa ukucushwa kwakho okhethwayo.
Amafayela amabili ayisisekelo Robots.txt
Umsebenzisi-agent: *
Vumela: /
Leli fayela lithi noma iyiphi i-robot (I-user-agent: *) eyifinyelela kufanele ilandele wonke amakhasi kusayithi (Vumela: /).
Umsebenzisi-agent: *
Vumela:
Leli fayela lithi nanoma iyiphi i-robot (I-user-agent: *) eyifinyelela ivumelekile ukubuka wonke amakhasi kusayithi (Vumela:).
Ungenza lokhu ngokushiya ifayela lakho le-robots.txt elingenalutho noma ungenalo nhlobo kusayithi lakho.
Vikela ama-Specific Izincwajana kusuka kumaRobhothi
Umsebenzisi-agent: *
Ungavumeli: / cgi-bin /
Vumela: / ithempeli /
Leli fayela lithi noma iyiphi i-robot (I-user-agent: *) eyifinyelela kufanele izinaki i-directories / cgi-bin / kanye / temp / (Disallow: / cgi-bin / Disallow: / temp /).
Vikela Amakhasi athile avela kumarobhothi
Umsebenzisi-agent: *
Vumela: /jenns-stuff.htm
Ungavumeli: /private.php
Leli fayela lithi noma iyiphi i-robot (I-user-agent: *) eyifinyelela kufanele izinaki amafayela /jenns-stuff.htm kanye /private.php (Ungavumeli: /jenns-stuff.htm Ungavumeli: /private.php).
Vimbela iRobot ethize kusuka ekufinyeleleni indawo yakho
Umsebenzisi-agent: Lycos / xx
Vumela: /
Leli fayela lithi i-Lycos bot (I-User-agent: Lycos / xx) ayivunyelwe ukufinyelela noma kuphi kusayithi (Vumela: /).
Vumela Ukufinyelela Okulodwa Okuqondile KwamaRobhothi
Umsebenzisi-agent: *
Vumela: /
Umsebenzisi-agent: Googlebot
Vumela:
Leli fayela kuqala livumela wonke ama-robot njengalokhu senza ngenhla, bese ivumela i-Googlebot (i-User-agent: i-Googlebot) ukuthi ikwazi ukufinyelela konke (Vumela:).
Hlanganisa amaLandi amaningi ukuze uthole ngokuqondile lokho okufakiwe okufunayo
Nakuba kungcono ukusebenzisa umugqa womsebenzisi womuntu obandakanya kakhulu, njengoMsebenzisi-agent: *, ungaba oqondile njengoba uthanda. Khumbula ukuthi ama-robot afunde ifayela ngokulandelana. Ngakho uma imigqa yokuqala ithi wonke ama-robot avinjiwe kuzo zonke izinto, bese kuthi kamuva efayilini lithi wonke ama-robot avunyelwe ukufinyelela konke, ama-robot azokwazi ukufinyelela konke.
Uma ungaqiniseki ukuthi ngabe ubhale ifayela lakho le-robots.txt ngendlela efanele, ungasebenzisa ama-Webmaster Amathuluzi we-Google ukuhlola ifayela lakho le-robots.txt noma ubhale entsha.