doutuo3899 2012-07-15 03:21
浏览 59
已采纳

如何停止谷歌抓取未来页面

When I was developing my site. I made a typo in one place, for example, all my pages are dir1/dir2/page.htm/par1-par2, but my typo was dir1/dir2/page/par1-par2 (note: without .htm).

It was in production for 1 day only, but Google is keep crawling those links. How to stop Google doing that?

By the way, that's not 1 page, but hundreds or thousands of pages.

  • 写回答

3条回答 默认 最新

  • doujia3441 2012-07-15 03:35
    关注

    Try use robots.txt to deny access to this page (url)

    http://www.robotstxt.org/robotstxt.html

    http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

    test robots.txt here : http://www.frobee.com/robots-txt-check/

    patterns must begin with / because robots.txt patterns always match absolute URLs. 
    * matches zero or more of any character. 
    $ at the end of a pattern matches the end of the URL; elsewhere $ matches itself. 
    * at the end of a pattern is redundant, because robots.txt patterns always match any URL which begins with the pattern.
    
    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(2条)

报告相同问题?

悬赏问题

  • ¥15 咨询一下有关于王者荣耀赢藏战绩
  • ¥100 求购一套带接口实现实习自动签到打卡
  • ¥50 MacOS 使用虚拟机安装k8s
  • ¥500 亚马逊 COOKIE我如何才能实现 登录一个亚马逊账户 下发新 COOKIE ..我使用下发新COOKIE 导入ADS 指纹浏览器登录,我把账户密码 修改过后,原来下发新COOKIE 不会失效的方式
  • ¥20 玩游戏gpu和cpu利用率特别低,玩游戏卡顿
  • ¥25 oracle中的正则匹配
  • ¥15 关于#vscode#的问题:把软件卸载不会再出现蓝屏
  • ¥15 vimplus出现的错误
  • ¥15 usb无线网卡转typec口
  • ¥30 怎么使用AVL fire ESE软件自带的优化模式来优化设计Soot和NOx?