I need a script that can spider a website and return the list of all crawled pages in plain-text or similar format; which I will submit to search engines as sitemap. Can I use WGET to generate a sitemap of a website? Or is there a PHP script that can do the same?
2条回答 默认 最新
- douyun3631 2011-07-19 13:15关注
wget --spider --recursive --no-verbose --output-file=wgetlog.txt http://somewebsite.com sed -n "s@.\+ URL:\([^ ]\+\) .\+@\1@p" wgetlog.txt | sed "s@&@\&@" > sedlog.txt
This creates a file called
sedlog.txt
that contains all links found on the specified website. You can use PHP or a shell script to convert the text file sitemap into an XML sitemap. Tweak the parameters of the wget command (accept/reject/include/exclude) to get only the links you need.本回答被题主选为最佳回答 , 对您是否有帮助呢?解决 无用评论 打赏 举报
悬赏问题
- ¥15 下图接收小电路,谁知道原理
- ¥15 装 pytorch 的时候出了好多问题,遇到这种情况怎么处理?
- ¥20 IOS游览器某宝手机网页版自动立即购买JavaScript脚本
- ¥15 手机接入宽带网线,如何释放宽带全部速度
- ¥30 关于#r语言#的问题:如何对R语言中mfgarch包中构建的garch-midas模型进行样本内长期波动率预测和样本外长期波动率预测
- ¥15 ETLCloud 处理json多层级问题
- ¥15 matlab中使用gurobi时报错
- ¥15 这个主板怎么能扩出一两个sata口
- ¥15 不是,这到底错哪儿了😭
- ¥15 2020长安杯与连接网探