请教大神，一个爬虫问题

Question

import requests
import re
url = 'http://s.taobao.com/search?q=书包&s=0'
def gethtmltext(url, timeout = 30):

    kv = {'user-agent':'Mozilla/5.0'}
    r = requests.get(url,headers = kv)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
    return r.text
gethtmltext('http://s.taobao.com/search?q=书包&s=0')

用pycharm返回的的text是这样的

<script>
    // 千牛不提示
    var isQianniu = navigator.userAgent.toLowerCase().indexOf('qianniu') !== -1;

    var checkValue = "cookieCheck"+"="+"57041";
    if(!isQianniu && document.cookie != null && document.cookie.indexOf(checkValue) < 0){
        alert("您的浏览器限制了第三方Cookie，这将影响您正常登录，您可以更改浏览器的隐私设置，解除限制后重试。");
    }
</script>


    <div class="hd">

     <div class="login-switch" onselectstart="return false;">
                        <i class="iconfont quick" id="J_Static2Quick" onclick="javascript:goldlog.record('/member.13.1','','','H46777383')">&#xe628;</i>
                        <i class="iconfont static" id="J_Quick2Static">&#xe626;</i>
    </div>
    <div class="login-tip">
        <div class="poptip">
            <div class="poptip-arrow">
                <em></em>
                <span></span>
            </div>
            <div class="poptip-content">
                <i class="iconfont">&#xe625;</i>扫码登录更安全
            </div>
        </div>
    </div>                                 

    </div>
<div class="bd">
        <!--登录的错误信息结束-->

         <div id="J_QuickLogin" class="ww-login hidden">
            <form action="" class="ww-form">
                <div class="login-title">
                    选择其中一个已登录的账户
                </div>

                <div class="ww-userlist">

                </div>
                <div class="trigger">

                </div>
                <div class="submit">
                    <button type="submit" class="J_Submit"  id="J_SubmitQuick">登 录</button>
                </div>
                <div class="other-login">
                    <a href=""  class="light-link" id="J_Sso2Static">使用其他账户登录</a>
                </div>
            </form>
        </div>
        <div id="J_MiserLogin" class="ww-login hidden">
                        <form action="" class="ww-form">
                            <input type="hidden" id="x_token" value="">
                            <div class="login-title">
                                选择其中一个已登录的账户
                            </div>

                            <div class="ww-userlist">

                            </div>
                            <div class="trigger">

                            </div>
                            <div class="submit">
                                <p><input type="checkbox" id="J_Mdelete"> 不再记住密码</p>
                                <button type="submit"  tabindex="5"
                                        id="J_SubmitMiser">登 录</button>
                            </div>
                            <div class="other-login">
                                <a href="" title="使用其他账户登录" class="light-link" id="J_Miser2Static">使用其他账户登录</a>
                            </div>
                        </form>
          </div>

        <!--标准登录框-->
        <div class="static-form " id="J_StaticForm">
            <div class="login-title">


                密码登录

            </div>

            <form action="/member/login.jhtml?redirectURL=http%3A%2F%2Fs.taobao.com%2Fsearch%3Fq%3D%25E4%25B9%25A6%25E5%258C%2585%26s%3D0"  method="post" id="J_Form">
            <div id="J_Message"
                style="display:none;"
                class="login-msg error">
                <i class="iconfont">&#xe604;</i>

            <p class="error"></p>

             </div>
                <!-- 手机号登录 -->



                <div class="field ph-hide username-field ">
                    <label for="TPL_username_1"> <i class="iconfont" title="会员名">&#xe601;</i></label>
                    <span class="ph-label">会员名/邮箱/手机号</span> <input type="text"
                                                                             name="TPL_username" id="TPL_username_1"
                        class="login-text J_UserName"
                                                                             value=""
                                                                             maxlength="32" tabindex="1" aria-label="会员名/邮箱/手机号"/>
                </div>

但是我自己用浏览器搜索这个url的时候，源代码是有商品信息的，请教大神如何解决呢？

Answer 1

解决方法：通过浏览器查看 登录状态的cookie信息，假设为 Cookie: a=b,c=d
然后在gethtmltext 函数的请求头 kv中添加键值对，也就是在kv那行的下面添加 kv['Cookie']='a=b,c=d'
这时再去爬取，应该可以爬到需要的数据。