ajax - Python requests module gets the same results despite incrementing page number -


the thing changes in url page number, incremented after each request.

other selenium or related tools, i’m not sure approach used traverse pages. instinct there may header/query combination data directly, don't know find it.

url = 'http://therunningbug.co.uk/events/find-races.aspx?eventname=&addressregion=&addresscounty=&date=&surface=#sort=date&page='  page = 1  while true:      pagedata = beautifulsoup(requests.get(url + str(page)).content)      articles = pagedata.find('div', {'class':"items-content"})      in articles.find_all('article'):         name = a.find('span', {'itemprop':"name"}).text         d, t = a.find('time').get('datetime').split('t')          timedata = t[:-3]          datedata = d.split('-')         date = (datedata[1] + '/' + datedata[2] + '/' + datedata[0][2:]).strip()         description = a.find('p', {'itemprop':"description"}).text.strip()         weblink = 'http://therunningbug.co.uk' + a.find('a', {'itemprop':"url"}).get('href')         category = a.find('span', {'class':"surface"}).text         location = a.find('span', {'class':"region"}).text + ', ' + a.find('span', {'class':"county"}).text          print name, ' -- name'         print date, ', ', timedata, ' -- date, time'         print description, ' -- description'         print weblink, ' -- website link'         print category, ' -- category'         print location, ' -- location\n'      page += 1

the problem url encoding. can urlencode:

url = 'http://therunningbug.co.uk/events/find-races.aspx' payload = {'page': page} pagedata = beautifulsoup(requests.get(url, params = payload).content)

this works there no complex characters in uri url encode.

url = 'http://therunningbug.co.uk/events/find-races.aspx' pagedata = beautifulsoup(requests.get(url + '?page=' + str(page)).content)

see requests documentation url encoding. http://docs.python-requests.org/en/master/user/quickstart/

complete code:

#!/usr/bin/env python  import requests bs4 import beautifulsoup  page = 1 while true:      url = 'http://therunningbug.co.uk/events/find-races.aspx'     payload = {'page': page}     pagedata = beautifulsoup(requests.get(url, params = payload).content)      articles = pagedata.find('div', {'class':"items-content"})      in articles.find_all('article'):         name = a.find('span', {'itemprop':"name"}).text         d, t = a.find('time').get('datetime').split('t')          timedata = t[:-3]          datedata = d.split('-')         date = (datedata[1] + '/' + datedata[2] + '/' + datedata[0][2:]).strip()         description = a.find('p', {'itemprop':"description"}).text.strip()         weblink = 'http://therunningbug.co.uk' + a.find('a', {'itemprop':"url"}).get('href')         category = a.find('span', {'class':"surface"}).text         location = a.find('span', {'class':"region"}).text + ', ' + a.find('span', {'class':"county"}).text          print name, ' -- name'         print date, ', ', timedata, ' -- date, time'         print description, ' -- description'         print weblink, ' -- website link'         print category, ' -- category'         print location, ' -- location\n'      page += 1

-

Comments

Post a Comment

Popular posts from this blog

java - SSE Emitter : Manage timeouts and complete() -

i writing web app multiple listeners(evcentsource sse clients js) connected server . want store sse emitter per connected listener : can done in memory or other means assigning id each client able achieve far now question ; how send response/event specific client connected web app ? while doing sseemmiters stored either getting completed or timed out. how prevent ? how keep sseemmiter available open infinite time (till client closes) , send events selectively .
Read more

jquery - uncaught exception: DataTables Editor - remote hosting of code not allowed -

i'm trying implement crud operation using datatableseditor using datatables.but i'm getting error messages: 1)uncaught exception: datatables editor - remote hosting of code not allowed. please see http://editor.datatables.net details on how purchase editor license 2)typeerror: a.editor undefined 3) $.fn.datatable.editor not constructor" what reason? here configuration: <%@ page language="java" contenttype="text/html; charset=iso-8859-1" pageencoding="iso-8859-1"%> <!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http://www.w3.org/tr/html4/loose.dtd"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> <title>insert title here</title> <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.12/css/jquery.datatables.min.css"> <link re...
Read more

java - How to resolve error - package com.squareup.okhttp3 doesn't exist? -

i trying convert json java object use of gson in maven, following youtube video guidance - https://www.youtube.com/watch?v=vqgghm9pwe0 , theres error occurs when in main class error - package com.squareup.okhttp3 doesn't exist. code below: java package com.codebeasty.json; import com.squareup.okhttp3.okhttpclient; public class main { private static okhttpclient client = new okhttpclient(); public static void main (string [] args) { } } i put in dependency in pom.xml: <dependencies> <dependency> <groupid>com.squareup.okhttp3</groupid> <artifactid>okhttp</artifactid> <version>3.4.2</version> </dependency> <dependency> <groupid>com.google.code.gson</groupid> <artifactid>gson</artifactid> <version>2.8.0</version> </dependency> </dependencies> i dont understand why doesn't recognize com.squareup. there may need download? have downl...
Read more