web scraping - Python -- AttributeError: 'NoneType' object has no attribute 'find_all' -


i'm trying image links , description pages in python keep getting attributeerror:

traceback (most recent call last):   file "d:\downloads\test.py", line 78, in <module>     get_detail_book_list( get_book_list() )   file "d:\downloads\test.py", line 59, in get_detail_book_list     image = div.find_all('img') attributeerror: 'nonetype' object has no attribute 'find_all'

here's code:

from bs4 import beautifulsoup selenium import webdriver  class book():     """docstring book"""     def __init__(self):         self.title = ""         self.link = ""         self.image = "" #add new items class         self.des = ""   def get_book_list():          driver = webdriver.phantomjs(executable_path = r'd:\programs\phantomjs\bin\phantomjs.exe')      url = 'https://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3daps&field-keywords=python+programming'      driver.get(url)      soup = beautifulsoup(driver.page_source, 'lxml')      ul = soup.find('ul', {'id':'s-results-list-atf'}) # container whole book list      book_list = []      li in ul.find_all('li', class_ = 's-result-item celwidget'):   # each block product          all_a = li.find_all('a') #find links product detail page         # print all_a[1].text         # print all_a[1]['href']          new_book = book()         new_book.title = all_a[1].text         new_book.link = all_a[1]['href']         book_list.append(new_book)       driver.quit()      return book_list  def get_detail_book_list(book_list):      driver = webdriver.phantomjs(executable_path = r'd:\programs\phantomjs\bin\phantomjs.exe')      b in book_list[0:2]:    #for every book (b) in book list / can try printing first 2 instead of test e.g:                                  #for b in book_list[0:2]:           #url = 'https://www.amazon.com/python-programming-introduction-computer-science/dp/1590282418/ref=sr_1_1?ie=utf8&qid=1473731166&sr=8-1&keywords=python+programming'         url = b.link          driver.get(url)          soup = beautifulsoup(driver.page_source,'lxml')          div = soup.find('div', class_ = 'rscontainer')          image = div.find_all('img')          links in img:              print links['src']          b.image = image # not sure if need @ because same data printed without          description = soup.find('p', class_ =  'read-more-text')          print description.text          b.des = description          print '\n'        driver.quit()  get_detail_book_list( get_book_list() )


-

Comments

Post a Comment

Popular posts from this blog

java - SSE Emitter : Manage timeouts and complete() -

i writing web app multiple listeners(evcentsource sse clients js) connected server . want store sse emitter per connected listener : can done in memory or other means assigning id each client able achieve far now question ; how send response/event specific client connected web app ? while doing sseemmiters stored either getting completed or timed out. how prevent ? how keep sseemmiter available open infinite time (till client closes) , send events selectively .
Read more

jquery - uncaught exception: DataTables Editor - remote hosting of code not allowed -

i'm trying implement crud operation using datatableseditor using datatables.but i'm getting error messages: 1)uncaught exception: datatables editor - remote hosting of code not allowed. please see http://editor.datatables.net details on how purchase editor license 2)typeerror: a.editor undefined 3) $.fn.datatable.editor not constructor" what reason? here configuration: <%@ page language="java" contenttype="text/html; charset=iso-8859-1" pageencoding="iso-8859-1"%> <!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http://www.w3.org/tr/html4/loose.dtd"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> <title>insert title here</title> <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.12/css/jquery.datatables.min.css"> <link re...
Read more

java - How to resolve error - package com.squareup.okhttp3 doesn't exist? -

i trying convert json java object use of gson in maven, following youtube video guidance - https://www.youtube.com/watch?v=vqgghm9pwe0 , theres error occurs when in main class error - package com.squareup.okhttp3 doesn't exist. code below: java package com.codebeasty.json; import com.squareup.okhttp3.okhttpclient; public class main { private static okhttpclient client = new okhttpclient(); public static void main (string [] args) { } } i put in dependency in pom.xml: <dependencies> <dependency> <groupid>com.squareup.okhttp3</groupid> <artifactid>okhttp</artifactid> <version>3.4.2</version> </dependency> <dependency> <groupid>com.google.code.gson</groupid> <artifactid>gson</artifactid> <version>2.8.0</version> </dependency> </dependencies> i dont understand why doesn't recognize com.squareup. there may need download? have downl...
Read more