python - Why doesn't findall work on the table data I pulled from javascript (using python3) -


i can 2 parts of want separately, not together. i'm using anaconda 3 distribution.

the tables want dynamically loaded javascript tables , want extract them use them in pandas , sqllite3.

this give me text output, delivered separate lines each cell, of information want:

import sys pyqt5.qtwidgets import qapplication pyqt5.qtcore import qurl pyqt5.qtwebkitwidgets import qwebpage  import bs4 bs import requests import pandas pd  class client(qwebpage):      def __init__(self, url):         self.app = qapplication(sys.argv)          qwebpage.__init__(self)           self.loadfinished.connect(self.on_page_load)         self.mainframe().load(qurl(url))         self.app.exec()      def on_page_load(self):         self.app.quit()    #only run until page loads  f = open('hmarathont.txt', 'w') url = 'http://results.houstonmarathon.com/2017/?page=2&event=mara&pid=search&search%5bclub%5d=%25&search%5bcompany%5d=%25&search%5bnation%5d=%25&search_sort=name'     client_response = client(url)     source = client_response.mainframe().tohtml()    soup =bs.beautifulsoup(source, 'lxml') js_table = soup.find("table", {"class": "list-table"}) js_table_content = js_table.text  f.write(js_table_content)

which great, imagine taking text file , parsing right format (currently it's line each cell , want original format) not best way it, when there way direct table straight pandas.

i have had success getting tables pandas, code wrote below, not dynamically loaded tables. have code works taking csv table data sqlite3, that's why export csv dataframe.

rows = js_table.findall('tr')[1:]  data = {'ovrl_pl' :[],'gndr_pl' :[],'place_div' :[],'raw_name' :[],'gender' :[],'state' :[],'age_cat' :[],'age' :[],'time_net' :[],'time_gun' :[]}  row in rows:     cols = row.find_all('td')      data['ovrl_pl'].append( cols[0].get_text() )      data['gndr_pl'].append( cols[1].get_text() )     data['place_div'].append( cols[2].get_text() )      data['raw_name'].append( cols[3].get_text() )     data['gender'].append( cols[4].get_text() )     data['state'].append( cols[5].get_text() )     data['age_cat'].append( cols[6].get_text() )     data['age'].append( cols[7].get_text() )       data['time_net'].append( cols[8].get_text() )     data['time_gun'].append( cols[9].get_text() )  hmarathon = pd.dataframe(data) hmarathon.tocsv("hmarathon2017test.csv")

the error keep getting @ point try use findall on js_table. nonetype, i'm guessing existed briefly , gone? (or maybe don't understand of :-/ ).

i managed capture table cell contents text, why can't capture dictionary?


Comments

Post a Comment

Popular posts from this blog

javascript - Clear button on addentry page doesn't work -

this addentry blog page have made scratch , there underlying problems code particularly clear button supposed clear text entered when button clicked on doesn't work have saved html file problem. also 1 other problem submit button works doesn't pass of text index file have written php , reason whenever click on submit button redirects me index page without posting entry have made. <!doctype html> <html> <head> <script> function message() { alert("clear form?"); } </script> <style> body {background-color: beige;} h1 {color: black;} p {color: black;} textarea { width: 50%; height: 150px; padding: 12px 20px; box-sizing: border-box; border: 2px solid #ccc; border-radius: 4px; background-color: white; font-size: 16px; resize: none; } div { padding-left: 20px; } </style> </head> <meta charset="utf-8"> <link rel="stylesheet" href=".
Read more

c# - Selenium Authentication Popup preventing driver close or quit -

i have selenium automated task setup login website retrieve information. it's basic in automation. logging in , clicking 2 links. running issue proving problem. the site in question uses authentication required popup not javascript popup rather regular windows auth popup. have read isn't handled out of box selenium. i using " http://username:password@url.com " site every , site has issues on end prompt me again username/password randomly. have enter username/password twice in row when navigating there hand. since task loops continuously few hours picking needed info figured use webdriverwait verify element present on page. if authentication popup appears, element isn't present. no need care run, start over. the problem having when authentication popup displayed , hit timeout limit want close driver/browser window can start task on again. however, driver.close() , driver.quit() not doing anything. browser remains open , unloaded page still there login p
Read more

tensorflow when input_data MNIST_data , zlib.error: Error -3 while decompressing: invalid block type -

i'm geting started tensorflow,followed tutorials,but cannot download mnist data. import tensorflow tf tensorflow.examples.tutorials.mnist import input_data mnist=input_data.read_data_sets("mnist_data/",one_hot=true) print("download done!") error information: extracting mnist_data/train-images-idx3-ubyte.gz traceback (most recent call last): file "/home/sj/program/tesorflow_try.py", line 6, in <module> mnist=input_data.read_data_sets("mnist_data/",one_hot=true) file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 213, in read_data_sets train_images = extract_images(f) file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 60, in extract_images buf = bytestream.read(rows * cols * num_images) file "/usr/lib/python2.7/gzip.py", line 268, in read self._read(readsize) file
Read more