python - Selenium, trouble with the Section tag -
i have been trying scrape website fanduel, have had been having trouble 'section' tag. want access data each player. here code far:
from selenium import webdriver pandas.io.html import read_html driver = webdriver.firefox() driver.get(https://www.fanduel.co.uk/fixtures/211/lineups/create?contestid=211-4550213') table = driver.find_element_by_xpath('//*[@id="root"]/div/main/div/div') <section class="lineup__main lineup__main--show_fixturelist_players"> <section class="lineup__pitch_view"> <div class="lineup__pitch" style="position: relative;"> <div style="overflow: visible; height: 0px; width: 0px;"> <div class="pitch_view pitch_view--3d" style="left: 58.5px; top: 0px; width: 700px; height: 535px;"><div class="pitch_view__wrapper" style="transform: rotatex(90deg) translatey(-20px);"><div class="pitch_view__pitch"><div class="pitch_view__position"><div class="lineup_position lineup_position--3d lineup_position--g lineup_position--active"><div class="lineup_position__label">gk</div><ul class="lineup_position__players"></ul></div></div><div class="pitch_view__position"><div class="lineup_position lineup_position--3d lineup_position--d"><div class="lineup_position__label">def</div><ul class="lineup_position__players"></ul></div></div><div class="pitch_view__position"><div class="lineup_position lineup_position--3d lineup_position--c"><div class="lineup_position__label">mid</div><ul class="lineup_position__players"></ul></div></div><div class="pitch_view__position"><div class="lineup_position lineup_position--3d lineup_position--f"><div class="lineup_position__label">fwd</div><ul class="lineup_position__players"></ul></div></div></div></div></div></div><div class="resize-triggers"><div class="expand-trigger"><div style="width: 848px; height: 566px;"></div></div><div class="contract-trigger"></div></div></div></section><section class="lineup__fixturelist_players" style="transform: translatey(0%);"><header class="fixturelist_players__filters"><ul class="position_filter"><li class="position_filter__item"><button type="button" class="position_filter__button position_filter__button--selected"><span class="position_filter__label">gk</span><small class="position_filter__count">0</small></button></li><li class="position_filter__item"><button type="button" class="position_filter__button"><span class="position_filter__label">def</span><small class="position_filter__count">0</small></button></li><li class="position_filter__item"><button type="button" class="position_filter__button"><span class="position_filter__label">mid</span><small class="position_filter__count">0</small></button></li><li class="position_filter__item"><button type="button" class="position_filter__button"><span class="position_filter__label">fwd</span><small class="position_filter__count">0</small></button></li></ul><div class="player_filters"><div class="player_filters__search"><div class="player_search search_field"><span class="search_field__icon"><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player_search-search" style="fill: currentcolor; vertical-align: middle; width: 17px; height: 17px;"><title id="icon-aria-label-player_search-search"></title><g><path d="m16.854,16.149l-6.274-6.274c0.885-1.045,1.421-2.396,1.421-3.872c0-3.314-2.686-6-6-6c-3.314,0-6,2.686-6,6s2.686,6,6,6c1.477,0,2.827-0.536,3.872-1.421l6.274,6.274c0.098,0.098,0.226,0.146,0.354,0.146c0.128,0,0.256-0.049,0.354-0.146c17.049,16.661,17.049,16.345,16.854,16.149zm1,6.003c0-2.757,2.243-5,5-5s5,2.243,5,5s-2.243,5-5,5s1,8.76,1,6.003z"></path></g></svg></span><input type="text" role="search" class="search_field__input" value="" placeholder="search player or team"></div></div><div class="player_filters__sort"><section class="player_sort"><button type="button" class="player_sort__button pointer"><!-- react-text: 173 -->price hi-lo<!-- /react-text --><span class="player_sort__caret"><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player-sort-caret" style="fill: currentcolor; vertical-align: middle; width: 1.7rem; height: 1.7rem;"><title id="icon-aria-label-player-sort-caret"></title><g><path d="m3,6 l8.5,11 l14,6 l3,6 z"></path></g></svg></span></button></section></div></div></header><div class="fixturelist_players__list" style="position: relative;"><div style="overflow: visible; height: 0px; width: 0px;"><div aria-label="grid" class="reactvirtualized__grid reactvirtualized__list player_list__virtual_scroll with-data" role="grid" tabindex="0" style="box-sizing: border-box; direction: ltr; height: 242.203px; position: relative; width: 847px; will-change: transform; overflow: auto;"><div class="reactvirtualized__grid__innerscrollcontainer" style="width: auto; height: 4970px; max-width: 847px; max-height: 4970px; overflow: hidden;"><div id="player_211-67921" class="player_list__item" data-position="g" style="height: 71px; left: 0px; position: absolute; top: 0px; width: 100%;"><div class="player_card player_card--compact"><header class="player_card__header"><button type="button" class="pointer player_card__fullname">heaton</button></header><div class="player_card__body"><div class="player_card__aside"><div class="player_shirt player_card__image"><img class="player_shirt__image image image--visible" src="https://cdn.fanduel.co.uk/playerimages/soc/9521.svg" alt="burnley" title="burnley"></div></div><div class="player_card_data"><div class="player_card_data__item player_card_data__item--fixture"><div class="data_chunk data_chunk--zeta" id="211-67921_fixture"><div class="data_chunk__value" aria-labelledby="211-67921_fixture_label"><div class="fixture_details__teams"><span class="fixture_details__team fixture_details__team--home fixture_details__team--player" title="burnley"><span class="fixture_details__team_code">brn</span></span><span class="fixture_details__separator condensed"> v </span><span class="fixture_details__team fixture_details__team--away" title="tottenham hotspur"><span class="fixture_details__team_code">tot</span></span></div></div><div class="data_chunk__label condensed" id="211-67921_fixture_label"><time class="fixture_details__start_time" datetime="2017-04-01t14:00:00z">sat 3:00pm</time></div></div></div><div class="player_card_data__item player_card_data__item--played"><div class="data_chunk data_chunk--zeta" id="211-67921_matches"><div class="data_chunk__value" aria-labelledby="211-67921_matches_label">26</div><div class="data_chunk__label condensed" id="211-67921_matches_label">matches</div></div></div><div class="player_card_data__item player_card_data__item--fppg"><div class="data_chunk data_chunk--zeta" id="211-67921_fppg"><div class="data_chunk__value" aria-labelledby="211-67921_fppg_label">15.66</div><div class="data_chunk__label condensed" id="211-67921_fppg_label">avg pts</div></div></div></div><button class="player_action gamma player_action--add"><div class="data_chunk data_chunk--zeta data_chunk--right"><div class="data_chunk__value">£8m</div><div class="data_chunk__label condensed">price</div></div><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player-action-circle-211-67921" style="fill: currentcolor; vertical-align: middle; width: 2.5rem; height: 2.5rem;"><title id="icon-aria-label-player-action-circle-211-67921"></title><g><path d="m8.5,0c3.806,0,0,3.806,0,8.5s3.806,17,8.5,17s17,13.194,17,8.5s13.194,0,8.5,0z m8.5,16c4.365,16,1,12.635,1,8.5s4.365,1,8.5,1s16,4.365,16,8.5s12.635,16,8.5,16z"></path><path d="m12.5,8h9v4.5c9,4.224,8.776,4,8.5,4s8,4.224,8,4.5v8h4.5c4.224,8,4,8.224,4,8.5s4.224,9,4.5,9h8v3.5c8,12.776,8.224,13,8.5,13s9,12.776,9,12.5v9h3.5c12.776,9,13,8.776,13,8.5s12.776,8,12.5,8z"></path></g></svg></button></div></div></div><div id="player_211-68010" class="player_list__item" data-position="g" style="height: 71px; left: 0px; position: absolute; top: 71px; width: 100%;"><div class="player_card player_card--compact"><header class="player_card__header"><button type="button" class="pointer player_card__fullname">boruc</button></header><div class="player_card__body"><div class="player_card__aside"><div class="player_shirt player_card__image"><img class="player_shirt__image image image--visible" src="https://cdn.fanduel.co.uk/playerimages/soc/9513.svg" alt="bournemouth" title="bournemouth"></div></div><div class="player_card_data"><div class="player_card_data__item player_card_data__item--fixture"><div class="data_chunk data_chunk--zeta" id="211-68010_fixture"><div class="data_chunk__value" aria-labelledby="211-68010_fixture_label"><div class="fixture_details__teams"><span class="fixture_details__team fixture_details__team--home" title="southampton"><span class="fixture_details__team_code">sou</span></span><span class="fixture_details__separator condensed"> v </span><span class="fixture_details__team fixture_details__team--away fixture_details__team--player" title="bournemouth"><span class="fixture_details__team_code">bou</span></span></div></div><div class="data_chunk__label condensed" id="211-68010_fixture_label"><time class="fixture_details__start_time" datetime="2017-04-01t16:30:00z">sat 5:30pm</time></div></div></div><div class="player_card_data__item player_card_data__item--played"><div class="data_chunk data_chunk--zeta" id="211-68010_matches"><div class="data_chunk__value" aria-labelledby="211-68010_matches_label">27</div><div class="data_chunk__label condensed" id="211-68010_matches_label">matches</div></div></div><div class="player_card_data__item player_card_data__item--fppg"><div class="data_chunk data_chunk--zeta" id="211-68010_fppg"><div class="data_chunk__value" aria-labelledby="211-68010_fppg_label">11.40</div><div class="data_chunk__label condensed" id="211-68010_fppg_label">avg pts</div></div></div></div><button class="player_action gamma player_action--add"><div class="data_chunk data_chunk--zeta data_chunk--right"><div class="data_chunk__value">£7.8m</div><div class="data_chunk__label condensed">price</div></div><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player-action-circle-211-68010" style="fill: currentcolor; vertical-align: middle; width: 2.5rem; height: 2.5rem;"><title id="icon-aria-label-player-action-circle-211-68010"></title><g><path d="m8.5,0c3.806,0,0,3.806,0,8.5s3.806,17,8.5,17s17,13.194,17,8.5s13.194,0,8.5,0z m8.5,16c4.365,16,1,12.635,1,8.5s4.365,1,8.5,1s16,4.365,16,8.5s12.635,16,8.5,16z"></path><path d="m12.5,8h9v4.5c9,4.224,8.776,4,8.5,4s8,4.224,8,4.5v8h4.5c4.224,8,4,8.224,4,8.5s4.224,9,4.5,9h8v3.5c8,12.776,8.224,13,8.5,13s9,12.776,9,12.5v9h3.5c12.776,9,13,8.776,13,8.5s12.776,8,12.5,8z"></path></g></svg></button></div></div></div><div id="player_211-68390" class="player_list__item" data-position="g" style="height: 71px; left: 0px; position: absolute; top: 142px; width: 100%;"><div class="player_card player_card--compact"><header class="player_card__header"><button type="button" class="pointer player_card__fullname">robles</button></header><div class="player_card__body"><div class="player_card__aside"><div class="player_shirt player_card__image"><img class="player_shirt__image image image--visible" src="https://cdn.fanduel.co.uk/playerimages/soc/9512.svg" alt="everton" title="everton"></div></div><div class="player_card_data"><div class="player_card_data__item player_card_data__item--fixture"><div class="data_chunk data_chunk--zeta" id="211-68390_fixture"><div class="data_chunk__value" aria-labelledby="211-68390_fixture_label"><div class="fixture_details__teams"><span class="fixture_details__team fixture_details__team--home" title="liverpool"><span class="fixture_details__team_code">liv</span></span><span class="fixture_details__separator condensed"> v </span><span class="fixture_details__team fixture_details__team--away fixture_details__team--player" title="everton"><span class="fixture_details__team_code">eve</span></span></div></div><div class="data_chunk__label condensed" id="211-68390_fixture_label"><time class="fixture_details__start_time" datetime="2017-04-01t11:30:00z">sat 12:30pm</time></div></div></div><div class="player_card_data__item player_card_data__item--played"><div class="data_chunk data_chunk--zeta" id="211-68390_matches"><div class="data_chunk__value" aria-labelledby="211-68390_matches_label">14</div><div class="data_chunk__label condensed" id="211-68390_matches_label">matches</div></div></div><div class="player_card_data__item player_card_data__item--fppg"><div class="data_chunk data_chunk--zeta" id="211-68390_fppg"><div class="data_chunk__value" aria-labelledby="211-68390_fppg_label">16.20</div><div class="data_chunk__label condensed" id="211-68390_fppg_label">avg pts</div></div></div></div><button class="player_action gamma player_action--add"><div class="data_chunk data_chunk--zeta data_chunk--right"><div class="data_chunk__value">£7.6m</div><div class="data_chunk__label condensed">price</div></div><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player-action-circle-211-68390" style="fill: currentcolor; vertical-align: middle; width: 2.5rem; height: 2.5rem;"><title id="icon-aria-label-player-action-circle-211-68390"></title><g><path d="m8.5,0c3.806,0,0,3.806,0,8.5s3.806,17,8.5,17s17,13.194,17,8.5s13.194,0,8.5,0z m8.5,16c4.365,16,1,12.635,1,8.5s4.365,1,8.5,1s16,4.365,16,8.5s12.635,16,8.5,16z"></path><path d="m12.5,8h9v4.5c9,4.224,8.776,4,8.5,4s8,4.224,8,4.5v8h4.5c4.224,8,4,8.224,4,8.5s4.224,9,4.5,9h8v3.5c8,12.776,8.224,13,8.5,13s9,12.776,9,12.5v9h3.5c12.776,9,13,8.776,13,8.5s12.776,8,12.5,8z"></path></g></svg></button></div></div></div><div id="player_211-68261" class="player_list__item" data-position="g" style="height: 71px; left: 0px; position: absolute; top: 213px; width: 100%;"><div class="player_card player_card--compact"><header class="player_card__header"><button type="button" class="pointer player_card__fullname">pickford</button></header><div class="player_card__body"><div class="player_card__aside"><div class="player_shirt player_card__image"><img class="player_shirt__image image image--visible" src="https://cdn.fanduel.co.uk/playerimages/soc/9507.svg" alt="sunderland" title="sunderland"></div></div><div class="player_card_data"><div class="player_card_data__item player_card_data__item--fixture"><div class="data_chunk data_chunk--zeta" id="211-68261_fixture"><div class="data_chunk__value" aria-labelledby="211-68261_fixture_label"><div class="fixture_details__teams"><span class="fixture_details__team fixture_details__team--home" title="watford"><span class="fixture_details__team_code">wat</span></span><span class="fixture_details__separator condensed"> v </span><span class="fixture_details__team fixture_details__team--away fixture_details__team--player" title="sunderland"><span class="fixture_details__team_code">sun</span></span></div></div><div class="data_chunk__label condensed" id="211-68261_fixture_label"><time class="fixture_details__start_time" datetime="2017-04-01t14:00:00z">sat 3:00pm</time></div></div></div><div class="player_card_data__item player_card_data__item--played"><div class="data_chunk data_chunk--zeta" id="211-68261_matches"><div class="data_chunk__value" aria-labelledby="211-68261_matches_label">19</div><div class="data_chunk__label condensed" id="211-68261_matches_label">matches</div></div></div><div class="player_card_data__item player_card_data__item--fppg"><div class="data_chunk data_chunk--zeta" id="211-68261_fppg"><div class="data_chunk__value" aria-labelledby="211-68261_fppg_label">16.37</div><div class="data_chunk__label condensed" id="211-68261_fppg_label">avg pts</div></div></div></div><button class="player_action gamma player_action--add"><div class="data_chunk data_chunk--zeta data_chunk--right"><div class="data_chunk__value">£7.5m</div><div class="data_chunk__label condensed">price</div></div><svg class="icon" x="0px" y="0px" viewbox="0 0 17 17" role="img" aria-labelledby="icon-aria-label-player-action-circle-211-68261" style="fill: currentcolor; vertical-align: middle; width: 2.5rem; height: 2.5rem;"><title id="icon-aria-label-player-action-circle-211-68261"></title><g><path d="m8.5,0c3.806,0,0,3.806,0,8.5s3.806,17,8.5,17s17,13.194,17,8.5s13.194,0,8.5,0z m8.5,16c4.365,16,1,12.635,1,8.5s4.365,1,8.5,1s16,4.365,16,8.5s12.635,16,8.5,16z"></path><path d="m12.5,8h9v4.5c9,4.224,8.776,4,8.5,4s8,4.224,8,4.5v8h4.5c4.224,8,4,8.224,4,8.5s4.224,9,4.5,9h8v3.5c8,12.776,8.224,13,8.5,13s9,12.776,9,12.5v9h3.5c12.776,9,13,8.776,13,8.5s12.776,8,12.5,8z"></path></g></svg></button></div></div></div></div></div></div><div class="resize-triggers"><div class="expand-trigger"><div style="width: 848px; height: 243px;"></div></div><div class="contract-trigger"></div></div></div></section></section>
required content seem dynamically generated, might need wait until javascript
code executed.
try use below code , let me know result:
from selenium.webdriver.common.by import selenium.webdriver.support import expected_conditions ec selenium.webdriver.support.ui import webdriverwait wait selenium import webdriver pandas.io.html import read_html driver = webdriver.firefox() driver.get(https://www.fanduel.co.uk/fixtures/211/lineups/create?contestid=211-4550213') table = wait(driver, 10).until(ec.presence_of_element_located((by.xpath, '//section[contains(@class,"lineup__main")]')))
this should allow find target section
right after it's generated , present in dom
Comments
Post a Comment