0

I just found out about beautifulsoup(4). I have a lot of links and I want to print the <p> tag of multiple websites at once, but I don't know how to do it as I'm a beginner. I can't really find anything on stackoverflow what fits for me too.
Something like this doesn't work:

from bs4 import BeautifulSoup
import requests
import warnings

warnings.filterwarnings("ignore", category=UserWarning, module='bs4')
url = ["http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://google.com=text&format=text", "http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://example.com&format=text&format=text"]

# add header
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, "lxml")
print( soup.find('p').text )

Error I get with this (I didn't expect it to work anyways(Giving me a possible duplicate answer to this error won't help me, read my question in the title first):

Traceback (most recent call last):
  File "C:\Users\Gebruiker\Desktop\apitoshortened.py", line 10, in <module>
    r = requests.get(url, headers=headers)
  File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 640, in send
    adapter = self.get_adapter(url=request.url)
  File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 731, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for '['http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://google.com=text&format=text', 'http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://example.com&format=text&format=text']'

I didn't really expect it to be so simple tough, any help wouldbe appreciated!

BabyCoder20
  • 25
  • 4
  • @Trenton_M Did you even read what I wrote? I already have that fixed, I have the problem of not being able to print the p of multiple urls, this has nothing to do with the ``` – BabyCoder20 Sep 03 '19 at 15:52
  • use `for`-loop to work with list of links (or any other list) – furas Sep 03 '19 at 16:01
  • Possible duplicate of [No connection adapters were found for Python3-Requests](https://stackoverflow.com/questions/43604304/no-connection-adapters-were-found-for-python3-requests) – fixatd Sep 03 '19 at 16:03

2 Answers2

0

if you have list then use for loop 

for item in url:
    r = requests.get(item, headers=headers)
    soup = BeautifulSoup(r.content, "lxml")
    print(soup.find('p').text)

By the way: your url doesn't return any HTML but some text with link - so code can't find <p>.

See this returned text

for item in url:
    r = requests.get(item, headers=headers)
    print(r.text)

Result

https://fc.lc/C4FNiXbY
furas
  • 134,197
  • 12
  • 106
  • 148
0

Use for loop and then check if p tag present.If so print the text.

from bs4 import BeautifulSoup
import requests
import warnings

warnings.filterwarnings("ignore", category=UserWarning, module='bs4')
urls = ["http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://google.com=text&format=text", "http://fc.lc/api?api=9053290fd05b5e5eb091b550078fa1e30935c92c&url=https://wow-ht.ml?s=https://cutlinks.pro/api?api=e6a8809e51daedcf30d9d6270fd0bfeba73c1dcb&url=https://example.com&format=text&format=text"]

# add header
for url in urls:
 headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}
 r = requests.get(url, headers=headers)
 soup = BeautifulSoup(r.content, "lxml")
 if soup.find('p'):
    print( soup.find('p').text)
KunduK
  • 32,888
  • 5
  • 17
  • 41