Open a web page and write in a text file

Question

I am trying to open a webpage and write the page in a text file. This is the code so far and its not working. Cam anyone give me a general idea about what I am doing wrong?

import urllib
opener = urllib.FancyURLopener({})
f = opener.open("http://www.python.org/")
g = open("data2.txt", "w")
g.write(str(f))      # str() converts to string
g.close()

All I gets when executing the code is text file data2.txt with below line only:

<addinfourl at 43347592L whose fp = <socket._fileobject object at 0x000000000294C480>>

*"It's not working"* is not a sufficient problem description. Please edit your question and state what result you expected, and what result you actually got. — Lukas Graf, Sep 05 '14 at 10:33
See the duplicate I linked - you need to call `f.read()` to actually read the response data. — Lukas Graf, Sep 05 '14 at 10:42
f.read will display the text in command promt, not write in file as I want. NewBie solution serves the purpose. — Abhishek dot py, Sep 05 '14 at 10:54

score 2 · Answer 1 · answered Sep 05 '14 at 11:01

2

There are better tools for this job, like requests. For example

import requests

url = 'http://www.python.org'
r = requests.get(url)

text_file = open("Output.txt", "w")
text_file.write(r.text)
text_file.close()

answered Sep 05 '14 at 11:01

sidj9n

29
5

This is good. I didn't know this module. – Abhishek dot py Sep 05 '14 at 11:16

score 0 · Accepted Answer · answered Sep 05 '14 at 10:47

0

  import urllib

  opener = urllib.FancyURLopener({})
  f = opener.open("http://www.python.org/")
  lines = f.read() # you missed this
  g = open("data2.txt", "w")
  g.write(str(lines))      # str() converts to string
  g.close()

answered Sep 05 '14 at 10:47

Shruthi Raghunath

96
6

score 0 · Answer 3 · answered Sep 05 '14 at 11:04

0

You can use 'grab' module for this too.

from grab import Grab

g = Grab()
r = g.go('http://www.python.org')
open('page.htm', 'w').write(r.body)

answered Sep 05 '14 at 11:04

Anton Barycheuski

712
2
9
21

Open a web page and write in a text file

3 Answers3