Use Webclient to download several files

Question

I'm trying to download files from webserver via a NUnit-testcase like this:

[TestCase("url_to_test_server/456.pdf")]
[TestCase("url_to_test_server/457.pdf")]
[TestCase("url_to_test_server/458.pdf")]
public void Test(string url) 
{
    using (WebClient client = new WebClient())
    {
        client.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0)");
        client.DownloadFile(new Uri(url), @"C:\Temp\" + Path.GetFileName(url));
    }
}

This code works, but when i'm trying to get the file size, it hangs.

[TestCase("url_to_test_server/456.pdf")]
[TestCase("url_to_test_server/457.pdf")]
[TestCase("url_to_test_server/458.pdf")]
public void Test(string url) 
{
    using (WebClient client = new WebClient())
    {
        client.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0)");
        client.OpenRead(url);
        Int64 bytes_total = Convert.ToInt64(client.ResponseHeaders["Content-Length"]);
        client.DownloadFile(new Uri(url), @"C:\Temp\" + Path.GetFileName(url));
    }
}

How to solve this?

I would use the method shown in the top answer here to get just the headers: http://stackoverflow.com/questions/4507941/c-sharp-webclient-openread-url — Daniel C, May 30 '14 at 08:52

Zunair · Answer 1 · 2017-10-19T15:31:10.127

Waking up dead post but here is the answer...

This issue happens when server is not providing Content-Length in the response header. You have to fix that on the server side.

Another reason when it happens is when we have reached the connection limit to the server. So I am assuming that your issue was similar and it was hanging on second or third try in a loop.

When we call OpenRead, it opens up a stream. We just need to close this stream after getting our file size to make it work properly.

Here is the code I use to get the size:

    /// <summary>
    /// Gets file size from a url using WebClient and Stream classes
    /// </summary>
    /// <param name="address">url</param>
    /// <param name="useHeaderOnly">requests only headers instead of full file</param>
    /// <returns>File size or -1 if their is an issue.</returns>
    static Int64 GetFileSize(string address, bool useHeaderOnly = false)
    {
        Int64 retVal = 0;
        try
        {
            if(useHeaderOnly)
            {
                WebRequest request = WebRequest.Create(address);
                request.Method = "HEAD";

                // WebResponse also has to be closed otherwise we get the same issue of hanging on the connection limit. Using statement closes it automatically.
                using (WebResponse response = request.GetResponse())
                {
                    if (response != null)
                    {
                        retVal = response.ContentLength;
                        //retVal =  Convert.ToInt64(response.Headers["Content-Length"]);
                    }
                }
                request = null;
            }
            else
            {
                using (WebClient client = new WebClient())
                {
                    // Stream has to be closed otherwise we get the issue of hanging on the connection limit. Using statement closes it automatically.
                    using (Stream response = client.OpenRead(address))
                    {
                        retVal = Convert.ToInt64(client.ResponseHeaders["Content-Length"]);
                    }
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
            retVal = -1;
        }


        return retVal;
    }

AHH · Answer 2 · 2014-05-30T12:48:17.807

0

You have to make sure that your server supports this header too. It doesn't seem to be a client problem.

I'd download the file in a browser and check the communications using firebug or some similar program. You have to see Content-length explicitly returned in the response. If not, you need to check the server, otherwise your problem is on the client side. I actually can't imagine a reason why the client can't read the header if it is indeed beeing returned.

enter image description here

edited May 30 '14 at 12:48

answered May 30 '14 at 08:54

AHH

981
2
10
26

I don't understand why I can't get the file size. I've looked at the solution Daniuel C provided but there is no solution on client.ResponseHeaders there? – user3624378 May 30 '14 at 11:33
I'd download the file in a browser and check the communications using firebug or some similar program. You have to see Content-length explicitly returned in the response. – AHH May 30 '14 at 12:47
Ok, i tested to download file via FireFox and I could get Content-Length. The problem is when I try to download the second file in my program and try to get the file size via Int64 bytes_total = Convert.ToInt64(client.ResponseHeaders["Content-Length"]). Then it hangs. Is it a problem with async??? – user3624378 Jun 02 '14 at 07:16
Are you using proxy? And can you not just check the file size after downloading it ? – AHH Jun 02 '14 at 08:00
You can also checkt this question: http://stackoverflow.com/questions/21004406/webclient-freeze-after-download – AHH Jun 02 '14 at 08:06

Use Webclient to download several files

2 Answers2

Linked