out of memory exception on StringWriter using MVC, can memory be increased?

Question

I have to create a large XML Spreadsheet and it now seems to have hit the threshold on what it can cope with in one go.

I'm not sure on how to split it out so wanted to know if there is a way to increase the memory allocation for internal purposes?

Code that is causing the error is below:

private string RenderViewToString()
{
    using (var writer = new StringWriter())
    {
        var view = ViewEngines.Engines.FindView(_context, _viewName, null).View as RazorView;
        var viewDataDictionary = new ViewDataDictionary<TModel>(_model);
        var viewCtx = new ViewContext(_context, view, viewDataDictionary, new TempDataDictionary(), writer);
        viewCtx.View.Render(viewCtx, writer);
        return writer.ToString();
    }
}

It falls on the Render() when a user decides to download ALL data from the possible combinations of data, before the enrichment of data this file would be circa 55mb converted into an Excel document. Now that the columns have been doubled, the line of code rendering the writer is falling over.

viewCtx.View.Render(viewCtx, writer);

Has anyone got a workaround or know a way to split the relevant data into manageable chunks without recoding it all?

It obtains the data from a view model and that itself takes from 1 to tens of thousands of rows across multiple tables and all is needed by the users in a singular Excel download which is causing issues.

The string is going out to this code:

    protected override void WriteFile(HttpResponseBase response)
    {
        response.Write(RenderViewToString());

    }

But the whole issue of the of whether it needs to be returned string is irrelevant as it never gets that far due to the ViewContext.View.Render(ViewContext, TextWriter) falling over, so is there a way to render the ViewContext to something without it falling over due to a relatively small file (60+ MB) causing it issues?

I have tried to allow the large file use in web.config (A colleague suggested it) but that didnt work.

At the moment the ViewContext is too large to render so I just need to find a way to render it without using the in built one, any ideas?

----- Edit -----

Big thanks to Joe, I modified your code as I dont have IIS Set up to use the HttpContext headers, but the following code did the trick:

        HttpContext.Current.Response.Clear();
        using (var writer = new StreamWriter(HttpContext.Current.Response.OutputStream))
        {
            var view = ViewEngines.Engines.FindView(_context, _viewName, null).View
                as RazorView;
            var viewDataDictionary = new ViewDataDictionary<TModel>(_model);
            var viewCtx = new ViewContext(_context,
                view, viewDataDictionary, new TempDataDictionary(), writer);
            viewCtx.View.Render(viewCtx, writer);
        }
        return null;

Now produces a 70MB file no problems.

More RAM isn't going to help (unless you can put 1TB in a 64bit PC maybe:) ). A strategy for paging/splitting the data is required here. — Jason Evans, Oct 17 '13 at 15:13
Do you really need the string itself, or are you writing to an output of some kind, like a file or the HTTP response? — Joe Enos, Oct 17 '13 at 15:19
@Joe - The output is an Excel file that allows the user to save or open it. Nothing more than that. — Simon Osbon, Oct 17 '13 at 16:05
Checkout http://stackoverflow.com/questions/1569532/asp-net-mvc-returning-plaintext-file-to-download-from-controller-method. It shows how to return plain-text files, but you can definitely tweak it to fit your needs. — Jim D'Angelo, Oct 17 '13 at 16:13

score 1 · Answer 1 · answered Oct 17 '13 at 16:32

You're writing the view output to a string, then writing it to a response. What you may want to do is skip the middle-man, and write the view directly to the response. Since the Render method accepts any TextWriter, you can pass in Response.OutputStream to your StreamWriter instead of just writing to a plain string.

So something like:

System.Web.HttpContext.Current.Response.Clear();
System.Web.HttpContext.Current.Response.Headers["content-disposition"] = 
    "attachment;filename=somefile.txt"; // or whatever
System.Web.HttpContext.Current.Response.Headers["content-type"] =
    "text/plain"; // or whatever
using (var writer = new StreamWriter(System.Web.HttpContext.Current.Response.OutputStream))
{
    var view = ViewEngines.Engines.FindView(_context, _viewName, null).View
        as RazorView;
    var viewDataDictionary = new ViewDataDictionary<Foo>(_model);
    var viewCtx = new ViewContext(_context, 
        view, viewDataDictionary, new TempDataDictionary(), writer);
    viewCtx.View.Render(viewCtx, writer);
}
return null;

now its failing again after working for one day... any other ideas? — Simon Osbon, Oct 19 '13 at 13:29
Not really - this should take care of the problem you had from a large string. But if you've got a different large item in memory, like your model or dictionary, I don't know if you can easily fix that. Resolving this type of problem usually means streaming data instead of building it all upfront, but I don't know if it's something you'd be able to do given this type of data. I'd probably recommend exploring splitting your data into smaller chunks, so you can keep less in memory at a time. — Joe Enos, Oct 19 '13 at 15:13

out of memory exception on StringWriter using MVC, can memory be increased?

1 Answers1