0

I'm struggling to provide ability in my ASP Core 2.2 app to upload and download large files, up to 50gb. Currently, for testing purposes, I'm saving the files on local storage but in future, I will move it to some cloud storage provider.

Files will be sent by other server written in Java, more specifically it will be Jenkins plugin that sends project builds to my ASP Core server using This library.

Currently, I use classic Controller class with HttpPost to upload the files, but this seems to me like not the best solution for my purposes since I won't use any webpage to attach files from client. 

  [HttpPost]
  [RequestFormLimits(MultipartBodyLengthLimit = 50000000000)]
  [RequestSizeLimit(50000000000)]
  [AllowAnonymous]
  [Route("[controller]/upload")]
  public async Task<IActionResult> Upload()
  {
       var files = Request.Form.Files;
       SetProgress(HttpContext.Session, 0);
       long totalBytes = files.Sum(f => f.Length);

       if (!IsMultipartContentType(HttpContext.Request.ContentType))
                return StatusCode(415);

       foreach (IFormFile file in files)
       {
            ContentDispositionHeaderValue contentDispositionHeaderValue =
                    ContentDispositionHeaderValue.Parse(file.ContentDisposition);

            string filename = contentDispositionHeaderValue.FileName.Trim().ToString();

            byte[] buffer = new byte[16 * 1024];

            using (FileStream output = System.IO.File.Create(GetPathAndFilename(filename)))
            {
                using (Stream input = file.OpenReadStream())
                {
                    long totalReadBytes = 0;
                    int readBytes;

                    while ((readBytes = input.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        await output.WriteAsync(buffer, 0, readBytes);
                        totalReadBytes += readBytes;
                        int progress = (int)((float)totalReadBytes / (float)totalBytes * 100.0);
                        SetProgress(HttpContext.Session, progress);

                        Log($"SetProgress: {progress}", @"\LogSet.txt");

                        await Task.Delay(100);
                    }
                }
            }
        }

        return Content("success");

    }

I'm using this code now to upload files but for larger files >300mb it takes ages to start the upload.

I tried looking for many articles on how to achieve this, such as: Official docs or Stack

But none of the solutions seems to work for me since the upload takes ages and I also noticed that for files ~200MB(the largest file I could upload for now) the more data is uploaded the more my PC gets slower.

I need a piece of advice if I am following the right path or maybe I should change my approach. Thank you.

Maciejek
  • 569
  • 1
  • 5
  • 21
  • What is in these large files you want to upload? If you run your web api behind a load balancer (more than 1 instance running at same time) will it be an issue if files are only on one of the servers? – Ken Tucker May 06 '19 at 10:12
  • The large files are project builds prepared by Jenkins. – Maciejek May 06 '19 at 10:27
  • HTTP as a protocol is not suited to large file transfers, especially when you're talking about 50GB. Use a protocol such as FTP/SFTP. – Chris Pratt May 06 '19 at 13:48
  • @ChrisPratt Can I use it with ASP Core or I have to create a new server to handle ftp requests – Maciejek May 06 '19 at 13:54
  • 1
    It has nothing to do with ASP.NET Core. It's just another communication type, like HTTP, but geared specifically for file transfers. SFTP is actually just SSH, mimicking FTP, and as such, is your best bet from a security standpoint. With that, you can actually do certificate auth or issue SSH keys, meaning no need for transferring a username/pass to login. It can be the same server or a different server. Doesn't matter. – Chris Pratt May 06 '19 at 14:01
  • @Maciejek why it shows error in my case while uploading large file .net core 2.2 and passing `IFormFile` in parameter of function – Ahmad Oct 03 '19 at 16:07

0 Answers0