How to load millions of records in asp.net mvc 3 or 4 to client side. (slow issue)

Question

I have a project that loads millions of records. I will be using asp.net mvc 3 or 4. I am sure my page would load very slow because of much data retrieved from the server. I have been creating SQL Server agent job to perform early queries and save it in a temporary table which will be used in the site. I am sure there are many better ways. Please help. I have heard of IndexedDB or WebSQL for client side Database. Any help/suggestion would be appreciated.

My first Post/Question here in stackover!

Thanks, In Advance!

You probably don't need to load millions of records all at once, do you? — Paul Alan Taylor, Jun 19 '13 at 06:11
Paul, It is possible because the user can select date range. I think "1 year" would have 700,000 plus. — Taerni, Jun 19 '13 at 06:16
client. because the user would select date range and some fields. the page should not take long to get the data because i have to draw using html5. — Taerni, Jun 19 '13 at 07:42

score 2 · Answer 1 · edited May 23 '17 at 12:13

2

You might want to look at pagination. If the search returns 700,000+ records you can separate them into different pages (100 per page or something) so the page won't take forever to load.

Check similar question here

edited May 23 '17 at 12:13

Community

1
1

answered Jun 19 '13 at 06:28

whastupduck

1,156
11
25

I am not using grid or paging. It will just be an ordinary List object and will be converted to json in client side. User will select a date range and some fields to select. so it would post again back to the server to query based on the selected date range. I am looking for a better way than posting back and forth to the server. – Taerni Jun 19 '13 at 07:41

score 1 · Answer 2 · answered Jun 19 '13 at 15:49

I've been dealing with a similar problem 500K data stored on client side in IndexedDB. I use multiple web workers to store the data on the client side (supported only in chrome) and I only post the id's and the action that is used on the data to the server side.

In order to achieve greater speed, I've found that the optimal data transfer rate is achieved using 4-8 web workers all retrieving and storing 1K - 2.5K items per call, that's about 1MB of data.

Things you should consider are:

limiting the actions that are allowed during synchronization process
updates of the data on server vs client side - I use the approach of always updating data on server side and calling sync procedure to sync the changed data back to the client side
what are you going to store in memory - Google Chrome limit is 1.6GB so be careful with memory leaks, i currently use no more than 300MB during the synchronization - 100-150MB is the average during regular work

score 0 · Answer 3 · answered Jun 19 '13 at 07:17

0

You need to split the data by pages. Retrieve certain range from the database. Do you really need millions of rows at once. I think there should be a filter first of all.

answered Jun 19 '13 at 07:17

iefpw

6,816
15
55
79

score 0 · Answer 4 · answered Jun 19 '13 at 08:03

Paging is definitely the way to go. However, you want to make the site responsive, so what you should do is use Ajax to run in the background to load up the data continuously while the letting the user to interact with the site with initial set of data.

How to load millions of records in asp.net mvc 3 or 4 to client side. (slow issue)

4 Answers4