1

I am building a react website, where many of the pages will be d3 data visualizations. I'm loading large data files (each ~5MB of csv or json) for these visualizations.

My folder structure for one of these pages is something like this:

 -src/
   --pages/
   ---vis1/
   ----app.js
   ----styles.scss
   ----bigdata.csv
   ----otherdata.json

Then, in app.js I can just import data from './bigdata.csv'; and use it in the component. This is working alright currently, but I have no idea how this design will scale in the future. I may end up needing even larger data files, and I also have plans to include hundreds of photos eventually. Should I be sticking this stuff in a database and writing server-side code? AWS cloud storage? Something else? I have no idea how any of that works, so I'm trying to figure out if it makes sense to learn.

Most of the react tutorials I've seen deal with little if any data loading. Blogs and posts specifically about design/data like this, or this, or this haven't really helped either.

Elmo
  • 79
  • 7

1 Answers1

2

My suggestion is, if you dont have to import all of them then, you should solve with backend side. Store your data in backend, then fetch when you need (lazy loading). You can do with pagination(like data tables) or scroll(like social media main pages)

lead.eg
  • 76
  • 4
  • thanks for the suggestion lead.eg. I may be wrong, but I think my current setup is already a kind of lazy loading? I'm only importing each dataset in the component that needs it. So the loading of each dataset should only happen when the relevant component is loaded, not all at once. Does that sound right to you, or am I misunderstanding react? Unfortunately, I'm using this data for things like plots (where all the data is displayed at once) or custom built svg maps. I can make a lot of cool things with d3, but often they are designed to be viewed without scrolling or pagination. – Elmo Jun 27 '21 at 22:06
  • I dont know how you implemented imported objects. But I assume that you import 5000 objects from csv and loop over all and render each object as a Component. If so, no it is not lazy loading. If you want to solve on Frontend. You can do like this. İmport your data as you did. Than define a state object like "shownObjects = []" and in didMounted push first 30 element from csv. Then you'll need a threshold(for example scrolled to bottom) to add next 30 element to the array – lead.eg Jul 01 '21 at 16:18
  • It's not really being rendered as 5000 objects where each gets their own component. It's more like 5000 rows of data, and they are all used in a single component. For a basic example, the single component would be a plot ([like this](https://www.d3-graph-gallery.com/graph/scatter_basic.html) ). Each of the 5000 data rows is used to define a point on the plot. So you can see in this particular case it doesn't make sense to load only 30 at a time since there is no scrolling or pagination. The plot will need to display all 5000 points at once in order to be useful. – Elmo Jul 02 '21 at 19:17
  • However, with the other use case I mentioned of loading a large number of pictures, the lazy loading is exactly what I need. I'll wait another couple days and then accept your answer. Thanks again! – Elmo Jul 02 '21 at 19:19
  • Oh I see now. Data for visualization.. Maybe you can use D3 based libraries to handle it. If you want to handle manually to improve your skills, you should learn working with svgs or canvas for performant solution. In my opinion, drawing 5000 divs is bad idea. I found some information about canvas and svg below. [this link](https://stackoverflow.com/questions/5882716/html5-canvas-vs-svg-vs-div) – lead.eg Jul 02 '21 at 23:53