0

As part of a 'pet project' Flutter app that I am trying to build (in Android Studio) I am looking to add a database of information (possibly with Firebase) for users to use with the app.

My current understanding/capabilities

At the moment, I understand how to (and have already) build a database in Cloud Firestore, where the users can store their own data. A good example of this would be a to-do list app, where the user can create a new item, which is stored in the database with their uid. This remains there until they delete it manually. They are also able to update the entry, such as change the name of the item, themselves in the app.

The aim

I've got a set of data at the moment, which is in Excel format, that has the potential to have up to 1000s of rows. I would like to be able to incorporate this into my app, such that the user is able to query the database, either via multiple dependent drop-down menus, or a search widget.

My question

Is there an easy way to convert a reasonably large set of data, currently in Excel format, into a firebase-type database (such as cloud firestore or realtime database) without having to manually enter all of the data?

James
  • 669
  • 1
  • 10
  • 21
  • This is a very broad question and without more details, any answer would be guesswork. Thousands of 'rows' could really mean anything; it could be 10,000 words or 100,000 bits of binary data or each row could contain 100 columns. If you're asking if Firestore can handle that amount of data, the answer is yes, easily. However, getting the data into Firestore in a manner that's useful is going to be use case dependent and will likely require you crafting an app. I will suggest heading over the Firebase google groups as that's a better forum for this kind of question. – Jay Feb 24 '20 at 18:36

1 Answers1

2

For RTDB, you can use use some Excel-to-JSON tool and import that JSON into RTDB. However, I doubt that the exported format will be efficient to use in your app, so you might have to do some transformations (in your language of choice).

If your data is very large (1000s of rows, but... how many columns?), you might have to split your import into multiple smaller imports at different paths of your database.

Doing huge RTDB imports in Firebase console has caused my projects to "misbehave" for a little while, but it goes back to normal quickly, so don't freak out if that happens to you too.

For Firestore, which has no direct JSON import AFAIK, take a look at How to import CSV or JSON to firebase cloud firestore for some ideas.

Stratubas
  • 2,939
  • 1
  • 13
  • 18
  • Thanks for the tips! So, in the database, there are 1000s of rows and around 8 columns. I'll try to take a look at the link you've provided (thanks for that) to see if there's something that'll help me. I've seen quite a few examples of how it has been done (Excel to CSV to JSON etc.) but none of them have been done with Flutter. – James Feb 24 '20 at 18:31
  • @James Wait a minute..... You aren't going to do the imports USING Flutter, right? – Stratubas Feb 24 '20 at 18:31
  • I mean all the importing stuff has nothing to do with Flutter. – Stratubas Feb 24 '20 at 18:37
  • @James I will cut to the chase here to save you a bunch of time. It's highly *unlikely* doing a direct import of your data into Firestore will make the data useful for your use case. There are usually relationships between data in columns and just importing it won't be the correct structure for queries to support those relationships. Your best bet is to write an app that reads in that data and generates the correct structures to be stored in Firestore. – Jay Feb 24 '20 at 18:39
  • @Jay OP doesn't need to create an app to write to the database directly. Creating the new data structure will involve some trial and error. Why do that on the database directly? Creating a JSON file locally and importing it seems like a better option to me. Am I missing something? – Stratubas Feb 24 '20 at 18:47
  • lol, you may or may not be missing something depending on what kind of data it is. Running queries is common practice with Firestore and as you know Excel is rows and columns; a very 'flat file' format. In the OP's example, suppose it's a To Do list for a number of users with the column heading being the user and their To Do's in each following row. In Firestore, it's generally best practice to create a users collection which stores user info, and then a separate collection where To Do's are stored for each user. Importing a flat file in this case won't provide a usable structure. – Jay Feb 24 '20 at 19:27
  • @Jay of course, that's what I mean by "you might have to do some transformations". My objection is about "making an app that writes data to RTDB" versus "making a simpler app that makes a local JSON". – Stratubas Feb 24 '20 at 19:30
  • 1
    *note that I am not disagreeing with your answer at all. Just want to provide additional information. My suggestion is that rows and columns, even converted to JSON is not NoSQL friendly - but then again, it depends on the data which we don't know since it wasn't included in the question. Perhaps the OP can clarify what they are asking. – Jay Feb 24 '20 at 19:30
  • Exactly. @James feel free to ask for recommended database structures for your data. – Stratubas Feb 24 '20 at 19:31
  • IMO the bottom line is that directly importing data from a flat file, even in JSON format could be useless (depending on the data). So crafting an app to transform it into something thats Firestore friendly; collections, documents, references, auto-generated documentId's, fields etc will provide much more flexibility in the long run and will make queries work and offer the ability to denormalize the data where needed. So - I agree with your answer but again, without understanding the entire use case it could be going the wrong direction (so could my comments!) – Jay Feb 24 '20 at 19:38
  • Thanks for the input guys. The data I have at the moment, which I want to include in my app, is quite lengthly (approx. 1000 x 8). In the 'to-do list' app that I made, that writes the users' data to Cloud Firestore in its typical document etc. format. That information is user generated. – James Feb 24 '20 at 22:58
  • For the app I am making at the moment, there will be some user-generated data (which I am happy to handle). However, I need to be able to use this bank of data as a reference, possibly with nestest drop-down menus. The best way I can describe this is on something like 'autotrader.co.uk' where you select the make of the car you're after, then depending on the make, the models show up and when you select a model, the variants of that model show up. – James Feb 24 '20 at 22:58
  • In light of what you guys have been discussing, can you recommend a good way for me to be able to add this data to my Flutter app, for users to use? I'm quite new to it, so I'm learning as I go. Therefore, I'd really appreciate your thoughts! – James Feb 24 '20 at 22:59
  • @James I guess the comments section isn't the best place to discuss such things. If you are ok with the technical stuff about importing data into Firebase, it'd be best if you started a new question and gave more details about your data and your views (since modeling the data after the app's views is a great start). – Stratubas Feb 24 '20 at 23:08
  • @James 100% agree and this statement *modeling the data after the app's views is a great start* is something to really take to heart; model your structure by what you want to get out of it (queries, UI etc). I would also agree another question is in order - but - SO really needs to see some code and an example structure so if you post, be sure to include a sample of the data you're working with, an attempt to model it in Firestore and some code you're attempted to get it out. Start small but if you want directly, we need to understand what 1000 * 8 data is, and what you want to do with it. – Jay Feb 25 '20 at 00:08
  • Thanks very much for the advice. I will try and get a question with sample data and some code up soon, so that it is a lot clearer for you, so it is easier to understand what it is that I am trying to achieve. Thanks again! – James Feb 25 '20 at 09:09