Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data import #307

Open
Vortec4800 opened this issue Apr 29, 2016 · 41 comments · May be fixed by #2360
Open

Data import #307

Vortec4800 opened this issue Apr 29, 2016 · 41 comments · May be fixed by #2360
Labels
bounty:$50 Bounty applies for fixing this issue (Parse Bounty Program) type:feature New feature or improvement of existing feature

Comments

@Vortec4800
Copy link
Contributor

I know on the hosted Parse dashboard (non-beta) there's an option to import data via JSON or CSV which is great. I haven't been able to find this option on the hosted beta dashboard, or on the parse-dashboard project.

Is this functionality available in either place? Is it a planned addition to the dashboard project, or out of scope? Seeding in data from a CSV file is a really useful feature and I was sad to not find it.

@neophob
Copy link

neophob commented Apr 30, 2016

should be obselte now as you can import the data with mongoimport. or are there other advantages when importing via webui?

@Vortec4800
Copy link
Contributor Author

That's a good and valid point. I've been treating the mongo database like a black box, so I don't mess with the way the Parse server is storing data. I suppose it wouldn't be crazy to import directly to the database.

I do still think having the capability in the dashboard would be useful from a UX standpoint. It's much easier in general and much harder to screw up. I like to hand the keys to a project over to a client, and lately that's been including a Parse dashboard login with instructions on how to do basic data management through there. It would be great if they could import data to their databases via CSV on the dashboard without needing us to get involved on the database itself. The more my clients and project stakeholders can manage on their own, the better for me.

@davimacedo
Copy link
Member

I think it should be added to the dashboard by the community. My team can work on it in the future. Meanwhile, you can use cloud functions to import data. I wrote an example how you can do it:
https://gist.github.com/davimacedo/c7738203a2a616b48a22fdc7c00b37d2

@JoaqStar
Copy link

I also used the import feature to import csv to my parse database. I looked up mongoimport and do not understand how it works. Can you please let me know what is the easiest way to import csv data to my parse table?

@sighjones
Copy link

I think the removal of the import function on the dashboard is extremely short sighted. As Vortex rightly points out, giving the client the ability to import simple csv data is EXTREMELY useful. Can you give any idea of a timeline for adding it back in?

@drew-gross
Copy link
Contributor

mongoimport can also work with CSV. Check the docs here: https://docs.mongodb.com/manual/reference/program/mongoimport/

@dailyneedsapp
Copy link

Yes definitely the import data via CSV/Json should be added back to Parse Dashboard.

@mitchellporter
Copy link

+1 for adding it back. Was very easy to use, even for someone that doesn't code.

@ndmeiri
Copy link

ndmeiri commented May 20, 2016

+1

@kikrlabs
Copy link

Hi, I don't see the import data via CSV/Json, was this added back? I accidentally deleted a class with 98 rows. Please add this button back as my data is in json format and need to import my data soon.

@RafaRuiz
Copy link

Yes they removed the button even on the non-beta dashboard :(

@KapoorKarn
Copy link

+1 This function was crucial to helping build apps fast.

@KapoorKarn
Copy link

I'm down to put a bounty on this feature. Anyone else want to join to see if we can get it large enough for someone to tackle?

@miracle7
Copy link

+1, can't believe I can no longer do this with self hosted dashboard.

@stretchnz
Copy link

+1 I would very much like this feature, more or less instant CSV import was great for our use case, now uploading data using node (in the legacy .com system) which takes considerable time to complete due limits on connection rates etc.

@KapoorKarn
Copy link

All, take a look at mongochef (http://3t.io/mongochef/), i've been using it since the removed the bulk import function from the dashboard. Mongochef has allowed me to easily add/remove/modify classes.
I've been able to import semi large csv's to my dbs (roughly 10k records). For large csv files i've been doing a bulk import. There is a commercial and non-commercial option available.

I know some other have mentioned some ways to import as well, but i'm not very skilled in scripts, shells, or yadda yadda. Anyway, just wanted to post in case it helped someone.

@Knana
Copy link

Knana commented Aug 24, 2016

NodeChef has rolled out a feature that allows you to easily import CSV and JSON data into your Parse Server database hosted on NodeChef with just a click of a button. Documentation available here.

@zh-wowtv
Copy link

Without this, how are we suppose to import my previous data? Directly operating on mongodb is never a good idea.

@Treverr
Copy link

Treverr commented Apr 6, 2017

Any update on this? Or know of any options, I've tried the cloud function, it doesn't work anymore.

@AmrKafina
Copy link

AmrKafina commented May 6, 2017

@Treverr I recently set up a simple tool that automates the process. JSON or CSV formatted data is supported.

@nyxee
Copy link

nyxee commented Jun 18, 2017

@AmrKafina, how can we use the tool for local databases please???

@ebelinski
Copy link

@AmrKafina your link is dead.

@montymxb
Copy link
Contributor

Hey everyone, we're not focused on this atm (importing straight to mongodb is usually the way to go) but I too would like to see this back in place :) , it was a nice feature. Considering this would be a convenience, but not a critical component, we'll probably get to this after handling other issues with the dashboard first.

We're totally open to anyone who wants to get started on this in the meantime. It would be prudent to note that if we're going to add an import function we're going to want to also add a compatible export function.

@Macstyg
Copy link

Macstyg commented Feb 2, 2018

@montymxb I have exported .json files from Parse Dashboard, from Back4App service. In their dashboard the import/export functionality is still there. When I try to do mongoimport from .json file to my local DB everything is importing fine. And I can see that data is my database through the mongo shell. But in ParseDashboard I don't see the new collections. Can you know any possible solutions for this problem?

@montymxb
Copy link
Contributor

@Macstyg Hmmm, that doesn't sound quite right. I haven't worked with migrating to (or from) back4app personally so I can't attest to the process there. I would reach out to their support and see if there's something you're missing in order to do a complete import. Although it should just work. A recent migration I oversaw away from a 3rd party provider went relatively smooth via mongoimport.

@wimbledon
Copy link

@Macstyg back4app server settings has mongo db connection string with the credentials. You can use that to connect to your back4app's mongo db and use mongoimport to import your json data that way.

@AmrKafina
Copy link

I believe importing the data directly wouldn't work, because parse's exports (and I assume Back4App's) have a schema different from the one you'll find in the mongodb (ie objectId in parse, _id in mongo).

@wimbledon
Copy link

@AmrKafina You have to also import _SCHEMA collections. This is the collections that tells parse-server what classes and field types are in your classes.

@montymxb
Copy link
Contributor

montymxb commented Feb 28, 2018 via email

@Simone-cogno
Copy link

Any news on this feature? It can be very useful as collaborators do not need to login into the server and mess with commands on the database.

@meirka
Copy link

meirka commented Apr 13, 2021

It is very handy tool, is there a progress or is it on roadmap?

@mtrezza mtrezza added bounty:$20 Bounty applies for fixing this issue (Parse Bounty Program) type:feature New feature or improvement of existing feature and removed help wanted labels Sep 30, 2021
@mtrezza mtrezza added bounty:$50 Bounty applies for fixing this issue (Parse Bounty Program) and removed bounty:$20 Bounty applies for fixing this issue (Parse Bounty Program) labels Nov 6, 2021
@tanzeelrana
Copy link

this feature is a must !! any luck anyone ?

@Sbef98
Copy link

Sbef98 commented Nov 10, 2022

We're still waiting! If i'm lucky i may eventually provide it myself, but this is a seriously short sighted move.

@mtrezza
Copy link
Member

mtrezza commented Nov 10, 2022

I think we could provide at least a very basic version for data import.

It should accept the same format as the existing data export feature, i.e. it should be possible to export a row, delete it from the DB then re-import it without having to modify the data.

The basic version would not cover:

  • Automatically or manually resolve any conflicts, e.g. importing an object with an objectId that already exists
  • Batched import, i.e. importing large data sets my have a neg. impact on the database performance

Open questions:

  • How would that data import work? It should not invoke any Cloud Code triggers, so we can't just save the object. Parse Server will likely need a new interface that allows to import directly to the DB, bypassing a few mechanisms.

@Sbef98
Copy link

Sbef98 commented Nov 10, 2022

  • How would that data import work? It should not invoke any Cloud Code triggers, so we can't just save the object. Parse Server will likely need a new interface that allows to import directly to the DB, bypassing a few mechanisms.

Why should it not activate the triggers? What would it be the perks of implementing such import at that point? In my perspective, the advantage of importing data directly from parse dashboard, apart from the ease of use, would be exactly that my triggers would assure that the data imported are consistent with the policies i'm implementing.

Especially in the scenario in which it's the customer/user the one who is importing datas, allowing him to do it through the dashboard but still forcing him to comply to the implemented policies through the triggers would be a beautiful plus to me!

In such case, even if something is going to change during the migration/import process, such changes are done consistently to the same rules i'd apply to fresh data! Setting new ACL policies, setting new data consistency checks etcetera. This would somehow improve the import also for these who want to move, for example, from an Acess dataset to a Parse istance (and similar cases).

@mtrezza
Copy link
Member

mtrezza commented Nov 10, 2022

That is an interesting take on data import. I don't think that such a behavior would be widely expected though. Data import / export are usually understood as being idempotent operations. Maybe this could be an optional behavior, but it can get pretty complicated, as import order suddenly becomes important, etc.

Here are 2 uses cases:

  • a) A user wants to bulk import / update products in a store: they use the import data as a replacement for manually adding rows in the data browser; each imported product should be validated and invoke triggers, just like it had been entered manually; the import "mimics" a manual data entry, so the order of data imported (also across different classes) becomes important
  • b) A developer wants to restore data in their development environment: they just want to reset all rows in a collection to a previous state, so they delete all the modified data and re-import all the initial data; they don't want any trigger to be invoked, because the initial data is already in its final state

@dblythy
Copy link
Member

dblythy commented Jan 19, 2023

How should this handle objectId, createdAt, and updatedAt? Currently in Parse Server these fields are read-only.

This was referenced Jan 19, 2023
@mtrezza
Copy link
Member

mtrezza commented Jan 20, 2023

The answer lies in which feature we want, see #307 (comment).

I would choose whatever is easiest and requires the least change.

It seems there are 2 restrictions that are Parse Server-side:

  • objectId, createdAt, updatedAt are readonly and cannot be imported
  • triggers being invoked

If overcoming these restriction require a Parse Server PR, we probably should focus on use case (a).

Another supporting argument for (a) is that an import without the aforementioned restrictions is already possible via a raw database import, using MongoDB or Postgres tools.

@dblythy
Copy link
Member

dblythy commented Jan 20, 2023

Option a requires the least effort, but I would assume if objectId is defined in the payload, it would be expected that it is respected. Otherwise it makes it difficult to import pointers etc.

@mtrezza
Copy link
Member

mtrezza commented Jan 20, 2023

We can approach this incrementally and start with a basic version with limitations.

Basic version could do only import of data according to use case (a).

Future feature improvements:

  • Option to modify pointers on-the-fly during import
  • Option whether to invoke class triggers during import
  • Option to import data as-is (incl. read-only fields like createdAt, updatedAt, objectId), which is use case (b)

These improvements will likely require changes in Parse Server as well. Therefore I'd suggest to take an incremental approach to down-size the scope and complexity of each PR.

@dblythy
Copy link
Member

dblythy commented Jan 20, 2023

Sounds like a plan to me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bounty:$50 Bounty applies for fixing this issue (Parse Bounty Program) type:feature New feature or improvement of existing feature
Projects
None yet