[meta/discussion] Import/Export functionality

[Monirzadeh]

this feature request a better way to import from other service as we talk about them in different issue and discussion like:
#768 #762

What is the problem?

currently shiori just can import some service like wallabag and pocket from terminal.
we need a better solution that solve multiple problem

1. we should done that in webui too.
2. currently command-line import them from specific functions inside https://github.com/go-shiori/shiori/tree/master/internal/cmd that is too more specific. we should combine all of them in one general function (terminal and webui) for easier maintain
3. in webui we should have a system like notification to inform user from import progress
4. maintaining of this should not be complicated

What we can do?

1. user get export from any service
2. convert that to specific format shiori understand (community can help to create script for convert different service to the same format)
3. shiori import that format
   with this workflow we don't change code each time to support new service.

What about UI?

when user upload file and press import in settings menu. while it importing a bell icon show up above logout buttom.
maybe it can show progress as a ring around that. when it is finish bell icon disappear.

What should be consider?

some service has reading status and tag. how should we handle that? (as shiori don't have reading status yet)

currently showing progress percentage in UI is the problem that i can't imperilment yet.
is there anybody better idea for import functionality?

[fmartingr]

I'm going to do a brain dump here on my thoughts on this issue:

• The scope of the import/export for us (Shiori) should be allowing users to export and import data in a Shiori format can understand. This is useful when someone is switching provider or database engine. We need to pave a way for users to create a bundle of some structured data format file that we can read and import into the database. It would be nice if the user could toggle the export of files (allowing a way smaller file size if we only want the bookmarks and not the archives/thumbs/etc). if we do this, we need to take into account Save hasEbook, hasArchive, etc... in database #675.
  • We should isolate the logic outside the CLI/API, potentially in a domain or even a separate package.
• If we want to allow users to do these actions via the Web UI we need to implement some kind of job system that shiori can use to run tasks in the background. This could be leveraged to create archives/ebooks/download content as well, instead of just firing a goroutine and call it a day. This is more complex than it seems, since this are jobs and we need to display information from them to the user we need to store data in the database and add a new way in the UI to show when a job is running.
  • I like your idea of having that in the UI, but it would need to be expanded to display how many jobs are running and potentially adding a new page that shows the list of jobs and their status.
• Once we have an import/export system that Shiori understands in place, we can open up to the community the creation of the conversions from other services to Shiori. We can participate in some of them or migrate code we already have, but maintaining them long-term is not feasible for us.

[LLKoder]

@Monirzadeh I know about that. My question about how I can use Shiori now. How I can move Chrome and Firefox Bookmarks to Shiori now any cases.

For example there is converter Netscape Bookmark File<->Chrome/Firefox Bookmark File or I have to use other steps. Or I have to write this converter. What would my steps to get temporary solution during you decide how it'll work? I do not know Go but I know other laguages include SQL.

I'm interested in Shiori 'cause it's bookmarks keeper in local cloud. But it'll become unuseful for me and for other users if we cannot move our bookmarks to Shiory from top browsers. I have no opportunity move ~100 thousand bookmarks by hand.

[Monirzadeh]

@LLKoder you can use and modify scripts that we have in documentation.

for example

1. export your firefox bookmarks as html and move thet file to the shiori server
2. use bellow script to import your bookmarks to shiori

#!/bin/bash
grep -Eoi '<a [^>]+>' $1 | grep -Eo 'HREF="[^\"]+"' | cut -d'=' -f 2- | tr -d '"' | tac > firefox2shiori.txt

# Reading the URLs one by one and adding to shiori
while IFS= read -r line; do
    shiori add $line
done < firefox2shiori.txt

rm firefox2shiori.txt
exit 0

i don't have chrome base browser(so i don't have accesses to the bookmark export sample) to update script for that too but general concept is simple.
extract list of URLs from any backup file and loop them in shiori add command

[LLKoder]

@Monirzadeh Thanks.
Can shiori add get not only link? Lets say title or other Shiori's parameters.
Or can Shiori get title for url automatically itself?

[Monirzadeh]

@Monirzadeh Thanks. Can shiori add get not only link? Lets say title or other Shiori's parameters. Or can Shiori get title for url automatically itself?

yes.
You can use -i flag shiori add $url -i $customName.
You can see more options here.
Unfortunately --help does not show this option, it may be helpful if you can send a PR for adding it to --help or at least to shiori documentation in docs.

[mpr1255]

Major support for either an export functionality, or saving the files in some kind of format that ANY other software can read. I thought they might have been warc or mhtml, but nope. They are some kind of binary that only shiori can access?? And yet there is no API to use shiori from the CLI to push them out into anything that any other application can access? This makes Shiori a walled garden -- we can easily add things, but cannot get anything out except manually opening it and saving it as a file in the browser. DownThemAll! kinda works -- but they lose all the css, so not true archives.

This has come up before: #337

I'm puzzled by the choice to save the files as (apparently?) a unique binary format? Why not just save them in the standard warc format, or perhaps mhtml? That is still a single file. PDFs are stored not as pdf either. I'm very curious about these design choices. Anyway, sadly cannot use shiori because of this. It's far better than the other tools I've been trying today but the fact that one cannot get one's data out makes it unusable for the main purpose: archiving stuff for reliably accessing later.

[LLKoder]

@Monirzadeh I add PR (#794) but it was blocked.

[LLKoder]

@fmartingr @Monirzadeh I'm writing Python script for Import Chrome/Firefox Bookmarks to Shiori. Script will convert folder structure to tags structure. If it need I can do PR as temparary solve.

[Goosegit11]

@LLKoder

Script will convert folder structure to tags structure.

But what if I have both folder and tag structure in my Firefox?

[Goosegit11]

@mpr1255 Completely agree, that's a mess! I'm hoping that the concept will change in the future..

[LLKoder]

But what if I have both folder and tag structure in my Firefox?

@Goosegit11 How do you have tags in Firefox? Chrome/Firefox don't support tags. Or am I wrong?

[LLKoder]

@Goosegit11 I'll able to post a script here when I have a time to finish it, if need. Just developers not say merge the script to main branch or not. I not want to do waste PR.

[Goosegit11]

@LLKoder

[fmartingr]

Hey people, trying to address all of the comments here. Thanks a lot to @Monirzadeh for chiming in and helping people move forward.

When I started using Shiori I noticed this "walled garden" problem, but for me at the time supposed no issue (I could just write a script to move things in and out), but now that I am here, the steps to move out of that problem are:

• Improve the API. There has been problems with the API for some time, it was usable but suboptimal. The new API won't have session problems (thus, the extension will benefit from this) and will hopefully allow people to write it's own things on top of Shiori. (APIv1: Tags #658 APIv1: Ebook #676 APIv1: Accounts #657 APIv1: Bookmarks #659 and probably others).
• Move from WARC to Obelisk (Support Obelisk archiving #353). Obelisk will create archives in a single HTML file rather than a binary file. Still needs to be investigated if we can easily migrate all current WARC files to Obelisk readable files easily, but the idea would be to add support to Obelisk while keeping WARC, to allow backwards compatibility and make archiving modular, allowing to add more archiving options (like, for example, sending a webhook to archivebox).
• Overhaul the import/export (this issue). Support exporting and importing to a machine-readable format that Shiori can use and others can convert to/from other bookmarking systems.

This is the current roardmap, and along the way some other issues had been found. We would love support in those areas if people are interested, I can provide guidance in issues for contributors.

If you have any more concerns about this please raise them here, the more we know before doing changes the better.

[LLKoder]

@Goosegit11 Ahahaha... I have not known it till today. I don't have tags field when I save bookmarks, 'cause I have no active tags. But there is tags in bookmarks manager as I have seen. :) Thanks. I try add Firefox tags to my script.

@fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better?

[fmartingr]

@fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better?

If you make a script to work with Shiori bookmarks you can create a repository/gist for it and we can link it from the documentation under community supported tooling.