fix: Ensure bookmark files are correctly downloaded before deleting current ones #683

Monirzadeh · 2023-07-28T18:15:06Z

This pull request should recover old version of Archive and e-book if download Processes failed.
It should close #634
fix #524

TODO:

Things need more time:

should we recover thumb image? What if thumb image remove from source like #524. How we should detect that

fmartingr · 2023-07-29T07:12:12Z

I'd say the path going forward should be: Download the bookmark archive/article/thumb/ebook to a temporary folder. If everything goes ok, we replace the old ones, if not, we raise an error. But it should be open enough to allow multiple operations.

This is a bigger task that you may think, and we need to refactor the logic into smaller chunks to made them easily testable.

internal/view/assets/js/page/home.js

Monirzadeh · 2023-08-05T22:41:03Z

@fmartingr please review this PR and if every thing okays inform me to start update unit test.
Sidenote about warc.NewArchiv: I can't change library, so done that inside ProcessBookmark function.

fmartingr

I have done a first pass really quick. I'm concerned about the backwards compatibility if we change the ebook filename now for people that already have 1.5.5.

I will need to make a second pass trying it out locally.

internal/core/ebook.go

internal/core/processing.go

internal/webserver/handler-ui.go

internal/webserver/handler-api.go

internal/view/assets/js/page/home.js

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

Monirzadeh · 2023-08-07T10:11:59Z

Hi about unit test
I can't reproduce field situation warc and failed internet connection in my unit test.
I add some sample file to use them later instead of rely on someone else file but not use them right now.

fmartingr · 2023-08-12T10:09:29Z

i decide not to do that in this PR so i revert that in return .epub so it is work like as before.

Oh wait, so you wanted to change from file.epub to simply file (without extension)?. It thought it was the opposite!

I think we should leave the extension, so there's nothing to change if that's the case.

Monirzadeh · 2023-08-12T11:32:18Z

Oh wait, so you wanted to change from file.epub to simply file (without extension)?. It thought it was the opposite!

I think we should leave the extension, so there's nothing to change if that's the case.
before this PR we have .epub

No
at the first in this PR .epub remove but after your first opinion about that i revert change. now we have .epub like before and noting change about that.
file store with .epub inside /ebook something like before ebook/1.epub
now i think we should decide about that later not in this PR.

fmartingr

Hey, amazing work on this! I have put out two small changes on a docstring and an error handler but this should be good to go as well.

I still think that MoveFileToDestination should receive two paths and it should handle opening the file by itself, but we can revisit that later on.

Do you need this merged to start working on the go-epub migration?

internal/core/ebook.go

internal/core/processing.go

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

Monirzadeh · 2023-08-20T12:04:06Z

Hi. yes i need this to start working on the go-epub.
i should change things in #679 so i review MoveFileToDestination there
i don't like model.Bookmark too but current logic for update UI need to return model.Bookmark

fmartingr · 2023-08-20T16:20:09Z

i should change things in #679 so i review MoveFileToDestination there
i don't like model.Bookmark too but current logic for update UI need to return model.Bookmark

We can leave it like that for now, don't worry.

…0@628826c by renovate (#19427) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/go-shiori/shiori](https://togithub.com/go-shiori/shiori) | minor | `v1.5.5` -> `v1.6.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>go-shiori/shiori (ghcr.io/go-shiori/shiori)</summary> ### [`v1.6.0`](https://togithub.com/go-shiori/shiori/releases/tag/v1.6.0) [Compare Source](https://togithub.com/go-shiori/shiori/compare/v1.5.5...v1.6.0) It's finally here! After some work we have started moving towards **a more usable and open API** for others to consume, with **proper session handling** (no more random logouts on server shutdowns!), improvements to **ePubs**, UX and some more! See details below for more information (important notes are the breaking changes) and please fill an issue if you see anything weird, better be safe than sorry! #### Breaking changes - The `serve` command is considered deprecated and will be removed in a future release. Right now just proxies to a new `server` command that is the one that should be used from now on. - The **server** command uses a new http backend. This **should be transparent to users** and all things should keep working as usual, but that meant refactoring some of the underlying systems too, so experiences may vary between deployments and operating systems. Please fill an issue if you see that something is not working as expected. - We are moving the API to a more stable, documented and with a proper code structure. Right now there's a mix between old and new endpoints until migration is completed. Check [the documentation](https://togithub.com/go-shiori/shiori/blob/master/docs/APIv1.md) for more information on the new API and [this roadmap filter](https://togithub.com/orgs/go-shiori/projects/2/views/11) to see progress on the API migration. - Authentication to the API now uses JWTs instead of session tokens and the **endpoint has changed to a new one**, please check the documentation mentioned above. This means that there's no longer logout issues when the server is restarted or when you log in in other computer/browser. - The `--webroot` flag **no longer modifies the routes internally**, is up to the user to proxy the routes properly to Shiori without the prefix used to serve it. That means that if you want to serve Shiori under `domain.com/shiori` you need to send the path back to Shiori without the `/shiori` prefix so routes keep working. This can be done in most reverse proxies that we're aware of. We provided [a sample configuration for Nginx](https://togithub.com/go-shiori/shiori/blob/master/docs/Configuration.md#reverse-proxies-and-the-webroot-path). PRs are welcome for other reverse proxies. - The `shiori/gopher` initial user is a full fledged user instead of being hardcoded into Shiori. If you want a new user you need to create a new owner user and then remove the `shiori` user. #### Release cadence Right now the release cadence has been slow because we made too many changes at once and we had to test and finish everything before doing this release. Plan moving forward is to iterate and release faster so we're planning smaller milestones to provide new features and fixes faster to you. I will post an update announcement when [the roadmap](https://togithub.com/orgs/go-shiori/projects/2/views/4) is reviewed. #### What's Changed - APIv1: Start working on new REST API. Refactor logic in domains. by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#497 - Run legacy API and new API at the same time. by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#648 - fix: docker buildx tags by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#650 - Allow JWT authentication into legacy APIs by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#651 - Show version in login page by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#652 - fix: package-name in cleanup tag by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#655 - fix: pr tag prune using other action by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#656 - fix: title is never retrieved when adding bookmark by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#664 - Show Shiori version on server command by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#669 - chore: remove irc badge from README by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#674 - fix: title overwritten if user has defined it by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#684 - Proper SQLite default database and warn SHIORI_DBMS users by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#667 - chore: remove verbose logger by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#685 - Fix CI incorrectly tagging RC releases and disables docker builds on forks by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#686 - preserve fragment in URLs ([#315](https://togithub.com/go-shiori/shiori/issues/315)) by [@arakimo](https://togithub.com/arakimo) in [go-shiori/shiori#687 - Swagger improvements by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#666 - fix: Ensure bookmark files are correctly downloaded before deleting current ones by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#683 - fix(db): handle usage of special characters in searches by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#721 - fix: properly parse mysql connection string, docs update by [@rutkai](https://togithub.com/rutkai) in [go-shiori/shiori#730 - deps: upgrade to Go 1.21 by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#698 - deps: upgrade github.com/gofrs/uuid to v5 by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#736 - feat: build css from less files locally by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#735 - refactor: Migrate ePub generation to go-epub by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#679 - chore(deps): bump the all group with 6 updates by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#738 - chore(deps): bump the all group with 1 update by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#740 - feat: use new JWT auth in all frontend API calls by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#743 - chore(deps): bump the all group with 1 update by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#746 - fix: styles-check and swag-check monitor just needed directory not project root by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#747 - feat: allow resize the dialogbox for bigger/hidpi screens by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#732 - feat: allow per-user settings and store them in database by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#639 - fix: Remove unneeded variable and unify the way send token in header by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#763 - add create ebook by default in settings by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#761 - fix: Actions in overlays on mobile hard to press by [@cbe](https://togithub.com/cbe) in [go-shiori/shiori#759 - fix: Use webp as thumbnail by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#758 - Update documentation for add links to shiori in android devices from share menu by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#757 - Fix typos by [@shirayu](https://togithub.com/shirayu) in [go-shiori/shiori#756 - chore(deps): bump the all group with 1 update by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#767 - refactor: migrate ebook routes by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#742 - Make suggestions tapable/clickable by [@cbe](https://togithub.com/cbe) in [go-shiori/shiori#765 - chore: frontend formatting by [@cbe](https://togithub.com/cbe) in [go-shiori/shiori#764 - ci: add codecov reporting by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#776 - deps: update go dependencies by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#777 - fix typo by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#778 - docs: example deployment for kubernetes by [@JPFrancoia](https://togithub.com/JPFrancoia) in [go-shiori/shiori#754 - Add Documentation>CLI>Add bookmark by [@LLKoder](https://togithub.com/LLKoder) in [go-shiori/shiori#794 - fix: generate coverage profile by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#797 - fix: use noreferer to prevent exposing shiori instance url to archived websites by [@istiak101](https://togithub.com/istiak101) in [go-shiori/shiori#802 - deps: upgrade by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#804 - refactor: migrate bookmark static pages to new http server by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#775 - Fixed lint errors after refactor by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#806 - docs: updated configuration page by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#808 - fix: force usage of shiori prefix for environment variables in configuration by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#807 - deps: updated docker image versions by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#809 - chore(deps): bump the all group with 3 updates by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#812 - chore(deps): bump the all group with 3 updates by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#815 - chore(deps): bump the all group with 3 updates by [@dependabot](https://togithub.com/dependabot) in [go-shiori/shiori#830 - fix: fixes path issues on windows by [@Monirzadeh](https://togithub.com/Monirzadeh) in [go-shiori/shiori#829 - fix: regressions and documentation from 1.5.5 upgrade by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#837 - fix: update go-epub to latest version to avoid filename errors on windows by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#840 - fix: point go-epub go correct repository by [@fmartingr](https://togithub.com/fmartingr) in [go-shiori/shiori#842 - feat: allow authentication using proxy request header by [@PterX](https://togithub.com/PterX) in [go-shiori/shiori#836 #### New Contributors - [@arakimo](https://togithub.com/arakimo) made their first contribution in [go-shiori/shiori#687 - [@rutkai](https://togithub.com/rutkai) made their first contribution in [go-shiori/shiori#730 - [@cbe](https://togithub.com/cbe) made their first contribution in [go-shiori/shiori#759 - [@shirayu](https://togithub.com/shirayu) made their first contribution in [go-shiori/shiori#756 - [@JPFrancoia](https://togithub.com/JPFrancoia) made their first contribution in [go-shiori/shiori#754 - [@LLKoder](https://togithub.com/LLKoder) made their first contribution in [go-shiori/shiori#794 - [@istiak101](https://togithub.com/istiak101) made their first contribution in [go-shiori/shiori#802 - [@PterX](https://togithub.com/PterX) made their first contribution in [go-shiori/shiori#836 **Full Changelog**: go-shiori/shiori@v1.5.5...v1.6.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

Monirzadeh added 2 commits July 28, 2023 20:28

generate ebook get dstPath

7aa37d2

Archive and ebook can recover if download faild

91937b3

Monirzadeh mentioned this pull request Jul 29, 2023

thumbnail will update in update archive #680

Closed

recover thumb if download faild

c12e3f2

Monirzadeh force-pushed the cache-recovery branch from 6296803 to c12e3f2 Compare August 5, 2023 08:51

Monirzadeh added 6 commits August 5, 2023 17:09

thumb image create just if image processing is sucssesful

379a3ba

create epub in tmp if it sucssesful copy to destination

42afd23

archive file create in tmp if it successful move to destination

624a746

move to destination as function

2ec8e5e

update ebook download api and remove .epub from file name

44b9a13

report faild item to user

622ff9d

Monirzadeh commented Aug 5, 2023

View reviewed changes

internal/view/assets/js/page/home.js Outdated Show resolved Hide resolved

Monirzadeh added 2 commits August 6, 2023 01:24

not show dialog if error not happen

bf31c3b

update thumbnail based on last status of bookmark fix go-shiori#524

658c80f

Monirzadeh marked this pull request as ready for review August 5, 2023 22:38

fmartingr requested changes Aug 6, 2023

View reviewed changes

Monirzadeh and others added 7 commits August 6, 2023 13:11

better warning massage

87b6861

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

tmpFile without .epub

c54da26

MoveToDestination change to MoveFileToDestination

0a752e1

return .epub

89c04d6

log if downloadBookImage return error

2b489ea

fix bug remove imgPath just if download last image be unsuccessful

3588119

update old unit test

81703cb

Monirzadeh mentioned this pull request Aug 6, 2023

Better error massage if Archive or ebook update faild #692

Open

Monirzadeh added 3 commits August 7, 2023 00:52

add processing.go unit test

e56ef8f

small massage for report failded item to the user

947eefa

add some more unit test and samplefile

757599f

Monirzadeh added 10 commits August 12, 2023 16:10

update error to user and log that too

a44caad

add more comment

612c439

update comment

9a42103

change variable name parentDir to dstDir

55bb071

more simpler error handling

3da6206

remove unneeded defer

1630092

remvoe unneeded epubWriter.Close()

afba6ff

more readable unit test in processing

17f984d

more readable unit test for ebooks

bf95321

delete all defer os.RemoveAll from unit tests

7f26e3f

Monirzadeh requested a review from fmartingr August 14, 2023 12:33

Merge branch 'master' into cache-recovery

9341e4d

fmartingr approved these changes Aug 20, 2023

View reviewed changes

internal/core/ebook.go Outdated Show resolved Hide resolved

internal/core/processing.go Outdated Show resolved Hide resolved

Monirzadeh and others added 2 commits August 20, 2023 15:28

Better comment

cdf6fab

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

Better Error output

587340b

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

fix err.String() method

3003099

fmartingr added this to the 1.5.6 milestone Aug 20, 2023

fmartingr merged commit f4817cb into go-shiori:master Aug 20, 2023
5 checks passed

fmartingr modified the milestones: 1.5.6, 1.6.0 Aug 20, 2023

Monirzadeh deleted the cache-recovery branch September 17, 2023 10:21

This was referenced Jan 27, 2024

fix: internal filesystem using path based on the underlying OS #823

Closed

Image Caching Issue When Re-creating Bookmarks #833

Closed

Monirzadeh self-assigned this Feb 5, 2024

Monirzadeh mentioned this pull request Mar 26, 2024

fix: thumbnail should show last thumbnail in server in any condition #858

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Ensure bookmark files are correctly downloaded before deleting current ones #683

fix: Ensure bookmark files are correctly downloaded before deleting current ones #683

Monirzadeh commented Jul 28, 2023 •

edited

fmartingr commented Jul 29, 2023

Monirzadeh commented Aug 5, 2023 •

edited

fmartingr left a comment

Monirzadeh commented Aug 7, 2023

fmartingr commented Aug 12, 2023

Monirzadeh commented Aug 12, 2023 •

edited

fmartingr left a comment

Monirzadeh commented Aug 20, 2023

fmartingr commented Aug 20, 2023

fix: Ensure bookmark files are correctly downloaded before deleting current ones #683

fix: Ensure bookmark files are correctly downloaded before deleting current ones #683

Conversation

Monirzadeh commented Jul 28, 2023 • edited

TODO:

Things need more time:

fmartingr commented Jul 29, 2023

Monirzadeh commented Aug 5, 2023 • edited

fmartingr left a comment

Choose a reason for hiding this comment

Monirzadeh commented Aug 7, 2023

fmartingr commented Aug 12, 2023

Monirzadeh commented Aug 12, 2023 • edited

fmartingr left a comment

Choose a reason for hiding this comment

Monirzadeh commented Aug 20, 2023

fmartingr commented Aug 20, 2023

Monirzadeh commented Jul 28, 2023 •

edited

Monirzadeh commented Aug 5, 2023 •

edited

Monirzadeh commented Aug 12, 2023 •

edited