Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xhamster photo galleries stopping download at 24 images. #1777

Closed
Cocassu opened this issue Oct 27, 2020 · 15 comments · Fixed by #1807 · May be fixed by #1798
Closed

Xhamster photo galleries stopping download at 24 images. #1777

Cocassu opened this issue Oct 27, 2020 · 15 comments · Fixed by #1807 · May be fixed by #1798

Comments

@Cocassu
Copy link

Cocassu commented Oct 27, 2020

  • Ripme version: 1.7.93
  • Java version: 8_172
  • Operating system: Windows 7
  • Exact URL you were trying to rip when the problem occurred: https://xhamster.com/photos/gallery/sawaki-mayumi-11023311
  • Please include any additional information about how to reproduce the problem: This just started today and was working fine yesterday with 1 dropped image every few galleries with nothing changed on my end so it may have been something on xhamster's side that is causing the issue.

Expected Behavior

Expecting all 52 images of the gallery to be ripped.

Actual Behavior

Only 24 images are downloaded before it reports it as complete despite the gallery containing more images. Many galleries I have tried from Xhamster photo have had the same problem.

@Je-Te-Ma
Copy link

I started to have the same problem about 2 days ago, tried several galleries, deleting cached files with same result, home it gets fixed on next release.

@mukulj77
Copy link

mukulj77 commented Nov 7, 2020

@Je-Te-Ma I think I've fixed the issue. It's not official though. In the meantime, if you want, you can use this unofficial jar if you like.

https://github.com/mukulj77/ripme/releases/tag/1.7.94

@Je-Te-Ma
Copy link

Je-Te-Ma commented Nov 9, 2020

@Je-Te-Ma I think I've fixed the issue. It's not official though. In the meantime, if you want, you can use this unofficial jar if you like.

https://github.com/mukulj77/ripme/releases/tag/1.7.94

ThanX @mukulj77 sad part is I could not download the file, seems it was removed gives error 404

@ripdave
Copy link

ripdave commented Nov 11, 2020

@mukulj77 patch works fine for me. I no longer face the 24 image limit using his fork. He seems to have forgotton to release his version? Not sure how the release works but I checked out his repo and compiled. The jar file I got was 27 MB, a bit larger then the expected 4 MB, but is what I got with mvn clean compile assembly:single.

@Je-Te-Ma The patched file can be download here: https://github.com/ripdave/ripme/blob/master/ripme-1.7.93-jar_patch_mukulj77.zip

To satisfy Github.com, I had to zip the file to keep it under 25 MB.
Alternativly the uncompressed file can be download from https://we.tl/t-1iDe12O64E

Thanks a lot to @mukulj77 for the fix and to the maintainers of this project

@fuzzycheerio
Copy link

First off, I am a COMPLETE novice here. I too have been getting limited at 24 pictures. So I uninstalled RIPME from my comp, restarted, downloaded the patch via the link in @ripdave 's post and then tried to rip another large gallery... Still getting stuck at the 24 pic limit.

I'm new to ripme, and github, Im certainly not versed in programming etc. So is this a issue that is being worked on? Has it been fixed? Or is it still messing up for some folks out there?

Is there any OTHER way to go about correcting this issue?

@vindicator1
Copy link

yeah, it was working fine yesterday but went back to dying at 24 images this morning.

@mukulj77
Copy link

@vindicator1 , you are right. It's happening again. I'll look into it next week. Bit swampped with things.

@naamval
Copy link

naamval commented Nov 19, 2020

@vindicator1 , you are right. It's happening again. I'll look into it next week. Bit swampped with things.

@mukulj77 It seems that the pagination part that the ripper is searching for is now somehow obfuscated by the website. When opening the page source in Firefox, you get a <!-- rendering error -->, and I assume that's also what the ripper sees, leading it to conclude that there are no more pages to rip (as (doc.select("a[rel=next]").first() will return null)

There might be workarounds to get all pages: one could be to take the total number of images of the gallery, divide by 24 and round up, then loop through the pages (a bit messy, though). Another could be to make use of the <script id='initials-script'>, and make a loop that runs as long as the number "active" (the current page) is lower than or equal to "maxpage" - however I'm not sure whether that's even possible.

@mukulj77
Copy link

@vindicator1 , you are right. It's happening again. I'll look into it next week. Bit swampped with things.

@mukulj77 It seems that the pagination part that the ripper is searching for is now somehow obfuscated by the website. When opening the page source in Firefox, you get a <!-- rendering error -->, and I assume that's also what the ripper sees, leading it to conclude that there are no more pages to rip (as (doc.select("a[rel=next]").first() will return null)

There might be workarounds to get all pages: one could be to take the total number of images of the gallery, divide by 24 and round up, then loop through the pages (a bit messy, though). Another could be to make use of the <script id='initials-script'>, and make a loop that runs as long as the number "active" (the current page) is lower than or equal to "maxpage" - however I'm not sure whether that's even possible.

That makes lot of sense. I'll definitely look into those ideas.

@fuzzycheerio
Copy link

@mukulj77 and @naamval While that may be part of the issue, many albums have more than 24 images per page. So there might be some other issue here on top of what you already discovered.

@naamval
Copy link

naamval commented Nov 19, 2020

@fuzzycheerio No, that is expected. The ripper looks at the mobile website, which does display 24 images per page.

@fuzzycheerio
Copy link

@naamval , ah gotcha. Thanks so much for taking a deeper dive and explaining these things to me, a admitted noob. =)

mukulj77 added a commit to mukulj77/ripme that referenced this issue Nov 20, 2020
This is an expansive change. Added the fasterxml.jackson package to POM
for parsing the JSON object. TO-DO: Need to check the test cases.
@mukulj77
Copy link

@naamval The problem lied in quite a substantial revamp by the website. I think they are now dynamically generating the next pagination markup.
They have moved the data to the #inital-scripts tag on the page. I really had to dig in for about 4 hours to fix it. It was quite a learning experience. :)
I'm not sure though if my solution is elegant. Also, the Test cases are pending.

tl;dr folks, please check out the temporary jar I've created for now. Would liase with the folks maintianing the project to see if they accept my solution.

The jar is with all the dependencies, and hence large.

https://github.com/mukulj77/ripme/releases/tag/1.7.93

@yaleqb
Copy link

yaleqb commented Nov 20, 2020

@mukulj77 thank you for fixing this! I just tested 1.7.93 on an xhamster gallery with 149 images. It worked perfectly. This 24 image limit has been bugging me for a long time. Thank you!

mukulj77 added a commit to mukulj77/ripme that referenced this issue Nov 20, 2020
Just improving the bug-fix. TO-DO: Need to check the test cases.
@naamval
Copy link

naamval commented Nov 20, 2020

@mukulj77 Thanks for your efforts to fix this issue! I ran a bunch of tests and it's working flawlessly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants