Github: Timeout-problem importing a big repository

[01-Scripts]

Hello,

I'm facing a problem when I try to import a kind of big repository for the first time with the source-integration-plugin for Github.
I've just installed the newest version of mantis, applied the patch to get it working together with SourceGithub and tried to do a complete import of one of my repositories.

But after around 60s my provider ends the process.
This leads in some cases to a new project enry named "Imported ....current date&time". And in some other cases (if I only import som special branches), the commits of these branches are imported but now connection between the issues and the commits are made.

Any idea how I can solve that problem for me? Is it possible to break the import process into smaller pieces?

Regards,
Michael

[amyreese]

If the import times out, you should be able to continue the import using the "import latest option", where it will start importing where it left off before the timeout. The "Imported " is caused by a failed "import all", which is a poorly-named way to re-import the entirety of the repository, which will just have the same timeout issues. Import latest will not cause this though, so it might be tedious, but it will eventually get all of the changesets.

Of course, the alternate solution is to temporarily remove the PHP execution timeout, perform the import, and then reinstate the timeout, but import latest will work regardless.

[morgant]

Additionally, you can try using the Source plugin's import.php page which will use the long_process_timeout from your Mantis config file (the default value of '0' means that it will never time out). This is what I do when running via cron to automate imports. I can give further details tomorrow, if you'd like.

[01-Scripts]

Thank you for your hints. I will try using the "Import latest" option this evening and come back to you when I got some results.
Thank you so far.

[01-Scripts]

Using the "import latest" option I was able to import all commits of my repository. But there are no references between the commits and the issues. Although I mentioned all bug ids in the commit message. Any chance to manually parse the commit messages to create the references?

[01-Scripts]

Issue closed.
I wrote a small function to manually reparse all my entries in my database and create the proper connections between bugid and changeid.
For every connection there has to be an entry in the 'mantis_plugin_Source_bug_table'-table.

[dregad]

If you don't mind sharing, you might want to make your code available (e.g. as a gist) as it may be useful to others, and potentially something that could be integrated into the plugin itself

[01-Scripts]

For sure.
Here it is:
https://gist.github.com/01-Scripts/5731171
Just quick and dirty but it worked for me like a charm.

[sverhagen]

FYI, the $g_long_process_timeout seems to affect a part of the process, and not the process as a whole. When setting it to e.g. "10" (seconds implied) it takes a minute or so (>>10s) to fail:

Fatal error: Maximum execution time of 10 seconds exceeded in .../mantis/plugins/SourceBitBucket/SourceBitBucket.php on line 397

Setting it to 0 does not cause the above error anymore, but still runs me into the (as @01-Scripts described it) "my provider ends the process", or in other words:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@example.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

As for the script in the gist, that's great, thank you for that! One remark, my database might be case-sensitive (MySQL is a bit wonky in that area), unlike yours, and "mantis_plugin_source_changeset_table" needed to be "mantis_plugin_Source_changeset_table" for me to work.

[sverhagen]

Here's another approach that I'm tinkering with:

• Add $t_repo->save(); to plugins/Source/pages/repo_import_latest.php (near the end, where it says post_stats). How else will my "Import Latest Data" ever save it's data?
• That probably only works for an "Import Latest Data" at the tail of things. Intermediate ones will never reach the save() then, due to the timeout. So my guess is that only the last run of "Import Latest Data" that you do (and that presumably doesn't time out, if it's small enough) gets to this point
• If we were to fix that, that probably only works for the "Import Latest Data", and not for whatever is imported during the original "Import Everything", that also broke due to the timeout.
• Talking to my provider to find out more about the (for me) two minutes timeout.

After I deleted a repository, which should clean up the database, should it not, I ended up still seeing a lot of data in the "mantis_plugin_Source_bug_table". I'm afraid that's somehow to blame on the script provided in this thread. It would be better to just fix the plugin 😄

[01-Scripts]

Because of some new input to this issue (an because I run into the same timeout problem from time to time) I repoen the issue. Maybe we can fix that generally.

[luzpaz]

we tried to get source integration working for Scribus bugs.scribus.net without success as well.

[luzpaz]

#92