Create a curation tool

[booyaa]

Specification

A tool is required to extract links from a specific tracking issue for a campaign.

An example of these tracking issue can be see on #6.

The tool should be able to extract the links, de-dupe and save as an RSS file (XML). At a minimum it should extract the blog post page title, blog post link, publication date.

We expect the tool to be run repeatedly, if the RSS file already exists it should append new items to the end of the RSS file.

Additional information

Bonus if written in Rust, but any language that can be added to an integration service like travis-ci would be considered.

Mentor: @skade

---

Mentoring can be provided, just ask.
Please add comment if you are claiming this issue so we can assign it to you.

[skade]

I would be willing to mentor this. This is a good first issue for someone wanting to write a whole (small) project in Rust.

[fourplusone]

Would be very interested in writing this tool 😃 in rust

[o0Ignition0o]

This sounds like lots of fun, could I give it a try ?

I'd probably pick an http client such as Actix's or Hyper's, to (async) crawl the page and extract links into a BlogPost struct, then use serde to write / append the xml.

I'll have a look at the RSS spec, could I try to work on this ? :)

[o0Ignition0o]

Oh sorry @fourplusone, please go ahead if you want to :)

[skade]

I would actually recommend just to use reqwest. 👍

Can you please join our gitter channel? That might be easiest.

If you have any questions on the specification, please post them here, so that everyone sees them.

[fourplusone]

I just joined. Should this tool go into a separate repo or should it be part of this one?

[skade]

@fourplusone separate repos is probably the easiest

[fourplusone]

My WIP implementation for this can be found here: https://github.com/fourplusone/curate-issue

[fourplusone]

Here is a status update of the curate-issue tool.

• Extracts links from Github Issues + Comments
• Is able to extend existing RSS Feeds
• Detects duplicates
• Extracts Post Date & Title from (most) blog posts
• Compiles without warnings
• Has a few test cases
• Documentation of the code & what it does
• Moving out more stuff from main.rs
• Unit Tests which do not rely on Github / Example blog posts
• Some sort of caching to avoid visiting every page being linked

If you are missing any point, please let me know

[o0Ignition0o]

Awesome !
If you need help or would like me to review it, please let me know :)

[fourplusone]

@o0Ignition0o I'd be glad if you would review some code or improve it

Thanks for your help 👍

[fourplusone]

I think this is ready for testing. Can someone create a repo on /rust-community where the code will find its new home?

[fourplusone]

If you need any help in adding this tool to a .travis.yml file, just let me know

[booyaa]

I'll do some testing, will also raise this with the community team about transferring the repo. as this has implications regarding on-going maintenance.

Thanks again for your work!

[booyaa]

As discussed with @adityac8 on the irc yesterday, who will be testing the tool and providing feedback to @fourplusone. cc @wezm

17:47 <@booyaa> adityac8: do you want to try the curation tool against the posts you've collected for rustreach?
17:48 < adityac8> Sure. I would love to give that a try.
17:48 <@booyaa> we should raise an issue with readrust's author @wezm just to let him know we're going to do this. he might be able to make our curated posts stand out or create
                a category like "content-o-tron"?

[wezm]

Might be a bit late given the state of the tool but some of the work I did for Read Rust might be relevant. Specifically the add-url tool and feed finder crate

I’ll give the tool a look when I have a moment and work out a good way to surface posts that are part of a campaign.

[fourplusone]

The feed finder crate looks very useful. I think i will integrate this in an upcoming release.

[booyaa]

@fourplusone Discuss this with the whole community team, we're happy for you to transfer the ownership of the repo if you still want to do it. Just let us know when it's been done. Thanks!

[adityac8]

@badboy I think so we should be transferring this one to rust-community as well 😄
cc @fourplusone @booyaa

[fourplusone]

👋 @badboy is there anything you need from me in order to transfer the repo?

[badboy]

@fourplusone Simply transfer the repository to me and I will transfer it to the organization.

[badboy]

Done: https://github.com/rust-community/curate-issue

[badboy]

And also enabled Travis now: https://travis-ci.org/rust-community/curate-issue