Merge branch 'release/1.0.1'

WHATSNEW.md

@@ -1,9 +1,17 @@
-# Iframely Changelog
+# Iframely Parsers Changelog
 
 This is the history of the Iframely changes. Updates that are older than one year are not shown.
 
 To stay tuned and up-to-date, watch [Iframely on GitHub](https://github.com/itteco/iframely).
 
+### 2016.12.15, Version 1.0.1
+
+- Maintenance of the domain plugins
+- Added: iTunes singles, GoPro VR, Lynda.com
+- Better coverage for NBC, Sveriges radio, RedBull, Today.com, Pinterest pins and USA Today
+- New rel `playerjs` for providers that support [playback events API](https://iframely.com/docs/playerjs) (but without the neeed of iFrame adapters)
+
+
 ### 2016.11.08, Version 1.0.0
 
 - Add ability to specify custom plugins (thanks @iloire and @tbasse!) - change `CUSTOM_PLUGINS` dir in your local settings;
@@ -120,85 +128,3 @@ See sample config file for ways to customize. *Heads up:* `twitter.status` in co
  - Please run `npm update` as package dependencies have changed. 
  - If you run into `../lib/kerberos.h:5:27: fatal error: gssapi/gssapi.h: No such file or directory`  - see [this comment](https://github.com/itteco/iframely/commit/991406b37da76f0a27501611702cb7a414136a6b)
 
-
-### 2015.09.08, Version 0.8.10
-
- - Domains maintenance
-
-### 2015.07.03, Version 0.8.7
-
-- Cleanup and maintenance of domain plugins
-- New domains: Datawrapper, Widgetic
-- Option to send params for individual oEmbed providers (`ADD_OEMBED_PARAMS` in sample config)
-- General config option to group Iframely JSON's links by rel by default (`GROUP_LINKS`)
-- `media=1` optional query string API param that will make Iframely try to return actual media. Ex: Instagram MP4 video instead of status embed
-- Direct URLs to office docs are now proxied by Google Docs Viewer
-
-
-
-### 2015.06.08, Version 0.8.6
-
-Changes to existing publishers:
-
-- Added Imgur’s new `app` embeds
-- Support for direct links to Giphy’s gifs
-- All Google parsers: properly handle private content and return status 403 if login is required
-- Converted Comedy Central plugin to generic  `sm4` parser for MTV Network
-- Domain plugins now allow “*” as a mixin. It activates all other generic parsers. Number of domains have been moved to this approach
-- Returns players pinned to Pinterest: YouTube, Vimeo, SoundCloud, TED, DailyMotion
-
-
-New publishers:
-
-- theta360.com
-- npr.org (direct links to popup player)
-- CNBC videos
-- CNN video gallery (videos in articles still processed through whitelist)
-- Returned The Onion videos
-- Added Google Spreadsheets (thanks @j0k3r)
-
-
-
-### 2015.05.07, Version 0.8.5
-
-- Urgent fix for YouTube as they retired v2 API. 
-- Set `config.providerOptions.youtube.api_key` with your Google API key to restore all previous features. See sample local config.
-- API key is optional: if not given the parsers will fallback to oEmbed data. 
-- It may be the same key you have for new Google Maps, yet copy it in other option.
-
-
-### 2015.05.06, Version 0.8.3
-
-
-- New Google Maps plugin to support current URL scheme (Google finally stopped re-directing our user-agent to their classic URLs)
-- Google Maps require API key: get it [here](https://developers.google.com/maps/documentation/embed/guide#api_key) and configure as `providerOptions.google.maps_key` - see sample local config
-- Support of [Camo Proxy](https://github.com/atmos/camo) for all images (thanks guys from Redbooth). See sample config to activate
-- Fixes for Spotify and minor fixes for other domains
-- Restored domain plugin for The Guardian
-
-
-### 2015.04.02, Version 0.8.2
-
-- Fixes for some domain plugins like path.com, visual.ly, prezi, deviantart, storify, roomshare.jp
-- Twitter statuses can now have variable width (through `&maxwidth` param) and are aligned to center
-- New Facebook Video embeds. Yay!
-- New plugins for CartoDB, wasu.cn, ku6.com, datpiff.com, tudou.com, deezer
-
-
-
-### 2015.03.04, Version 0.8.0
-
-*Heads up:* 
-
-Starting from this version, the minimal Node.js version required for Iframely is 0.10.22. We had to make a choice to either support latest Node.js or earlier version due to incompatible libraries dependenices. Please, run 'npm update' to update libraries. Unfortunatelly, update likely won't work if your Node is earlier than 0.10.22.
-
-
-- Instagram status JS embeds with rel `app`
-- Tumblr status JS embeds with rel `app` (beware: embeds don't work with SSL)
-- International domains for Pinterest
-- Google custom maps
-- YouTube playlists and timed embeds
-- Google+ posts for international usernames. Plus, properly exclude posts in groups
-- Medium stories will now have JS embeds too
-- Fix issues with caching of JSONP requests
-- Number of fixes in various other plugins

config.js

@@ -115,7 +115,8 @@
             html5: "html5",
             gifv: "gifv",
 
-            promo: "promo"
+            promo: "promo",
+            playerjs: "playerjs"
         },
 
         // Whitelist settings.

docs/API.md

@@ -1,4 +1,4 @@
-# Iframely API for Responsive Embeds and Unified Meta
+# Iframely API endpoint
 
 You send Iframely an URL via HTTP GET request. Iframely will return you semantics and attribution `meta` data and embeds `links`. Together, they mimic the `<head>` element of requested web page.
 

docs/ERRORS.md

@@ -1,4 +1,4 @@
-# Error handling
+# Result codes & error handling
 
 If everything went well, Iframely will return HTTP status code 200. If there was a problem with processing of your URL, Iframely will return HTTP error code `4xx`.
 

docs/LINKS.md

@@ -1,4 +1,4 @@
-# Links in Iframely API
+# Iframely API: Embed Links
 
 `links` in [Iframely API](https://iframely.com/docs/iframely-api) is the list of objects with fields `rel`, `href`, `type` and `media`. Just like you would expect a `<link>` in `<html>` `<head>` of the page to be. 
 

docs/META.md

@@ -1,4 +1,4 @@
-# Iframely META
+# Iframely API: Meta
 
 Most web pages have organic `<meta>` data published using different semantics standards and optimized for different platforms. For example, oEmbed, Open Graph, Twitter Cards, core HTML meta for Google, Dublin Core, Parsely, Sailthru, etc.
 

docs/PROVIDERS.md

@@ -1,4 +1,4 @@
-# What URLs to send to Iframely
+# Which URLs to send to API?
 
 Iframely follows unique approach in determining which providers it supports. Conventional wisdom comes from oEmbed v1 spec for publishers, which requires them to list the supported URL schemes. Most embed libraries follow in these footsteps.
 

docs/SETUP.md

@@ -1,4 +1,4 @@
-# Deploy Iframely on Your Own Servers
+# Self-host Iframely APIs
 
 The source-code of [Iframely](https://iframely.com) parsers is hosted [on GitHub](https://github.com/itteco/iframely). It's a Node.js library and server published under MIT license. You can start self-hosting any time, as APIs are nearly identical (except the hosted version requires API key and can output the HTML of [smart iFrames](https://iframely.com/docs/iframes)).
 

docs/oEmbed.md

@@ -1,4 +1,4 @@
-# oEmbed API
+# oEmbed API endpoint
 
 Iframely gives you a classic [oEmbed v1](http://oembed.com) endpoint, where you send your URL and get embed codes in response. The HTML embed codes will be responsive in most cases, whenever possible.
 

lib/core.js

@@ -1279,7 +1279,7 @@
                     cb(null, allResults);
                 }
 
-                utils.sendLogToWhitelist(uri, context.meta, context.oembed, context.whitelistRecord);
+                utils.sendLogToWhitelist(uri, context.meta, context.oembed, context.oembedLinks, context.whitelistRecord);
             },
 
             // Recursive callback to continue run available plugins

lib/plugins/system/oembed/providers.json

@@ -93,14 +93,21 @@
         ],
         "endpoint": "http://www.youtube.com/oembed"
     },    
+    {
+        "name": "Today.com",
+        "templates": [
+            "today\\.com/video/.*"
+        ],
+        "endpoint": "http://www.today.com/oembed"        
+    },
     {
         "name": "Qik",
         "templates": [
             "qik\\.com/video/.*",
             "qik\\.com/.*"
         ],
         "endpoint": "http://qik.com/api/oembed.{format}"
-    },
+    },    
     {
         "name": "Revision3",
         "templates": [

lib/plugins/validators/async/22_imageSize.js

@@ -53,7 +53,7 @@ module.exports = {
                     };
                 }
 
-                if (error === 404 || (typeof error === 'string' && /invalid/i.test(error))) {
+                if (error >= 400 || (typeof error === 'string' && /invalid/i.test(error))) {
                     // Image not found. Exclude link from results.
                     link.error = error;
                 }

lib/plugins/validators/media.js

@@ -22,8 +22,12 @@ function makeMediaResponsive(link) {
             m["aspect-ratio"] = Math.round(m["aspect-ratio"] * 10000) / 10000;
         }
     }
+}
+
+function checkMediaAspectLimits(link) {
+    var m = link.media;
 
-    if (!m["max-width"] && m["aspect-ratio"] && m["aspect-ratio"] < 1 && (USUAL_WIDTH / m["aspect-ratio"] > USUAL_HEIGHT)) {
+    if (m && !m["max-width"] && m["aspect-ratio"] && m["aspect-ratio"] < 1 && (USUAL_WIDTH / m["aspect-ratio"] > USUAL_HEIGHT)) {
         m["max-width"] = Math.round(m["aspect-ratio"] * USUAL_HEIGHT);
     }
 }
@@ -106,10 +110,13 @@ module.exports = {
 
         moveMediaAttrs(link);
 
-        if (link.rel.indexOf('responsive') > -1
-            && link.rel.indexOf(CONFIG.R.player) > -1) {
+        if (link.rel.indexOf(CONFIG.R.player) > -1) {
+
+            if (link.rel.indexOf('responsive') > -1) {
+                makeMediaResponsive(link);
+            }
 
-            makeMediaResponsive(link);
+            checkMediaAspectLimits(link);
         }
     }
 };

lib/utils.js

@@ -501,10 +501,21 @@ var lowerCaseKeys = exports.lowerCaseKeys = function(obj) {
     }
 };
 
-exports.sendLogToWhitelist = function(uri, meta, oembed, whitelistRecord) {
+exports.sendLogToWhitelist = function(uri, meta, oembed, oembedLinks, whitelistRecord) {
 
     if (!CONFIG.WHITELIST_LOG_URL) {
-        return
+        return;
+    }
+
+    if (uri.match(/^(https?:)?\/\/[^\/]+\/?$/i)) {
+        // Skip base domain url.
+        return;
+    }
+
+    var oembedHref = oembedLinks && oembedLinks.length && oembedLinks[0].href;
+
+    if (oembedHref && oembedHref.match(/\/wp-json\/oembed\//)) {
+        return;
     }
 
     var data = getWhitelistLogData(meta, oembed);
@@ -530,6 +541,10 @@ exports.sendLogToWhitelist = function(uri, meta, oembed, whitelistRecord) {
 
         data.uri = uri;
 
+        if (oembedHref) {
+            data.oembed = oembedHref;
+        }
+
         request({
             uri: CONFIG.WHITELIST_LOG_URL,
             method: 'GET',
@@ -807,7 +822,7 @@ function getWhitelistLogData(meta, oembed) {
             !!(meta.sm4 && meta.sm4.video && meta.sm4.video.embed)
     }
 
-    if (oembed && oembed.type !== 'link' && !(oembed.type == 'rich' && isWordpress)) {        
+    if (oembed && oembed.type && oembed.type !== 'link' && !(oembed.type == 'rich' && isWordpress)) {
         r['oembed_' + oembed.type] = true;
     }
 

package.json

@@ -1,6 +1,6 @@
 {
     "name": "iframely",
-    "version": "1.0.0",
+    "version": "1.0.1",
 
     "description": "oEmbed/2 gateway endpoint. Get embed data for various http links through one self-hosted API",
     "keywords": ["oembed", "embed", "open graph", "og", "twitter cards"],

plugins/domains/500px/500px.com.js

@@ -32,7 +32,7 @@ module.exports = {
                     canonical: url
                 },
                 type: CONFIG.T.text_html,
-                rel: [CONFIG.R.image, CONFIG.R.inline, CONFIG.R.html5, CONFIG.R.ssl],
+                rel: [CONFIG.R.image, CONFIG.R.html5, CONFIG.R.ssl],
                 "aspect-ratio": oembed.width / oembed.height
 
             }

plugins/domains/animoto.com.js

@@ -12,13 +12,7 @@ module.exports = {
         "domain-icon"
     ],
 
-    tests: [
-        "http://animoto.com/play/k01h0yvqf59whdd80nla1q",
-        {
-            skipMixins: [
-                "oembed-thumbnail",
-                "oembed-author"
-            ]
-        }
-    ]
+    tests: [{
+        noFeeds: true
+    }]
 };

plugins/domains/bandcamp.com.js

@@ -3,7 +3,7 @@ module.exports = {
     re: [
         /^https?:\/\/[a-z0-9-]+\.bandcamp\.com\/(album|track)\/(\w+)/i,
         /^https?:\/\/[a-z0-9-]+\.bandcamp\.com/i,
-        /^https?:\/\/([a-z-\.]+)\/(album|track)\/([a-z-]+)/i // watch out for overlay with play.spotify.com which has digits
+        /^https?:\/\/([a-z-\.]+)\/(album|track)\/([a-z-]+)\/?$/i // watch out for overlay with play.spotify.com which has digits
     ],
 
     mixins: [

plugins/domains/bbc.com.js

@@ -14,7 +14,7 @@ module.exports = {
 
                 return {
                     href: "http://www.bbc.com/" + playlist[1] + '/embed',
-                    rel: [CONFIG.R.player],
+                    rel: [CONFIG.R.player, CONFIG.R.html5],
                     type: CONFIG.T.text_html,
                     "aspect-ratio": 16/9,
                     'padding-bottom': 225, 

plugins/domains/bigthink.com.js

@@ -1,20 +1,22 @@
 module.exports = {
 
-    re: /^https?:\/\/bigthink\.com\/videos\//i,
+    re: /^https?:\/\/bigthink\.com\/videos\/([a-zA-Z0-9\-]+)/i,
 
     mixins: [
         "*"
     ],
 
-    provides: '__allowEmbedURL',
-
-    getData: function(urlMatch) {
+    getLink: function(urlMatch) {
         return {
-            __allowEmbedURL: true
+            href: "http://bigthink.com/embeds/video_idea/" + urlMatch[1],
+            rel: [CONFIG.R.player, CONFIG.R.html5],
+            type: CONFIG.T.text_html,
+            'aspect-ratio': 16 / 9
         };
     },
 
     tests: [
-        "http://bigthink.com/videos/bre-pettis-on-makerbot-3-d-printing"
+        "http://bigthink.com/videos/bre-pettis-on-makerbot-3-d-printing",
+        "http://bigthink.com/videos/vivek-wadhwa-every-industry-will-be-disrupted"
     ]
 };

plugins/domains/blacklist.js

@@ -10,7 +10,7 @@ module.exports = {
     getData: function(urlMatch, cb) {
 
         return cb({
-            responseStatusCode: 410
+            responseStatusCode: 418
         });
     }
 

plugins/domains/break.com.js

@@ -16,7 +16,7 @@ module.exports = {
             // no '/' at the end and no ?embed=1
 
             return {
-                href: 'http://www.break.com/embed/' + id + '?embed=1',
+                href: 'http://www.break.com/embed/' + id,
                 type: CONFIG.T.text_html,
                 rel: [CONFIG.R.player, CONFIG.R.html5],
                 "aspect-ratio": 464 / 280