Skip to content

kapouer/url-inspector

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

url-inspector

Synopsys

npx url-inspector <url>

Description

Get normalized metadata about what a URL mainly represents.

This is a Node.js module.

Sources of information:

  • HTTP response headers
  • embedded tags in binary formats (using exiftool)
  • OpenGraph, Twitter Cards, schema.org, json+ld, title and meta tags in HTML pages
  • oEmbed endpoints
  • if a URL is mainly wrapping a media, that media might be inspected too

Inspection stops when enough information has been gathered, or when a maximum number of bytes (depending on media type) have been downloaded.

Format

  • url: url of the inspected resource
  • title: title of the resource, or filename, or last component of pathname with query
  • description: optional longer description, without title in it, and only the first line.
  • site: the name of the site, or the domain name
  • mime: RFC 7231 mime type of the resource (defaults to Content-Type) The inspected mime type could be more accurate than the http header.
  • ext: The file extension, only derived from the mime type. Safe to be used as file extension.
  • what: what the resource represents page, image, video, audio, file
  • type: how the resource is used: link, image, video, audio, embed. Example: if what:image and mime:text/html, and no html snippet is found, type will be 'link'.
  • html: the html representation of the resource, according to type and use.
  • script: url of a script that must be installed along with the html representation.
  • date (YYYY-MD-DD format) creation or modification date
  • author: optional credit, author (without the @ prefix and with _ replaced by spaces)
  • keywords: optional array of collected keywords (lowercased words that are not in title words).
  • size: optional Content-Length as integer; discarded when type is embed
  • icon: optional link to the favicon of the site
  • width, height: optional dimensions as integers
  • duration: optional hh:mm:ss string
  • thumbnail: optional a URL to a thumbnail, could be a data-uri for embedded images
  • source: optional a URL that can go in a 'src' attribute; for example a resource can be an html page representing an image type. The URL of the image itself would be stored here; same thing for audio, video, embed types.
  • error: optional an http error code, or string

Install

url-inspector currently requires those external libraries/tools:

  • exiftool
  • libcurl (and libcurl-dev if node-libcurl needs to be rebuilt)

Both programs are well-maintained, and available in most linux distributions.

Usage

import Inspector from 'url-inspector';

const opts = {
 ua: "Mozilla/5.0", // override ua, defaults to somewhat modern browser
 nofavicon: false, // disable additional requests to get a favicon
 nosource: false, // disable main embedded media sub-inspection
 file: true, // local files inspection is only enabled by default when using CLI
 meta: {} // user-entered metadata, to be merged and normalized
 providers: null // custom providers (module path or array)
};

const inspector = new Inspector(opts);

const obj = await inspector.look(url);

Inspector throws http-errors instances.

By default oembed providers are

  • found from a curated list of providers
  • found from a custom list, required from opts.providers
  • discovered in the inspected web pages

It is possible to add custom providers in the options, by passing an array or a path to a module exporting an array.

See src/custom-oembed-providers.js for examples.

To normalize an already existing metadata object, including url rewriting done by providers, and other changes in fields, do:

await inspector.norm(obj);

url-inspector uses node-libcurl to make http requests, and exposes it as:

const req = await Inspector.get(urlObj);

where req.abort() stops the request, req.res is the response stream, and res.statusCode, res.headers are available.

Proxy support

url-inspector configures http(s) proxies through proxy-from-env package and environment variables (http_proxy, https_proxy, all_proxy, no_proxy):

Read proxy-from-env documentation.

License

Open Source, see ./LICENSE.