refactor: v4

[lowlighter]

See #1533

This PR introduces a major refactor to the codebase to offer more possibilities, make the code cleaner, safer and more maintainable while keeping a compatibility layer to make the migration smoothly for users

📝 Todo-list before pre-release

• Engine
  • Components framework
    • Plugin framework
      • Rendering and templating
    • Processor framework
    • API requests framework
    • Metadata and schemas
      • Inputs parsing
      • Outputs parsing
  • Configuration parsing
  • Versions and paths
  • Processing routine
  • Utilitaries
    • Raw command execution
    • IO and storage abstractions
    • Browser execution
      • Using local cached browser
      • Using remote browser service
    • Formatting
    • GitHub helpers
    • Graph rendering
      • Time graph
      • Line graph
      • Pie graph
      • Diff graph
    • Logging framework
    • Markdown rendering
      • Language syntax highlighting
    • Testing framework
• GitHub Action
  • Docker image
    • Publish to ghcr.io
  • Implement publish.gist
  • Implement publish.file
  • Implement publish.git
  • Compatibility layer
    • v3 to v4 inputs conversion
    • .legacy plugin to support v3 plugins directly (retrieving output from docker image)
• Web server
  • Sync models values
  • Config crafter
    • Retrieve schemas from server
    • Auto-generation from inputs based on schema
      • Show name and descriptions
      • Support basic inputs
      • Support unions
      • Support nested objects
      • Support arrays
      • Support record
      • Support secrets
    • Plugins and processors context edition
    • Configuration migration helper
    • Print scopes, permissions and linked markdown documentation
    • Add/Remove plugins and processors
    • Preview final configuration
    • Preview rendered SVG
  • OAuth support
• Serverless deployment
  • Deno deploy
• Plugins
  • Experiment with the engine
    • A simple plugin (introduction)
    • A plugin that requires browser execution (webscraping)
    • A plugin that requires an external library (rss)
    • A plugin that requires GraphQL API (calendar, gists, ...)
    • A plugin that requires REST API (lines, activity, ...)
    • A plugin that requires rendering markdown (activity)
    • A plugin that requires rendering d3 graphs (lines)
    • A plugin that requires executing a raw command (legacy)
  • Migrations
    • Legacy plugin
    • Activity
    • Calendar
    • Gists
    • Introduction
    • Lines
    • RSS
    • Webscraping
• Processors
  • Rendering
  • Render Twemojis, GitHub emojis and octicons
  • Transform to base64
  • Optimizations (svg, xml, css)
  • Inject raw content, script and style
  • Meta-directive (assert, delay, error)
  • Publish directives (console, file, gist, git)
• Execution directives
  • serve (see "web server" section)
  • run (see "github action" section)
  • create (skeleton for plugin and processors)
  • --version and --help
• Miscelleanous
  • Unit testing
  • Coverage

[lowlighter]

✈️ Migration guide will be made at the same time that cli/compat.ts is implemented

[lowlighter]

This is an unfinished draft that will be publied eventually in announcement when this pr somewhat ready. Subject to change

---

4️⃣ Metrics v4 _{alpha preview}

Note

While not available yet, metrics v4 is expected to offer a pre-release in the coming weeks.
You can track the progression in this pull request #1542, stay tuned !

🧩 Plugins changes and new features

🏛️ Legacy plugins _.legacy

It will take some time before all plugins and options are eventually migrated to v4.
Fortunately, the new .legacy plugin lets you call plugins directly from v3.x versions!

It will pull the corresponding docker image, pass your inputs, generate the render within v3 context and retrieve it inside the v4 engine.

Here's a simple example of how to achieve this:

plugins:
  - .legacy:
      version: v3.34
      inputs:
        plugin_isocalendar: yes
        plugin_isocalendar_duration: full-year

📰 Recent activity _activity

This plugin has been completly redesigned, to offer more data, code highlighting, and better event filtering.

📆 Commit calendar _calendar

Current isocalendar and calendar have been merged together to ensuring features consistency between both.
It means that you can now generate isometric calendars over multiples years, while the top-down calendar now have additional statistics along each year. You can also change colors more easily.

💻 Lines of code changed _lines

This plugin has been redesigned to offer more data granularity, where you'll be able to show diff per week and by contributor.
Additionally, the issue with the stats sometimes being empty have been fixed in this release!

📦 Miscellaneous

All plugins are now totally independant (actually the base plugin has not even be migrated yet).

Each migration also take a look at current opened issues, so if you've opened one and haven't receive a response yet it may actually be fixed in v4 ! Note that it also means that development of v3 is currently "frozen"

The following plugins have been migrated but don't really have any notable changes:

• 🎫 Gists
• 🙋 Introduction
• 🗼 Rss feed
• 📸 Webscraping

🪄 Processors and new features

The core plugin is being decommissioned entirely as it's being split into different "processors", which will offer more flexibility and power.

You can apply processors on a per-plugin basis, on even on a group basis using the .await plugin (which await for all previous plugins execution). Thanks to this you can now do neat tricks, such as using different templates per plugin, publishing your renders under different formats to various destinations, etc.

plugins:
  - introduction:
    template: terminal 
  - gists:
  - .await: 
    processors:
    - inject.style:
        style: .title { color: red; }
    - render:
        format: svg
    - optimize.css:
    - render.twemojis:
    - transform.base64:
    - publish.git:
        repository: my/repository-A
          commit:
            to: metrics-render-branch 
  - processors: # ".await" plugin is actually implicit :)
    - render:
        format: json
    - publish.gist:
        gist: ...

In the future, it is planned to offer more option to aggregate data together and also "rehydrate" data from cached data.

Below is a list of currently supported processors:

• 🧪 Assertions
• ⏰ Delay
• 🚩 Throw an error
• 🔩 Inject raw content
• 🔩 Inject JavaScript
• 🔩 Inject CSS
• 🧹 Optimize CSS
• 🧹 Optimize SVG
• 🧹 Optimize XML
• 📮 Publish to console
• 📮 Publish to local file
• 📮 Publish to GitHub Gist
• 📮 Publish to GitHub repository
• 🎨 Render image
• 🖋️ Render GitHub emojis
• 🖋️ Render GitHub octicons
• 🖋️ Render Twemojis
• 🏭 Transform images to base64 encoded images

🧼 Templates simplification

The classic template (the default one) is being reworked heavily.
The aim is to use simpler styling rules (without compromising design) to improve performances, but also to make it easily overridable by users through the new injection processors.

They're also designed to have better class conventions to make it easily manipulable with document.querySelector(). Additionnaly, styling related to single plugins are being moved out of the global stylesheet to avoid loading useless rules when a plugin is unused.

<section class="introduction">
  <div class="title">
    <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16">...</svg>
    About ...
  </div>
  <div class="body">
    <!-- ... -->
  </div>
  <style>
    .introduction .text {
      white-space: normal;
      margin: 0 13px 2px;
    }
  </style>
</section>

🚂 An entirely rewritten engine focusing on...

🔐 Security

Metrics now support GitHub fine-grained tokens and deno permissions, which means you are guaranteed to have full control on which data this action has access to and what it can do with it.

Since specifying all permissions may be tedious, metrics is relying on deno_make and describe the "minimal permissions subset" in the deno.jsonc configuration file, but you are free to specify your own permissions !

🍱 Portability

Configuration syntax has been simplified and unified across the different modes of execution which are officially supported.
Same configuration file can now be reused across the different modes.

• Web server
• GitHub Action
• Docker image ^{🌟 Improved}
• Serverless ^{✨ New !}
• CLI / binary ^{✨ New !}

☁️ Deployability

Currently the web version is hosted on a server that I rent. Since it's managed by myself I sometimes forget to do it (who said "invalid ssl certificate" ?).

The plan with this new engine is to make it compatible with serverless services, such as deno deploy and vercel (with deno), and also a docker image that is designed to be used outside of the metrics GitHub Action.

To achieve this, all features relying on prerequirements (browser, filesystem, etc.) are being abstracted so they can be replaced (for example the browser can use a remote service such as browserless.io or just being made optional. It probably won't be possible to achieve features parity but it'll still be close to it.

This will also be used to test and preview new pull requests changes.

🗿 Stability

Fully rewritten in TypeScript, configuration, inputs and outputs are now fully typed and strictly checked directly in the code with Zod, making it easier to work with Metrics.

/** Inputs */
readonly inputs = is.object({
  feed: is.string().url().default("https://news.ycombinator.com/rss").describe("RSS feed (e.g. `https://news.ycombinator.com/rss`)"),
  limit: is.number().int().min(1).nullable().default(4).describe("Display limit. Set to `null` to disable"),
})

/** Outputs */
readonly outputs = is.object({
  name: is.string().describe("Feed name"),
  description: is.string().describe("Feed description"),
  entries: is.array(is.object({
    title: is.string().describe("Entry title"),
    link: is.string().url().describe("Entry link"),
    date: is.date().nullable().describe("Entry date"),
  })).describe("Feed entries"),
})

🔄 Versatility

While it is not entirely ready yet, in the future you will be able to import metrics plugin and processors directly in external programs. The new inputs/outputs validations schemas ensure a stable API with consistant results.

So you can built new apps upon metrics:

import { Plugin } from "https://deno.land/x/metrics/plugins/mod.ts";
const {result} = await Plugin.run({
  context: {
    id: "introduction",
    entity: "user",
    handle: "octocat",
    retries: {},
    mock: true
  }
})

🧪 Reliability

The code is now tested more thoroughly with better tests and less hacky workarounds. It also now features code coverage.
The aim is to provide a fully tested codebase with a coverage close to 100%.

⚡ Rapidity

The boot-time is now significantly lower as code has been refactored and optimized to decrease bottlenecks (no more dependencies on base and core plugins, and no need to load up all plugins at startup).

Most of queries have been optimized to consume less requests, and templates have been refactored to reduce the number of nodes created to make simpler outputs.

Unlike node_modules, ESM modules offers easier tree-shaking which also means that running a non-cached version of the code will be faster as only the used code will be fetched to run it

♻️ Reusability

As mentionned previously, same plugin can now be called multiple times using differents scopes and contexts (e.g. different users, tokens, timezones, api endpoints, templates, etc.).

The end goal is to offer an easier version of the existing presets system by integrating it directly in the configuration file.
It means that you will be able to reuse a configuration from another user just by overriding the variables you want to.

plugins:
  - calendar:
      handle: octocat
      token: ${{ secrets.TOKEN_OCTOCAT }}
      timezone: Europe/Paris
  - calendar:
      handle: octodog
      token: ${{ secrets.TOKEN_OCTODOG }}
      timezone: America/Montreal

🎴 Extensibility

Using the power of ESM modules, you can easily include remote versions of plugins and templates (perfect to rollback a given plugin on a specific version or to use a custom version of it). Less patches to apply and more flexibility!

plugins:
  - https://raw.githubusercontent.com/lowlighter/metrics/v4/source/plugins/calendar/mod.ts:
      handle: octocat
    template: https://raw.githubusercontent.com/lowlighter/metrics/v4/source/plugins/calendar/templates/classic.ejs

🏛️ Compatibility

Configurations from v3 and earlier versions will still work thanks to the compatibility layer which will automatically convert old inputs into the new v4 format, and offering suggestions and tips for migration. Apart from really niche options and design changes, migrating will be almost transparent.

🧰 Ergonomy

The web UI has been entierly redesigned with Alpine.js and Tailwind CSS to make it simpler and friendlier. The inputs are now auto-generated through the inputs JSON schemas. Thanks to the improved test data, mocked previews are now more close to what will eventually be generated

🤝 Community

While the old codebase survived more than 50 versions, it wasn't really friendly for new contributions.
This rewrite hopes to make it easier to contribute by:

• Providing a clean codebase which is fully typed, commented, tested and covered
• Making it easier to setup and deploy by only requiring deno to be installed, and officially supporting serverless services
• Simplifying how to add plugins by making them self-contained (code, tests, templates and docs), and make it easier to use custom plugins and templates not from the official codebase

💪 Interested in contributing ?
Check out issues with the 🆕 v4 label
Contributions for v4 will be accepted once #1542 is landed
Whether it's to migrate old plugins from v3 to v4, or create new plugins or options, all contributions are welcomed !

💕 Liking (the future of) metrics ?
You can sponsor me to support development !
Even one-time contributions are welcomed 🥰

[github-actions]

This pull request seems to be inactive and will be automatically closed in a few days if it remains without any activity.

[Andre601]

This pull request seems to be inactive and will be automatically closed in a few days if it remains without any activity.

lmao.

[Andre601]

I just received a PR from dependabot that updates metrics to v4..... while it still isn't released here yet...
How does that even work?!