v1.5.0-rc.0

[ruslandoga]

👋

The highlights of this release are:

• City and region level geolocation (disabled by default, please see the comments below for the guide)
• Google Analytics integration: https://plausible.io/docs/google-analytics-import (WIP setup guide: add self-hosting GA-imports configuration docs#278)
• CSV data export: https://plausible.io/docs/export-stats
• ARM64 support

Upgrading Plausible Analytics to v1.5

In your docker-compose.yml update the image used for plausible to tag 1.5.0-rc.0:

plausible:
- image: plausible/analytics:v1.4
+ image: plausible/analytics:1.5.0-rc.0

Restart the container

$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

This will boot up the new version of the app.

Upgrading ClickHouse to 22.6 (Optional, but recommended)

In your docker-compose.yml, update the image used for plausible_events_db to the newest stable ClickHouse version:

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/community-edition#45

Upgrading PostgreSQL to 14 (Optional, but recommended)

Upgrading PostgreSQL is a bit trickier, here's the upgrade guide.

Changelog

Following changes have been made since v1.4:

Added

• The ability to escape | characters with \ in Stats API filter values
• An upper bound of 1000 to the limit parameter in Stats API
• The exclusions script extension now also takes a data-include attribute tag
• A file-downloads script extension for automatically tracking file downloads as custom events
• Integration with Matomo's referrer spam list to block known spammers
• API route PUT /api/v1/sites/goals with form params site_id, event_name and/or page_path, and goal_type with supported types event and page
• API route DELETE /api/v1/sites/goals/:goal_id with form params site_id
• The public breakdown endpoint can be queried with the "events" metric
• Data exported via the download button will contain CSV data for all visible graps in a zip file.
• Region and city-level geolocation City level location data #1449
• The u option can now be used in the manual extension to specify a URL when triggering events.
• Delete a site and all related data through the Sites API
• Subscribed users can see their Paddle invoices from the last 12 months under the user settings
• Allow custom styles to be passed to embedded iframe Allow custom styles to be passed to embedded iframe #1522
• New UTM Tags utm_content and utm_term Support for utm_term and utm_content #515
• If a session was started without a screen_size it is updated if an event with screen_size occurs
• Added LISTEN_IP configuration parameter no pg_hba.conf entry for host when connecting to Postgres via Heroku #1189
• The breakdown endpoint with the property query property=event:goal returns custom goal properties (within props)
• Added IPv6 Ecto support (via the environment-variable ECTO_IPV6)
• New filter type: contains, available for page, entry_page, exit_page
• Add filter for custom property
• Add ability to import historical data from GA: [Continued] Google Analytics import #1753
• API route GET /api/v1/sites/:site_id
• Hovering on top of list items will now show a tooltip with the exact number instead of a shortened version
• Filter goals in realtime filter by clicking goal name
• The time format (12 hour or 24 hour) for graph timelines is now presented based on the browser's defined language
• Choice of metric for main-graph both in UI and API (visitors, pageviews, bounce_rate, visit_duration) Adds Main Graph Metric Selection #1364
• New width=manual mode for embedded dashboards Embed improvements #2148
• Add more timezone options
• Add new strategy to recommend timezone when creating a new site
• Alert outgrown enterprise users of their usage Manually lock outgrown enterprise accounts #2197
• Manually lock and unlock enterprise users Manually lock outgrown enterprise accounts #2197
• ARM64 support for docker images ARM64 support for docker images #2103
• Add support for international domain names (IDNs) International Domain Names (IDN) Support #2034

Fixed

• Plausible script does not prevent default if it's been prevented by an external script plausible/analytics#1941
• Hash part of the URL can now be used when excluding pages with script.exclusions.hash.js.
• UI fix where multi-line text in pills would not be underlined properly on small screens.
• UI fix to align footer columns
• Guests can now use the favicon to toggle additional info about the site bing viewed (such as in public embeds).
• Fix SecurityError in tracking script when user has blocked all local storage
• Prevent dashboard graph from being selected when long pressing on the graph in a mobile browser
• The exported pages.csv file now includes pageviews again plausible/analytics#1878
• Fix a bug where city, region and country filters were filtering stats but not the location list
• Fix a bug where regions were not being saved
• Timezone offset labels now update with time changes
• Render 404 if shared link auth cannot be verified plausible/analytics#2225
• Restore compatibility with older format of shared links plausible/analytics#2225
• Fix 'All time' period for sites with no recorded stats plausible/analytics#2277
• Ensure settings page can be rendered after a form error plausible/analytics#2278

Changed

• script.file-downloads.outbound-links.js only sends an outbound link event when an outbound download link is clicked
• Plausible script now uses callback navigation (instead of waiting for 150ms every time) when sending custom events
• Cache the tracking script for 24 hours
• Move entry_page and exit_page to be part of the Page filter group
• Paginate /api/sites results and add a View all link to the site-switcher dropdown in the dashboard.
• Remove the + Add Site link to the site-switcher dropdown in the dashboard.
• DISABLE_REGISTRATIONS configuration parameter can now accept invite_only to allow invited users to register an account while keeping regular registrations disabled Adds "invite_only" disable_registration config option #1841
• New and improved Session tracking module for higher throughput and lower latency. PR#1934
• Do not display ZZ country code in countries report PR#1934
• Add fallback icon for when DDG favicon cannot be fetched PR#2279

---

This discussion was created from the release v1.5.0-rc.0.

[rphlo]

Can you describe how to enable city level geolocation? And if it is disabled wouldn't it be possible to hide the buttons "Regions" and "Cities" on the Location widget?

[ruslandoga]

@rphlo sure! Here's a quick guide.

⚠️ Note that this city level geolocation feature requires extra ~1GB of RAM to work. If you don't have enough RAM available, you might experience slow container startups and restarts.

⚠️ dbip-city-lite might not include all the necessary info for this feature, consider using maxmind, see replies below for details.

---

City level geolocation can be enabled by mounting a volume with the downloaded geonames.csv and dbip-city-lite-2022-10.mmdb and setting GEONAMES_SOURCE_FILE and IP_GEOLOCATION_DB to point to them:

$ # custom csv can be used, see https://github.com/plausible/location#cities for more info
$ curl -O https://s3.eu-central-1.wasabisys.com/plausible-application/geonames.csv

$ # https://db-ip.com/db/download/ip-to-city-lite, maxmind can also be used but it requires authorization
$ curl -O https://download.db-ip.com/free/dbip-city-lite-2022-10.mmdb.gz
$ gunzip dbip-city-lite-2022-10.mmdb.gz

docker-compose.yml

  plausible:
    image: plausible/analytics:1.5.0-rc.0
+   volumes:
+     - ./geonames.csv:/etc/app/geonames.csv:ro
+     - ./dbip-city-lite-2022-10.mmdb:/etc/app/dbip-city-lite-2022-10.mmdb:ro

plausible-conf.env

ADMIN_USER_EMAIL=replace-me
ADMIN_USER_NAME=replace-me
ADMIN_USER_PWD=replace-me
BASE_URL=replace-me
SECRET_KEY_BASE=replace-me
+ GEONAMES_SOURCE_FILE=/etc/app/geonames.csv
+ IP_GEOLOCATION_DB=/etc/app/dbip-city-lite-2022-10.mmdb

[ruslandoga]

PR that switches the geolocation library for the one that automatically loads and updates maxmind dbs: #2362

[joshavant]

@ruslandoga Re: City level geolocation requiring an extra ~1GB of RAM - Would it be possible to run this release on a server with only 1GB of RAM total, for a low traffic website (< 100 pageviews/day)? Would any performance hit only be tied to slow servers and restarts?

[ruslandoga]

@joshavant I think it should be possible if you use the built-in geonames file (= no GEONAMES_SOURCE_FILE set) and a maxmind geolite2 city mmdb.

[joshavant]

@ruslandoga Sorry if I'm missing something but... there is a built-in genomes file?

What's the advantages/differences of using the CSV file in the comments above (e.g. https://s3.eu-central-1.wasabisys.com/plausible-application/geonames.csv )?

[ruslandoga]

@joshavant it's the one that's included by default in the :location dependency, cities500.txt: https://github.com/plausible/location#citie. And it's used when GEONAMES_SOURCE_FILE is not set.

What's the advantages/differences of using the CSV file in the comments above

It has more geoname id -> city info mappings, that's all.

[rphlo]

For my second point which was hidding the "Regions" and "Cities" Pills in the locations widget if this feature is not enabled, I implemented myself.
Commit is here rphlo@6a6495e and could be added to the rc1 version...

[ruslandoga]

Nice! We could do it automatically though, without a flag, by checking if the ip db has city info data.

[rphlo]

Sure, that be nice, but that is beyond my know how with elixir

[mannp]

Done the postgres 12 to 14 upgrade successfully and was already on the specified clickhouse version.

Seems clickhouse is not happy with the migration....any pointers as to how to solve it please? Thanks in advance :)

Loading plausible..
Starting dependencies..
Starting repos..
Running migrations for Elixir.Plausible.Repo
Running migrations for Elixir.Plausible.ClickhouseRepo
** (Clickhousex.Error) Code: 10. DB::Exception: Wrong column name. Cannot find column `entry.meta` to drop. (NOT_FOUND_COLUMN_IN_BLOCK) (version 22.6.9.11 (official build))

    (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
    (elixir 1.13.4) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
    (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:1024: Ecto.Adapters.SQL.execute_ddl/4
    (ecto_sql 3.8.3) lib/ecto/migration/runner.ex:352: Ecto.Migration.Runner.log_and_execute_ddl/3
    (ecto_sql 3.8.3) lib/ecto/migration/runner.ex:117: anonymous fn/6 in Ecto.Migration.Runner.flush/0
    (elixir 1.13.4) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto_sql 3.8.3) lib/ecto/migration/runner.ex:116: Ecto.Migration.Runner.flush/0
    (stdlib 3.17.1) timer.erl:166: :timer.tc/1
bash: started: command not found

[ruslandoga]

It would need to be done once. After the migration runs successfully, it won't be run again, so removing the mounted file would be fine.

[ukutaht]

@ruslandoga Thanks, if there are more issues with this then it wouldn't hurt to make this change in the main codebase as well.

[mannp]

Thanks @ruslandoga, gave it a try and now have this error;

Running migrations for Elixir.Plausible.ClickhouseRepo
** (CompileError) lib/plausible-0.0.1/priv/clickhouse_repo/migrations/20220421161259_remove_entry_props.exs:7: undefined function remove_if_exists/1 (expected Plausible.ClickhouseRepo.Migrations.RemoveEntryProps to define such a function or for it to be imported, but none are available)
    (elixir 1.13.4) lib/code.ex:404: Code.validated_eval_string/3

[ruslandoga]

@mannp sorry, the migration I wrote is indeed incorrect. How about just doing nothing in it? That would definitely work with the same end result.

defmodule Plausible.ClickhouseRepo.Migrations.RemoveEntryProps do
  use Ecto.Migration

  def change do
  end
end

[mannp]

@mannp sorry, the migration I wrote is indeed incorrect. How about just doing nothing in it? That would definitely work with the same end result.

defmodule Plausible.ClickhouseRepo.Migrations.RemoveEntryProps do
  use Ecto.Migration

  def change do
  end
end

Thanks, that worked.

[mannp]

Hi @ruslandoga has GEOLITE2_COUNTRY_DB now been depreciated? as I get GeoLite2-City.mmdb updated automatically.

[ruslandoga]

No, it's not deprecated and would continue to work as before: if IP_GEOLOCATION_DB is not set, fallback to GEOLITE2_COUNTRY_DB, and if GEOLITE2_COUNTRY_DB is not set, fallback to dbip-country-lite downloaded when container image was built.

At least right now, IP_GEOLOCATION_DB and GEOLITE2_COUNTRY_DB are equivalent, but when both are specified, IP_GEOLOCATION_DB takes precedence.

[mannp]

GEOLITE2_COUNTRY_DB

Great, thanks for the clarification.

[paskal]

Nitpick, previous versions in dockerhub were named like v1.4.4, I propose to release future versions with leading v also, and add such tag for already published image.

[ruslandoga]

I agree. For some reason https://github.com/docker/metadata-action drops the v: https://github.com/docker/metadata-action#semver

@ukutaht should we maybe keep the v in the container image tag?

[ukutaht]

Yes we should for consistency with older versions

[ruslandoga]

#2340

[ruslandoga]

v1.5.0-rc.1 discussion: #2460

[ruslandoga]

v2.0.0-rc.0 discussion: #2896