v2.0.0-rc.0

[ruslandoga]

The highlights of this release are:

• period comparisons [docs] [example]
• ability to change domains names Allow domain change #2803
• automatic geolocation database downloads replace geolix with locus #2362
• more email adapters support (Mailgun, Mandrill, SendGrid) Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649

Upgrading Plausible Analytics to v2.0

⚠️ ⚠️ ⚠️

Upgrading to v2.0 requires performing a data migration.
Please read these notes until the end before deploying v2.0.0-rc.0

Ensure you are using a new ClickHouse version

The steps below have been tested with clickhouse/clickhouse-server:22.6-alpine please make sure to upgrade ClickHouse to at least this version.

Here's the excerpt from v1.5 release discussion regarding the ClickHouse upgrade

---

In your docker-compose.yml update the image used for plausible_events_db to a newer ClickHouse version:

Upgrading ClickHouse to 22.6

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/community-edition#45

---

Update image tag

In your docker-compose.yml update the image used for plausible to v2.0.0-rc.0

plausible:
- image: plausible/analytics:v1.5.1
+ image: plausible/analytics:v2.0.0-rc.0

and restart the container

$ cd hosting
$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

This will boot up the new version of the app.

If you open the dashboards now, you wouldn't see any past metrics. This is expected as v2.0 uses the new events_v2 and sessions_v2 tables to store analytics data. We need to perform data migration to copy the data into the new tables.

Run data migration

⚠️ ⚠️ ⚠️

You need enough free disk space available for x2 of the current plausible_events_db's event-data volume size. You can use something like docker system df -v | grep hosting_event-data to check how much space the current volume is occupying.

Connect to the running plausible container and start the migration flow

$ cd hosting
$ docker compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

You can attempt this migration multiple times unless you drop v1 tables.

Drop v1 tables (optional)

Once you verify the migration went well, the old tables can be dropped. It's easiest to use clickhouse-client for this

$ ch hosting
$ docker compose exec plausible_events_db clickhouse-client

:) \c plausible_events_db
:) show tables;
:) drop table events;
:) drop table sessions;

See https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#max-table-size-to-drop for how to drop tables with more than 50GB of data.

Enable automatic MaxMind GeoLite2 updates (optional)

In your plausible-config.env set MAXMIND_LICENSE_KEY environment variable and get an automatically updated GeoLite2 City geolocation database. The database edition is configurable with MAXMIND_EDITION environment variable and defaults to GeoLite2-City.

- IP_GEOLOCATION_DB=...
+ MAXMIND_LICENSE_KEY=LNpsJCCKPis6XvBP
+ MAXMIND_EDITION=GeoLite2-City

Note that for the changes in plausible-config.env to propagate to the plausible, the container needs to be recreated:

$ docker compose stop plausible
$ docker compose rm plausible
$ docker compose up -d

Also note that using GeoLite2-City edition requires more RAM than using GeoLite2-Country.

Now you can remove any other volumes and services used to download, store, and update geolocation databases.

Changelog

Following changes have been made since v1.5:

Added

• Ability to use '--' instead of '=' sign in the tagged-events classnames
• 'Last updated X seconds ago' info to 'current visitors' tooltips
• Add support for more Bamboo adapters, i.e. Bamboo.MailgunAdapter, Bamboo.MandrillAdapter, Bamboo.SendGridAdapter Support alternative mailing services (Mailgun, Mandrill, Sendgrid) #2649
• Ability to change domain for existing site (requires numeric IDs data migration, instructions will be provided separately) UI + API (PUT /api/v1/sites)

Fixed

• Make goal-filtered CSV export return only unique_conversions timeseries in the 'visitors.csv' file
• Stop treating page filter as an entry page filter
• City report showing N/A instead of city names with imported data Reject unknown imported cities from queries #2675
• Empty values for Screen Size, OS and Browser are uniformly replaced with "(not set)"
• Fix more pageviews with session prop filter than with no filters
• Cascade delete sent_renewal_notifications table when user is deleted Cascade delete sent_renewal_notifications table when user is deleted #2549
• Show appropriate top-stat metric labels on the realtime dashboard when filtering by a goal
• Fix breakdown API pagination when using event metrics Fix breakdown API pagination when using event metrics #2562
• Automatically update all visible dashboard reports in the realtime view
• Connect via TLS when using HTTPS scheme in ClickHouse URL update ecto to include db url scheme #2570
• Add error message in case a transfer to an invited (but not joined) user is requested improve transfer ownership error message #2651
• Fix bug with showing property breakdown with a prop filter
• Fix bug when combining goal and prop filters Fix unknown bind meta #2654

Changed

• Treat page filter as entry page filter for bounce_rate
• Reject events with long URIs and data URIs Reject events with long URIs and data URIs #2536
• Always show direct traffic in sources reports Always show direct traffic in sources reports #2531
• Stop recording XX and T1 country codes Ignore unknown countries #2556
• Device type is now determined from the User-Agent instead of window.innerWidth Use user-agent instead of screen_width to get device type #2711
• Add padding by default to embedded dashboards so that shadows are not cut off Embedded dashboard fixes #2744
• Update the User Agents database (https://github.com/matomo-org/device-detector/releases/tag/6.1.1)

Removed

• Remove Firewall plug and IP_BLOCKLIST environment variable
• Remove the ability to collapse the main graph Remove the ability to collapse the main graph + transition bug fix #2627

---

This discussion was created from the release v2.0.0-rc.0.

[timscha]

Thank you for the continues updates!
Update worked without problems so far. :)

[ruslandoga]

What about

$ docker compose exec plausible bin/plausible rpc 'IO.inspect(Plausible.Geo.database_type)'

(the output for lookup(1.1.1.1) seems like country edition is being used)

[timscha]

Now it's getting interesting @ruslandoga !

root@docker-ng:/opt/containers/plausible# docker compose exec plausible bin/plausible rpc 'IO.inspect(Plausible.Geo.database_type)'
"GeoLite2-City"

[ruslandoga]

I guess 1.1.1.1 was a bad example. For 3.2.3.4 I'm getting more data:

Plausible.Geo.lookup("3.2.3.4")

%{
  "city" => %{
    "geoname_id" => 4744870,
    "names" => %{"en" => "Ashburn", "ru" => "Ашберн"}
  },
  "continent" => %{
    "code" => "NA",
    "geoname_id" => 6255149,
    "names" => %{
      "de" => "Nordamerika",
      "en" => "North America",
      "es" => "Norteamérica",
      "fr" => "Amérique du Nord",
      "ja" => "北アメリカ",
      "pt-BR" => "América do Norte",
      "ru" => "Северная Америка",
      "zh-CN" => "北美洲"
    }
  },
  "country" => %{
    "geoname_id" => 6252001,
    "iso_code" => "US",
    "names" => %{
      "de" => "Vereinigte Staaten",
      "en" => "United States",
      "es" => "Estados Unidos",
      "fr" => "États Unis",
      "ja" => "アメリカ",
      "pt-BR" => "EUA",
      "ru" => "США",
      "zh-CN" => "美国"
    }
  },
  "location" => %{
    "accuracy_radius" => 20,
    "latitude" => 39.0469,
    "longitude" => -77.4903,
    "metro_code" => 511,
    "time_zone" => "America/New_York"
  },
  "postal" => %{"code" => "20149"},
  "registered_country" => %{
    "geoname_id" => 6252001,
    "iso_code" => "US",
    "names" => %{
      "de" => "Vereinigte Staaten",
      "en" => "United States",
      "es" => "Estados Unidos",
      "fr" => "États Unis",
      "ja" => "アメリカ",
      "pt-BR" => "EUA",
      "ru" => "США",
      "zh-CN" => "美国"
    }
  },
  "subdivisions" => [
    %{
      "geoname_id" => 6254928,
      "iso_code" => "VA",
      "names" => %{
        "en" => "Virginia",
        "fr" => "Virginie",
        "ja" => "バージニア州",
        "pt-BR" => "Virgínia",
        "ru" => "Вирджиния",
        "zh-CN" => "弗吉尼亚州"
      }
    }
  ]
}

[ruslandoga]

Could you please check it as well with

$ docker compose exec plausible bin/plausible rpc 'IO.inspect(Plausible.Geo.lookup("3.2.3.4"))'

[timscha]

At least it works through the states!

root@docker-ng:/opt/containers/plausible# docker compose exec plausible bin/plausible rpc 'IO.inspect(Plausible.Geo.lookup("3.2.3.4"))'
%{
  "city" => %{
    "geoname_id" => 4744870,
    "names" => %{"en" => "Ashburn", "ru" => "Ашберн"}
  },
  "continent" => %{
    "code" => "NA",
    "geoname_id" => 6255149,
    "names" => %{
      "de" => "Nordamerika",
      "en" => "North America",
      "es" => "Norteamérica",
      "fr" => "Amérique du Nord",
      "ja" => "北アメリカ",
      "pt-BR" => "América do Norte",
      "ru" => "Северная Америка",
      "zh-CN" => "北美洲"
    }
  },
  "country" => %{
    "geoname_id" => 6252001,
    "iso_code" => "US",
    "names" => %{
      "de" => "Vereinigte Staaten",
      "en" => "United States",
      "es" => "Estados Unidos",
      "fr" => "États Unis",
      "ja" => "アメリカ",
      "pt-BR" => "EUA",
      "ru" => "США",
      "zh-CN" => "美国"
    }
  },
  "location" => %{
    "accuracy_radius" => 20,
    "latitude" => 39.0469,
    "longitude" => -77.4903,
    "metro_code" => 511,
    "time_zone" => "America/New_York"
  },
  "postal" => %{"code" => "20149"},
  "registered_country" => %{
    "geoname_id" => 6252001,
    "iso_code" => "US",
    "names" => %{
      "de" => "Vereinigte Staaten",
      "en" => "United States",
      "es" => "Estados Unidos",
      "fr" => "États Unis",
      "ja" => "アメリカ",
      "pt-BR" => "EUA",
      "ru" => "США",
      "zh-CN" => "美国"
    }
  },
  "subdivisions" => [
    %{
      "geoname_id" => 6254928,
      "iso_code" => "VA",
      "names" => %{
        "en" => "Virginia",
        "fr" => "Virginie",
        "ja" => "バージニア州",
        "pt-BR" => "Virgínia",
        "ru" => "Вирджиния",
        "zh-CN" => "弗吉尼亚州"
      }
    }
  ]
}

[sandstrom]

A suggestion for 2.0:

Go through all the blessed event names, such as "Outbound Link: Click" (those with a special meaning), and give them a shared prefix and normalized casing.

For example:

• plaus$outbound_link:click (old: Outbound Link: Click)
• plaus$file:downloaded (old: File Download)
• plaus$error:404 (old: 404)

[fabbow1337]

Hey there, thanks so much for the update! Seems to have updated without issues, but wasn't this release supposed to include new metrics like total visits and views per visit? Curious if I had gotten anything wrong during the migration.

[ruslandoga]

It might be hidden behind a feature flag, I'll check.

[ruslandoga]

Yeah, seems so:

analytics/lib/plausible_web/controllers/stats_controller.ex

Lines 104 to 105 in 885ccaa

	FunWithFlags.enabled?(:visits_metric, for: conn.assigns[:current_user]) ->
	[:visitors, :pageviews, :visits, :views_per_visit, :bounce_rate, :visit_duration]

I'll check with the core team what to do about this (I guess it might be already out of preview by the time v2.0.0 is released). But as a workaround you can enable this feature manually on your instance with

$ cd hosting
$ docker compose exec plausible bin/plausible rpc 'FunWithFlags.enable(:visits_metric)'

[fabbow1337]

Thanks so much, fantastic naming here! I'd add that to the changelog, bet people were eagerly awaiting this.

[ruslandoga]

#2898 -- so it'd probably be automatically available in v2.0.0 (or v2.0.0-rc.1 if there is one)

[ruslandoga]

I'd add that to the changelog, bet people were eagerly awaiting this.

Thank you for the suggestion! I've added it as a highlight in #2981 (not sure if it's up to me to change the changelog! :)

[Olakeen]

Does Locus has city location ?
I have this error :

07:39:02.856 request_id=F1wvRwcmP0GpM******* [warning] Could not find city info - code: 3030082

By the way, I had to remove this line from plausible-conf.env to avoid crash :

IP_GEOLOCATION_DB=/etc/app/geolite2-city.mmdb

[ruslandoga]

👋 @Olakeen

Yes, locus does support databases with city-level geolocations.

I think the warning you are seeing comes from location which is used to lookup human-readable place names from geoname ids (only geoname ids are stored in ClickHouse to save space). Here's how to use it with allCountries.txt instead of the default cities500.txt.

[Olakeen]

Ok, but I don't have region and city.
Does it required Maxmind like v1.5 ?

[ruslandoga]

Ok, but I don't have region and city.

You don't have a database with that info or you don't have that info displayed in the dashboards?

Does it required Maxmind like v1.5 ?

Do you mean the DB-IP free version not including city geoname ids? If so, yes, if you want free city-level geolocation, you'd need to use MaxMind's GeoLite2 City geolocation db. With v2.0 changes you only need to set the license key to get that db downloaded and keep it updated.

[Olakeen]

With v2.0 changes you only need to set the license key to get that db downloaded and keep it updated.

Ok perfect, with this two line, it's work perfectly 👍

MAXMIND_LICENSE_KEY=my_licence_key
MAXMIND_EDITION=GeoLite2-City

[youradds]

Thanks for the update! I've just updated my copy, and while I get all the text charts (apart from "Realtime", which is odd!) - I don't get the graph loading. It just has the spinner on it. I notice this in the firebug console:

GET
[https://stats.xx.com/api/stats/graphicsfactory.com/top-stats?period=7d&date=2023-05-09&filters={}&with_imported=true](https://stats.xx.com/api/stats/graphicsfactory.com/top-stats?period=7d&date=2023-05-09&filters=%7B%7D&with_imported=true)
[HTTP/2 500 Internal Server Error 159ms]

Uncaught (in promise) SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data

I did:

Update docker-compose.yml , so we have image: plausible/analytics:v2.0.0-rc.0

Then run:

docker-compose stop plausible
docker-compose rm plausible
docker-compose up -d

Then update the DB:

docker-compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

All of those run fine. However, upon going back to the stats page I get:

https://ibb.co/34XMyMc

[youradds]

** (Ch.Error) Code: 457, e.displayText() = DB::Exception: Value true cannot be parsed as Bool for query parameter '$0' because it isn't parsed completely: only 0 of 4 bytes was parsed: (version 21.3.2.5 (official build))

This seems to be the error that prevents the main graph from loading. Upgrading ClickHouse to 22.6 might help.

Thanks. Any pointers as to how to do that? I'm a Perl kind guy, not so much Docker =) I found this:

https://clickhouse.com/docs/en/operations/update

I'm assuming its not as simple as just changing:

image: yandex/clickhouse-server:21.3.2.5

to a later version, and restarting plausible?

Thanks!

[ruslandoga]

Yes, here's what plausible/hosting is currently using for ClickHouse (configs, image, etc.): https://github.com/plausible/hosting/blob/3e1462eabd510cc9320a6cfed0ae45355d86bb05/docker-compose.yml#L16-L26

---

Sorry, I misread.

I'm assuming its not as simple as just changing:

I think it is. We switched from ClickHouse v21 to v22 during Plausible v1.5.0 upgrade and I don't remember anyone having problems. Let me find the discussion for that release.

[youradds]

Amazing - thanks! That did the trick I was expecting it to be much more complicated than that! haha

[ruslandoga]

From #2507:

---

In your docker-compose.yml update the image used for plausible_events_db to a newer ClickHouse version:

Upgrading ClickHouse to 22.6

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/community-edition#45

---

[youradds]

From #2507:

In your docker-compose.yml update the image used for plausible_events_db to a newer ClickHouse version:

Upgrading ClickHouse to 22.6

plausible_events_db:
- image: yandex/clickhouse-server:21.3.2.5
+ image: clickhouse/clickhouse-server:22.6-alpine

Restart the container

$ docker compose stop plausible plausible_events_db
$ docker compose rm plausible_events_db
$ docker compose up -d

This will boot up the new version of ClickHouse.

Related PR: plausible/hosting#45

Love the new compare feature btw! (main reason for updating). I was trying to compare some dates earlier, and it was a bit of a mission - so nice to now have the option to compare :)

[mathieufrncs]

Hi! I just installed plausible, first v1.5 and I upgraded to v2.0.0. I ran into some problems with the cities.

I installed everything following the docs on the website and I switched to MaxMind data by adding the config file when I start the container (-f geoip/docker-compose.geoip.yml). I tried to add the cities following the v1.5.1 discussion but when I added these lines:

  plausible:
    image: plausible/analytics:v1.5.1
+   volumes:
+     - ./geonames.csv:/etc/app/geonames.csv:ro
+     - ./geolite2-city.mmdb:/etc/app/geolite2-city.mmdb:ro

I got an error saying that geonames.csv is not a file

Kernel pid terminated (application_controller) ({application_start_failure,plausible,{bad_return,{{'Elixir.Plausible.Application',start,[normal,[]]},{'EXIT',{#{'__exception__' => true,'__struct__' => 'Elixir.File.Error',action => <<"stream">>,path => <<"/etc/app/geonames.csv">>,reason => eisdir},[{'Elixir.Enumerable.File.Stream','-reduce/3-fun-0-',3,[{file,"lib/file/stream.ex"},{line,83}]},{'Elixir.Stream','-resource/3-fun-0-',5,[{file,"lib/stream.ex"},{line,1609}]},{'Elixir.Enumerable.Stream',do_each,4,[{file,"lib/stream.ex"},{line,1811}]},{'Elixir.Task.Supervised',stream_reduce,7,[{file,"lib/task/supervised.ex"},{line,323}]},{'Elixir.Stream',run,1,[{file,"lib/stream.ex"},{line,689}]},{timer,tc,1,[{file,"timer.erl"},{line,235}]},{'Elixir.Location',load,1,[{file,"lib/location.ex"},{line,22}]},{'Elixir.Plausible.Application',start,2,[{file,"lib/plausible/application.ex"},{line,42}]}]}}}}})

I'm now running on v2.0.0, I removed the volumes line from the docker-compose.yml file, and added MAXMIND_LICENSE_KEY and MAXMIND_EDITION in the plausible-conf.env file and I'm getting cities in my reports.

1. Do I still need to include the docker-compose.geoip.yml file when starting the container?
2. Are the volumes in docker-compose.yml still needed in v2.0.0?

[ruslandoga]

👋

Do I still need to include the docker-compose.geoip.yml file when starting the container?

No, you don't. In v2.0 the geolocation databases are managed fully within Plausible without 3rd party dependencies.

Are the volumes in docker-compose.yml still needed in v2.0.0?

geonames.csv might still be needed for a geoname_id -> city name mapping in the dashboards.

[ruslandoga]

As for the error, it means that the mounted /etc/app/geonames.csv is a directory and expected to be a file. Check your local ./geonames.csv to be sure.

[mathieufrncs]

Thanks for the details it make much more sense now, I got it fixed!

[myazaki]

Hello I managed to update from 1.5.1 to v2.0.0-rc.0 everything works fine, except the migration of old data. When running

sudo docker compose exec plausible bin/plausible rpc Plausible.DataMigration.NumericIDs.run

I get following error:

--rpc-eval : RPC failed with reason :nodedown

to be honest everything involving "rpc" command ends up the same.

Any idea?

Thanks

[ruslandoga]

RELEASE_DISTRIBUTION=none probably disabled epmd and as the result -- ability to rpc

[ruslandoga]

Off-topic: - ./geolite2-city.mmdb:/etc/app/geolite2-city.mmdb:ro volume is no longer necessary.

[myazaki]

RELEASE_DISTRIBUTION=none probably disabled epmd and as the result -- ability to rpc

so instead of RELEASE_DISTRIBUTION=none what should I use?

[ruslandoga]

Unless there is a good reason to disable distribution, I'd remove that env var completely. Once removed, the defaults would take place (sname) and rpc would start working.

[myazaki]

Unless there is a good reason to disable distribution, I'd remove that env var completely. Once removed, the defaults would take place (sname) and rpc would start working.

Thanks, that did the trick, all migrated. The comment in Documentation for RELEASE_DISTRIBUTION=none missleaded me :-)

[ruslandoga]

v2.0.0-rc.1 has been released!

Changes: v2.0.0-rc.0...v2.0.0-rc.1