NEWS

OpenTSDB - Changelog
* Version 2.4.1 (2021-09-02)

Noteworthy Changes:
  - Add a config flag to enable splitting queries across the rollup and
    raw data tables. Another config determines when to split the queries.
  - Fix for CVE-2020-35476 that now validates and limits the inputs for 
    Gnuplot query parameters to prevent remote code execution.
  - Default log config will log CLI tools at INFO to standard out.
  - New check_tsd_v2 script that evaluates individual metric groups given
    a filter.
  - Collect stats from meta cache plugins.
  - Add a python script to list and pretty-print queries running on a TSD.
  - Add a single, standalone TSD systemd startup script and default to that
    instead of the multi-port TSD script.
  
Bug Fixes:
  - Fix the "--sync" flag for FSCK to wait for repairs to execute against
    storage before proceeding.
  - Fix expression queries to allow metric only and filterless queries.
  - Fix an NPE in /api/query/last when appends are enabled.
  - Fix races in the salt scanner and multigets on the storage maps.
  - Fix rollup queries that need sum and count for downsampling instead of
    group bys.
  - Fix fuzzy row filters for later versions of HBase by using filter pairs.
    And allow it to be combined with a regex filter.
  - Fix stats from the individual salt scanners and overall query stats
    concurrency issues.
  - Rename the stat "maxScannerUidtoStringTime" to 
    "maxScannerUidToStringTime"
  - Fix the min initial value for double values in the AggregationIterator
  - Fix rollup queries producing unexpected results.
  - Fix various UTs
  - Support rollups in FSCK

* Version 2.4.0 (2018-12-16)

Noteworthy Changes:
  - Set default data block encoding to `DIFF` in the create table script.
  - Add callbacks to log errors in the FSCK tool when a call was made to
    fix something.
  - Add a sum of squares aggregator "squareSum".
  - Add the diff aggregator that computes the difference between the first
    and last values.
  - Add a SystemD template to the RPM package.
  - Allow tags to be added via HTTP header.
  - Add example implementations for the Authorization and Authentication
    plugins.
  - Change `tsd.storage.use_otsdb_timestamp` to default to false.
  - Literal or filter now allows single character values.
  - Rollup query code now only uses the downsampler value to pick an interval.
  - Add jdk 8 in the debian script.
  - Setup fill policies in the Nagios check

Bug Fixes:
  - Fix rollup scanner filter for single aggregate queries.
  - Fix FSCK HBase timestamps when deduping. Sometimes they were negative.
  - Fix exception handling when writing data over HTTP with the sync flag enabled.
  - Fix missing source files in the Makefile.
  - Change UID cache to longs from ints and add hit and miss counters.
  - Fix HighestCurrent returning the wrong results.
  - Fix running query stats queryStart timestamp to millis.
  - Fix TimeShift millisecond bug.
  - Fix post remove step in the debian package.

* Version 2.3.2 (2018-12-16)

Noteworthy Changes:
  - A new Python wrapper script to make FSCK repair runs easier.
  - Track performance in the Nagios/Icinga script
  - Add a Contributions file.
  - Add a config, 'tsd.core.bulk.allow_out_of_order_timestamps' to allow out of
    order timestamps for bulk ingest.
  - NOTE: This version also includes a JDK 8 compiled version of Jackson due to 
    security patches. If you need to run with an older JDK please replace the 
	Jackson JARs with older versions.

Bug Fixes:
  - Unwrap NoSuchUniqueIds when writing data points to make it easier to understand
    exceptions.
  - Fix an NPE in the PutDataPointRpc class if a data point in the list is null. 
  - Fix a Makefile error in the clean portion.
  - Fix an NPOE in the UIDManager print result.
  - Fix a bug in the UI where Y formats may contain a percent sign.
  - Allow specifying the data block encoding and TTL in the HBase table creation
    script.
  - Change the make and TSDB scripts to use relative paths.
  - Fix parsing of `use_meta` from the URI for the search endpoint.
  - Fix the clean cache script to be a bit more OS agnostic.


* Version 2.4.0 RC2 (2017-10-08)

Noteworthy Changes:
  - Modify the RPC handler plugin system so that it parses only the first part of 
    the URI instead of the entire path. Now plugins can implement sub-paths.
  - Return the HTML 5 doctype for built-in UI pages
  - Add an optional byte and/or data point limit to the amount of data fetched
    from storage. This allows admins to prevent OOMing TSDs due to massive queries.
  - Allow a start time via config when enabling the date tiered compaction in HBase
  - Provide the option of using an LRU for caching UIDs to avoid OOMing writers and
    readers with too many strings
  - Optionally avoid writing to the forward or reverse UID maps when a specific TSD
    operational mode is enabled to avoid wasting memory on maps that will never be
    used.

* Version 2.3.1 (2018-04-21)

Noteworthy Changes:
  - When setting up aggregators, advance to the first data point equal to or greater
    than the query start timestamp. This helps with calendar downsampling intervals.
  - Add support to the Nagios check script for downsampling fill policies.

Bug Fixes:
  - Fix expression calculation by avoiding double execution and checking both
    output types for boolean values.
  - Fixing missing tools scripts in builds.
  - Default HBase 1.2.5 in the OSX install script
  - Upgrade AsyncBigtable to 0.3.1
  - Log query stats when a channel is closed unexpectedly.
  - Add the Java 8 path in the debian init script and remove Java 6.
  - Pass the column family name to the get requests in the compaction scheduler.
  - Fix a comparison issue in the UI on group by tags.
  - Filter annotation queries by the starting timestamp, excluding those in a row that
    began before the query start time.
  - Tiny stap at purging backticks from Gnuplot scripts.
  - Remove the `final` annotation from the meta classes so they can be extended.
  - Fix the javacc maven plugin version.
  - Fix the literal or filter to allow single character filters.
  - Fix query start stats logging to use the ms instead of nano time.
  - Move Jackson and Netty to newer versions for security reasons.
  - Upgrade to AsyncHBase 1.8.2 for compatibility with HBase 1.3 and 2.0
  - Fix the Highest Current calculation to handle empty time series.
  - Change the cache hits counters to longs.

* Version 2.3.0 (2016-12-31)

Noteworthy Changes:
  - Release of 2.3.0.
  - Add example classes for using the Java API.

Bug Fixes:
  - Same fixes as in 2.2.2
  - Fix a null UID check on decoding metric names from row keys.
  - Fix unit tests for JDK 8 and later.

* Version 2.2.2 (2016-12-29)

Bug Fixes:
  - Fix an issue with writing metadata where using custom tags could cause the compare-
    and-set to fail due to variable ordering in Java's heap. Now tags are sorted so the
    custom tag ordering will be consistent.
  - Fix millisecond queries that would miss data the top of the final hour if the end
    time was set to 1 second or less than the top of that final hour.
  - Fix a concurrent modification issue where salt scanners were not synchronized on the
    annotation map and could cause spinning threads.

* Version 2.3.0 RC2 (2016-10-08)

Noteworthy Changes:
  - Added a docker file and tool to build TSD docker containers (#871).
  - Log X-Forwarded-For addresses when handling HTTP requests.
  - Expand aggregator options in the Nagios check script.
  - Allow enabling or disabling the HTTP API or UI.
  - TSD will now exit when an unrecognized CLI param is passed.
  
Bug Fixes:
  - Improved ALPN version detection when using Google Bigtable.
  - Fix the DumpSeries class to support appended data point types.
  - Fix queries where groupby is set to false on all filters.
  - Fix a missing attribute in the Nagios check script (#728).
  - Fix a major security bug where requesting a PNG with certain URI params could execute code
    on the host (#793, #781).
  - Return a proper error code when dropping caches with the DELETE HTTP verb (#830).
  - Fix backwards compatibility with HBase 0.94 when using explicit tags by removing the
    fuzzy filter (#837).
  - Fix an RPM build issue when creating the GWT directory.s

* Version 2.2.1 (2016-10-08)

Noteworthy Changes
  - Generate an incrementing TSMeta request only if both enable_tsuid_incrementing and
    tsd.core.meta.enable_realtime_ts are enabled. Previously, increments would run
    regardless of whether or not the real time ts setting was enabled. If tsuid 
    incrementing is disabled then a get and optional put is executed each time without 
    modifying the meta counter field.
  - Improve metadata storage performance by removing an extra getFromStorage() call.
  - Add global Annotations to the gnuplot graphs (#773)
  - Allow creation of a TSMeta object without a TSUID (#778)
  - Move to AsyncHBase 1.7.2

Bug Fixes:
  - Fix Python scripts to use the environment directory.
  - Fix config name for "tsd.network.keep_alive" in included config files.
  - Fix an issue with the filter metric and tag resolution chain during queries.
  - Fix an issue with malformed, double dotted timestamps (#724).
  - Fix an issue with tag filters where we need a copy before modifying the list.
  - Fix comments in the config file around TCP no delay settings.
  - Fix some query stats calculations around averaging and estimating the number
    of data points (#784).
  - Clean out old .SWO files (#821)
  - Fix a live-lock situation when performing regular expression or wildcard queries (#823).
  - Change the static file path for the HTTP API to be relative (#857).
  - Fix an issue where the GUI could flicker when two or more tag filters were set (#708).

* Version 2.3.0 RC1 (2016-05-02)

Noteworthy Changes:
  - Introduced option --max-connection/tsd.core.connections.limit to set the maximum number
    of connection a TSD will accept (#638)
  - 'tsdb import' can now read from stdin (#580)
  - Added datapoints counter (#369)
  - Improved metadata storage performance (#699)
  - added checkbox for showing global annotations in UI (#736)
  - Added startup plugins, can be used for Service Discovery or other integration (#719)
  - Added MetaDataCache plugin api
  - Added timeshift() function (#175)
  - Now align downsampling to Gregorian Calendar (#548, #657)
  - Added None aggregator to fetch raw data along with first and last aggregators to 
    fetch only the first or last data points when downsampling.
  - Added script to build OpenTSDB/HBase on OSX (#674)
  - Add cross-series expressions with mathematical operators using Jexl
  - Added query epxressions (alias(), scale(), absolute(), movingAverage(), highestCurrent(),
    highestMax(), timeShift(), divide(), sum(), difference(), multiply()) (#625)
  - Add a Unique ID assignment filter API for enforcing UID assignment naming conventions.
  - Add a whitelist regular expression based UID assignment filter
  - Add a time series storage filter plugin API that allows processing time series data
    and determining if it should be stored or not.
  - Allow using OpenTSDB with Google's Bigtable cloud platform or with Apache Cassandra
    
Bug Fixes:
  - Some improperly formatted timestamps were allowed (#724)
  - Removed stdout logging from packaged logback.xml files (#715)
  - Restore the ability to create TSMeta objects via URI
  - Restore raw data points (along with post-filtered data points) in query stats
  - Built in UI will now properly display global annotations when the query string is passed

* Version 2.2.0 (2016-02-14)

Noteworthy Changes
  - Rework the QueryStats output to be a bit more useful and add timings from the
    various scanners and query components.
  - Modify the UI to allow for group by or aggregate per tag (use the new query feature)
  - Rework the UI skin with the new TSDB logo and color scheme.
  - Add the QueryLog config to logback.xml so users can optionally enable logging of
    all queries along with their stats.

Buf Fixes:
  - Properly handle append data points in the FSCK utility.
  - Fix FSCK to handle salting properly.
  - Fix the IncomingDataPoints class for the CLI import tool to properly account for
    salting.
  - Fix the QueryStats maps by making sure the hash accounts for an unmodified list of
    filters (return copies to callers so sorting won't break the hash code).
  - Fix the case-insensitive wildcard filter to properly ignore the case.
  - Fix the CLI dumper/scan utility to properly handle salted data.
  - Fix a case where the compaction queue could grow unbounded when salting was enabled.
  - Allow duplicate queries by default (as we did in the past) and users must now block
    them explicitly.
  - Fix the /api/stats endpoint to allow for returning a value if the max UID width is
    set to 8 for any type. Previously it would throw an exception.
  - Add a try catch to FSCK to debug issues where printing a problematic row would cause
    a hangup or no output.

* Version 2.2.0 RC3 (2015-11-11)

Bug Fixes:
  - Fix build issues where the static files were not copied into the proper location.

* Version 2.2.0 RC2 (2015-11-09)

Noteworthy Changes:
  - Allow overriding the metric and tag UID widths via config file instead of
    having to modify the source code.

Bug Fixes:
  - OOM handling script now handles multiple TSDs installed on the same host.
  - Fix a bug where queries never return if an exception is thrown from the
    storage layer.
  - Fix random metric UID assignment in the CLI tool.
  - Fix for meta data sync when salting is enabled.
  - 

* Version 2.2.0 RC1 (2015-09-12)

Noteworthy Changes:
  - Add the option to randomly assign UIDs to metrics to improve distribution across
    HBase region servers.
  - Introduce salting of data to improve distribution of high cardinality regions
    across region servers.
  - Introduce query stats for tracking various timings related to TSD queries.
  - Add more stats endpoints including /threads, /jvm and /region_clients
  - Allow for deleting UID mappings via CLI or the API
  - Name the various threads for easier debugging, particularly for distinguishing
    between TSD and AsyncHBase threads.
  - Allow for pre-fetching all of the meta information for the tables to improve
    performance.
  - Update to the latest AsyncHBase with support for secure HBase clusters and RPC
    timeouts.
  - Allow for overriding metric and tag widths via the config file. (Be careful!)
  - URLs from the API are now relative instead of absolute, allowing for easier reverse
    proxy use.
  - Allow for percent deviation in the Nagios check
  - Let queries skip over unknown tag values that may not exist yet (via config)
  - Add various query filters such as case (in)sensitive pipes, wildcards and pipes
    over tag values. Filters do not work over metrics at this time.
  - Add support for writing data points using Appends in HBase as a way of writing
    compacted data without having to read and re-write at the top of each hour.
  - Introduce an option to emit NaNs or Nulls in the JSON output when downsampling and
    a bucket is missing values.
  - Introduce query time flags to show the original query along with some timing stats
    in the response.
  - Introduce a storage exception handler plugin that will allow users to spool or
    requeue data points that fail writes to HBase due to various issues.
  - Rework the HTTP pipeline to support plugins with RPC implementations.
  - Allow for some style options in the Gnuplot graphs.
  - Allow for timing out long running HTTP queries.
  - Text importer will now log and continue bad rows instead of failing.
  - New percentile and count aggregators.
  - Add the /api/annotations endpoint to fetch multiple annotations in one call.
  - Add a class to support improved bulk imports by batching requests in memory for a 
    full hour before writing.
  
Bug Fixes:
  - Modify the .rpm build to allow dashes in the name.
  - Allow the Nagios check script to handle 0 values properly in checks.
  - Fix FSCK where floating point values were not processed correctly (#430)
  - Fix missing information from the /appi/uid/tsmeta calls (#498)
  - Fix more issues with the FSCK around deleting columns that were in the list (#436)
  - Avoid OOM issues over Telnet when the sending client isn't reading errors off it's
    socket fast enough by blocking writes.

* Version 2.1.4 (2016-02-14)

Bug Fixes:
  - Fix the meta table where the UID/TSMeta APIs were not sorting tags properly 
    prior to creating the row key, thus allowing for duplicates if the caller changed
    the order of tags.
  - Fix a situation where meta sync could hang forever if a routine threw an exception.
  - Fix an NPE thrown when accessing the /logs endpoint if the Cyclic appender is not
    enabled in the logback config.
  - Remove an overly chatty log line in TSMeta on new time series creation.

* Version 2.1.3 (2015-11-11)

Bug Fixes:
  - Fix build issues where the static files were not copied into the proper location.

* Version 2.1.2 (2015-11-09)

Bug Fixes:
  - Fix the built-in UI to handle query parameter parsing properly (found when Firefox 
    changed their URI behavior)
  - Fix comments about the Zookeeper quorum setting in various config files.
  - Fix quoting in the Makefile when installing.
  - Make sure builds write files in the proper location on FreeBSD.

* Version 2.1.1 (2015-09-12)

Bug Fixes:
  - Relax the pgrep regex to correctly find and kill the java process in the RPM init.d
    script.
  - Improve query performance slightly when aggregating multiple series.
  - Fix the /api/search/lookup API call to properly handle the limit parameter.
  - Fix the /api/query/last endpoint to properly handle missing tsdb-meta tables.

* Version 2.1.0 (2015-05-06)

Bug Fixes:
  - FSCK was not handling compacted and floating point duplicates properly. Now they 
    are merged correctly.
  - TSMeta data updates were not loading the latest data from storage on response
  - The config class will now trim spaces from booleans and integers
  - On shutdown, the idle state handler could prevent the TSD from shutting down
    gracefully. A new thread factory sets that thread as a daemon thread.
  - TSMeta objects were not generated if multiple writes for the same data point arrived
    in succession due to buffering atomic increments. Increments are no longer buffered.
  - Updated paths to the deprecated Google Code repo for dependencies.

* Version 2.1.0 RC2 (2015-04-04)

Noteworthy Changes:
  - Handle idle connections in Netty by closing them after some period of inactivity
  - Support compressed HTTP responses

Bug Fixes:
  - Various RPM script and package fixes
  - Properly handle deletions of the cache directory while running
  - Queries for non-extant metrics now return a 400 error
  - Fix an issue with the meta sync utility where entries were not created if the counter existed
  - Report stats properly when the UID is greater than 3 bytes wide
  - Fix UI hangups when incorrect tags are supplied
  - Log illegal put exceptions at the debug level
  - Fix global annotation retrieval where some entries were missing
  - Fix unit tests that were not properly mocking out clients or threads and causing JVM OOMs
  - Fix accounting bugs in FSCK
  - Sort aggregators in the UI
  - Properly throw an exception if the user supplies an empty tag on storage or retreival
  - Handle missing directories in the Config class

* Version 2.1.0 RC1 (2014-11-09)

Noteworthy Changes:
  - Add a server side timeout for sockets that haven't written data in some time
  - Major FSCK utility update to handle new objects, delete bad data and deal with duplicate data points.
  - Optionally preload portions of the name to UID maps at startup
  - Add read and write modes to the TSD to disable writing data points via telnet or HTTP
  - Optionally disable the diediedie commands to prevent users from shutting down a tsd
  - Optionally block the auto creation of tag keys and values
  - Downsampling is now aligned on modulus bondaries so that we avoid interpolation as much as possible. Data returned is now more along the lines of what users expect, e.g. 24 data points for day when downsampled on hourly intervals instead of random points based on the span's timestamps.
  - Add the /api/search/lookup endpoint and CLI endpoint for looking up time series based on the meta or data tables
  - Rework of the TSD compaction code to process compactions faster
  - Optionally handle duplicate data points gracefully during compaction or query time without throwing exceptions
  - Add Allow-Headers CORs support

* Version 2.0.1 (2014-11-09)

Bug Fixes:
  - Fix tree REST endpoint to allow modification of the strictMatch and storeFailure fields
  - Fix compacted millisecond columns throwing an error during fsck
  - Fix tsd startup directory permission checks
  - Fix thread safety of pending UID assignment when they are complete
  - Fix compaction queue flushing on tsd shutdown
  - Fix RTPublisher plugin floating point value detection where it was previously posting to the wrong method
  - Fix TsdbQuery.toString() to avoid looing up UIDs if an error was thrown as this can cause cause a deadlock

Noteworthy changes:
  - Add Travis CI build support
  - Update to Asynchbase 1.6.0 and update additional dependencies
  - Modify scripts to allow easier compilation under FreeBSD

* Version 2.0.0 (2014-05-5)

API incompatible changes:
  - The `TSDB' class now takes a `Config' object in argument instead of an
    HBaseClient and two strings.
  - The downsampling interval for the method `Query.downsample' went from
    being an `int' to a `long'.

Noteworthy changes:
  - Configuration can be provided in a properties file
  - New Jackson JSON helper class
  - GnuPlot batch file for Windows compatability
  - Add relative time option "n" for 30 days
  - Relative, unix epoch style timestamps work in CliQuery
  - New "max" parameter for /suggest that can fetch more than the default 25 
    results. If not supplied, default is used
  - New formalized HTTP API, deprecates many of the old HTTP API calls but it
    is still backwards compatible
  - New store data points over HTTP via JSON
  - New annotations for recording meta data along with data points in a time
    series
  - New global annotations to record meta data at a specific time but not
    associated with a specific series
  - New meta data for metrics, tag names, tag values and time series
  - New optional chunked encoding support for HTTP requests, configurable
  - Millisecond resolution support for writing data points and annotations
  - Variable length encoding for integer data points, any where from 1 to 8
    bytes instead of using 8 bytes for every point
  - CORs support for the HTTP API
  - New data injest plugin framework allowing support for different data
    formats
  - Search plugin framework to push meta data to a search engine for access
  - Real-Time publisher framework to publish data points to a secondary system
    as soon as they are received at the TSD
  - New aggregation functions with alternatives to interpolation including:
    - zero if missing sum: returns a 0 if a data point doesn't exist
    - max if missing min: returns the maximum value if a data point is missing
    - min if missing max: returns the minimum value if a data point is missing
  - New TSUID tracking that writes a counter to a new table to track the 
    different time series stored and how many data points were written
  - New meta data trees that allow for flattening time series into a 
    heirarchical representation similar to a file system
  - New meta and tree synchronization CLI tools
  - New statistics showing the number of UIDs used and available for each type
  - New statistics for the number of current connections to the TSD
  - New options for working with rate calculations on counters to rollover
    or reset on anomallys
  - New Debian package compilable from the source
  - New RPM package compilable from the source



* Version 1.1.1 (2013-??-??) [???????]

Noteworthy changes:
  - UIDs are now assigned in a lock-less fashion.



* Version 1.1.0 (2013-03-08) [12879d7]

Noteworthy changes:
  - Licensing adjustment: allow LGPLv2.1+ in addition to LGPLv3+.
  - Various fixes used when customizing size of UniqueId.  The default size
    is 3 bytes and is a compile-time constant rarely changed in practice.
  - New a new standard deviation aggregator, `dev'.
  - New `fgcolor', `bgcolor' and `smooth' query parameters to /q.
  - New `tz' query string parameter to allow specifying a custom time zone.
  - Stop accepting connections when shutting down.
  - A new `dropcaches' administrative command allows discarding in-memory
    caches.  Right now these are UID mappings.
  - Browser history support in the web UI.
  - Allow "1d-ago" style input in date boxes.
  - Fix the 30d integer overflow in the web UI.
  - Add the ability to use mouse for drag-to-zoom on graphs.
  - Integration with Maven.
  - Work around a Netty performance bug, increasing write throughput by 10x.
  - Properly parse floating point values in scientific notations.
  - Allow tuning the number of worker threads or using OIO.
  - Fix auto-completion bug causing partial results to show in the web UI.
  - Various internal bug fixes.
  - Update all dependencies.
  - TSDB data compaction is now enabled by default.


* Version 1.0.0 (2011-12-23) [66a6b42]

Initial release:
  - Upload data points through a telnet-style protocol.
  - HTTP interface to query data in ASCII and PNG.
  - Efficient, fully asynchronous write path.
  - Synchronous / blocking read path (to be rewritten).
  - TSDB data compaction (disabled by default).

-----

Copyright (C) 2011-2012  The OpenTSDB Authors.

This library is free software: you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License as published
by the Free Software Foundation, either version 2.1 of the License, or
(at your option) any later version.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License
along with this library.  If not, see <http://www.gnu.org/licenses/>.

Local Variables:
mode: outline
End: