Investigate debug logs of ecchronos

[masokol]

Investigate debug logs of ecchronos to see how much is actually logged. Maybe some stuff should be throttled or removed/moved to trace.

[masokol]

There's a lot of irrelevant logs on debug lvl. We should move them to trace instead.

[VictorCavichioli]

I will take this

[DanielwEriksson]

There are a few times debug is LOGed. Which should be kept and/or changed to info? A DoD needs to be defined for this issue

epkdaek@elx721027t9:/github/ecchronos$ git grep "LOG.debug"
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/ECChronosInternals.java: LOG.debug("Table {} last repaired at {}", tableReference, lastRepairedAt);
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/ECChronosInternals.java: LOG.debug("Table {} remaining repair time {}", tableReference, remainingRepairTime);
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Watching for changes in {}", absoluteFilePath);
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Watch service has been closed");
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Received event for {}/{}", baseDirectory, file);
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/spring/CassandraHealthIndicator.java: LOG.debug("Unable to connect over JMX", e);
application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/spring/CassandraHealthIndicator.java: LOG.debug("Unable to connect over CQL", e);
connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalJmxConnectionProvider.java: LOG.debug("Connecting JMX through {}, credentials: {}, tls: {}", jmxUrl, authEnabled, tlsEnabled);
connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalJmxConnectionProvider.java: LOG.debug("Connected JMX for {}", jmxUrl);
connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalNativeConnectionProvider.java: LOG.debug("Connecting to {}({}), local data center: {}", contactEndPoint, initialContact.getHostId(),
connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalNativeConnectionProvider.java: LOG.debug("Driver configuration: {}", driverConfigLoader.getInitialConfig().getDefaultProfile().entrySet());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/CASLock.java: LOG.debug("Locally highest priority ({}) is higher than current ({}), will not remove",
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/HostStatesImpl.java: LOG.debug("Host {} marked as up", host);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/HostStatesImpl.java: LOG.debug("Host {} marked as down", host);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/LockCache.java: LOG.debug("Encountered cached locking failure, throwing exception", e);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/TableStorageStatesImpl.java: LOG.debug("{} -> {}", tableReference, diskSpaceUsed);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("{} switched state to UP.", node);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("{} switched state to DOWN.", node);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("Session during setupConfiguration call was null.");
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/IncrementalRepairJob.java: LOG.debug("{} - last successful run: {}", this, myLastSuccessfulRun);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/IncrementalRepairTask.java: LOG.debug("{} for range {}", repairStatus, range);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairGroup.java: LOG.debug("Table {} running repair job {}", myTableReference, myReplicaRepairGroup);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairGroup.java: LOG.debug("", e);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("Found cached locking failure for {}, rethrowing", repairResource, e);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("{} - Unable to get repair resource lock '{}', releasing previously acquired locks - {}",
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("Lock ({} in datacenter {}) got error {}", resource, dataCenter, e.getMessage());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("{} completed successfully", this);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("Notification {}", notification.toString());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("Unknown JMXConnectionNotification type: {}", notification.getType());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeOnDemandRepairJob.java: LOG.debug("Total tokens for this job are 0");
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeRepairTask.java: LOG.debug("Unknown ranges: {}", unknownRanges);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeRepairTask.java: LOG.debug("Completed ranges: {}", completedRanges);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/EccRepairHistory.java: LOG.debug("Token range {} was not found in metadata", tokenRange);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/RepairStateImpl.java: LOG.debug("Table {} fully repaired at {}, next repair at/after {}", myTableReference,
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/RepairStateImpl.java: LOG.debug("Table {} partially repaired at {}, next repair at/after {}", myTableReference,
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("No last repaired at found for {}, iterating over all repair entries", tableReference);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Table {} snapshot created at {}, iterating repair entries until that time", tableReference,
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Ignoring entry {}, repair was not successful", repairEntry);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Ignoring entry {}, replicas {} not matching participants", repairEntry, nodes);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Run policy {} added", runPolicy);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Run policy {} removed", runPolicy);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Job {} rejected for {} ms by {}", job, nextRun, runPolicy);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Trying to acquire lock for {}", task);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Removing job: {}", job);
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Adding job: {}, Priority: {}", job, job.getPriority());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Retrieving job: {}, Priority: {}", job, job.getPriority());
core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/utils/ReplicatedTableProviderImpl.java: LOG.debug("Keyspace {} not replicated by local node, ignoring.", keyspace);
epkdaek@elx721027t9:/github/ecchronos$

[DanielwEriksson]

This is the suggestion

epkdaek@elx721027t9:/github/ecchronos$ git grep "LOG.debug"
INFO: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/ECChronosInternals.java: LOG.debug("Table {} last repaired at {}", tableReference, lastRepairedAt);
INFO: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/ECChronosInternals.java: LOG.debug("Table {} remaining repair time {}", tableReference, remainingRepairTime);
INFO: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Watching for changes in {}", absoluteFilePath);
INFO: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Watch service has been closed");
DEBUG: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/config/ConfigRefresher.java: LOG.debug("Received event for {}/{}", baseDirectory, file);
DEBUG: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/spring/CassandraHealthIndicator.java: LOG.debug("Unable to connect over JMX", e);
DEBUG: application/src/main/java/com/ericsson/bss/cassandra/ecchronos/application/spring/CassandraHealthIndicator.java: LOG.debug("Unable to connect over CQL", e);
INFO: connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalJmxConnectionProvider.java: LOG.debug("Connecting JMX through {}, credentials: {}, tls: {}", jmxUrl, authEnabled, tlsEnabled);
INFO: connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalJmxConnectionProvider.java: LOG.debug("Connected JMX for {}", jmxUrl);
INFO: connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalNativeConnectionProvider.java: LOG.debug("Connecting to {}({}), local data center: {}", contactEndPoint, initialContact.getHostId(),
INFO: connection.impl/src/main/java/com/ericsson/bss/cassandra/ecchronos/connection/impl/LocalNativeConnectionProvider.java: LOG.debug("Driver configuration: {}", driverConfigLoader.getInitialConfig().getDefaultProfile().entrySet());
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/CASLock.java: LOG.debug("Locally highest priority ({}) is higher than current ({}), will not remove",
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/HostStatesImpl.java: LOG.debug("Host {} marked as up", host);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/HostStatesImpl.java: LOG.debug("Host {} marked as down", host);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/LockCache.java: LOG.debug("Encountered cached locking failure, throwing exception", e);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/TableStorageStatesImpl.java: LOG.debug("{} -> {}", tableReference, diskSpaceUsed);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("{} switched state to UP.", node);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("{} switched state to DOWN.", node);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/DefaultRepairConfigurationProvider.java: LOG.debug("Session during setupConfiguration call was null.");
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/IncrementalRepairJob.java: LOG.debug("{} - last successful run: {}", this, myLastSuccessfulRun);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/IncrementalRepairTask.java: LOG.debug("{} for range {}", repairStatus, range);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairGroup.java: LOG.debug("Table {} running repair job {}", myTableReference, myReplicaRepairGroup);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairGroup.java: LOG.debug("", e);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("Found cached locking failure for {}, rethrowing", repairResource, e);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("{} - Unable to get repair resource lock '{}', releasing previously acquired locks - {}",
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairLockFactoryImpl.java: LOG.debug("Lock ({} in datacenter {}) got error {}", resource, dataCenter, e.getMessage());
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("{} completed successfully", this);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("Notification {}", notification.toString());
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/RepairTask.java: LOG.debug("Unknown JMXConnectionNotification type: {}", notification.getType());
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeOnDemandRepairJob.java: LOG.debug("Total tokens for this job are 0");
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeRepairTask.java: LOG.debug("Unknown ranges: {}", unknownRanges);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/VnodeRepairTask.java: LOG.debug("Completed ranges: {}", completedRanges);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/EccRepairHistory.java: LOG.debug("Token range {} was not found in metadata", tokenRange);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/RepairStateImpl.java: LOG.debug("Table {} fully repaired at {}, next repair at/after {}", myTableReference,
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/RepairStateImpl.java: LOG.debug("Table {} partially repaired at {}, next repair at/after {}", myTableReference,
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("No last repaired at found for {}, iterating over all repair entries", tableReference);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Table {} snapshot created at {}, iterating repair entries until that time", tableReference,
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Ignoring entry {}, repair was not successful", repairEntry);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/repair/state/VnodeRepairStateFactoryImpl.java: LOG.debug("Ignoring entry {}, replicas {} not matching participants", repairEntry, nodes);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Run policy {} added", runPolicy);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Run policy {} removed", runPolicy);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Job {} rejected for {} ms by {}", job, nextRun, runPolicy);
DEBUG: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduleManagerImpl.java: LOG.debug("Trying to acquire lock for {}", task);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Removing job: {}", job);
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Adding job: {}, Priority: {}", job, job.getPriority());
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/scheduling/ScheduledJobQueue.java: LOG.debug("Retrieving job: {}, Priority: {}", job, job.getPriority());
INFO: core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/utils/ReplicatedTableProviderImpl.java: LOG.debug("Keyspace {} not replicated by local node, ignoring.", keyspace);
epkdaek@elx721027t9:/github/ecchronos$

[DanielwEriksson]

Some might vene be changed to error?

LOG.debug("Unable to connect over JMX", e);
and
LOG.debug("Unable to connect over CQL", e);

sound like an error to me

[DanielwEriksson]

the changed logging shall use

https://github.com/Ericsson/ecchronos/blob/master/core/src/main/java/com/ericsson/bss/cassandra/ecchronos/core/utils/logging/ThrottlingLogger.java

[jwaeab]

Part of this will be done in issue #666