Merge master with federation (#773)

* Revert "Usage of postgres in test environment for database consistency (#686)" (#688)

This reverts commit 932a61f.

* Mitigate CVE-2020-13935 (#690)

* Bump spring parent version

* Liveness health indicator according to. spring-projects/spring-boot#22107

* Revert to test postgresql docker image using test-containers. (#694)

* Test cases and debug support.for submission (#698)

* Add integration test for future submission payload debugging

* Added more test cases around invalid temp key parameter scenarios

* Add support for printing the submission payload during tests

* Enhance debug test with sql statements generation

* Update services/submission/src/test/java/app/coronawarn/server/services/submission/integration/SubmissionPersistenceIT.java

Co-authored-by: Michael Frey <michael.frey@gmx.ch>

Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Co-authored-by: Michael Frey <michael.frey@gmx.ch>

* Update maven.config with current version 1.4.0-SNAPSHOT (#697)

Set current version to 1.3.0-SNAPSHOT

Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>

* -fix invalid test and name refactoring (#703)

* Add app-features as external configuration (#704)

* Add app features in configuration file
Adjust tests

* Rename env variable for plausible deniability active.
Remove env name for app feature label.

* Adjust test application.yaml

* Change active type from boolean to integer

* Change app feature 'active' property to 'value'

* use same version for spring-boot-starter-parent accross all components (#709)

* Fix/700 retention policy (#711)

* fix retention policy of stored diagnosis keys

* fix typo

* fix failing test case due to previously injected time

* refactor method names

* Submission Service - updated documentation (#713)

* Updated the validation for the submission service in SUMBISSION.md to match the new rules for TEK's

* markdownlint fixes

* Update docs/SUBMISSION.md

Co-authored-by: Sorin Stefan Iovita <sorin.stefan.iovita@sap.com>

Co-authored-by: Ioan Gut <ioan.gut@sap.com>
Co-authored-by: ioangut <67064882+ioangut@users.noreply.github.com>
Co-authored-by: Sorin Stefan Iovita <sorin.stefan.iovita@sap.com>

* Submission Service - Enhance TEK's  (#712)

* -initial draft commit

* updated checkDuplicateStartIntervalNumberLimit wip

* updated checkDuplicateStartIntervalNumberLimit

* -fix test broke after solved conflict

* changed checkDuplicateStartIntervalNumberLimit to use HashMap

* -fix checklist reports

* add env variables for submission service

* review update add more tests

* solved code smells

* rename attribute maxRollingPeriod

* renamed getter getMaxRollingPeriod

* Update ValidSubmissionPayload.java

* refactor implementation for consistency

* fix checkstyle errors

modify the Violation text accordingly

* fix code smell

* Update ValidSubmissionPayload.java

* Change validation for payloads based on rolling period

* Capture more test scenarios

* add checkstyle indentation

* create test data generation class

* add header to the new class

* resolved review conversation

* fixed checkstyle errors

* resolved code smell

* fix compile error

* create valid TEK, fix test failing

Co-authored-by: Emmet <emmet.power@sap.com>
Co-authored-by: Eugen M <eugen.madean@sap.com>
Co-authored-by: Pit Humke <pithumke@users.noreply.github.com>

* Feature/enrich app config (#726)

* Added supported countries to app config with application.yaml parameter

* Fixed Test failures

* Remove unused code

* Added app version parameters to application.yaml

* Added tests for app version parameters

* Refacted country list implementation

Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>

* Fix Sonar issues (#754)

* Bugfix/submission tek rolling period (#752)

* Replaced possible variable rolling period with constant value

* Add tests for flexible rolling period

* apply sugested review changes

* Add javaDoc comments for changed getExpiryDateTime

* Fix midnight check in SubmissionPayload Validator

Co-authored-by: Michael Burwig <michael.burwig@sap.com>

* Feature/update risk score master (#758)

* Update risk-score-classification.yaml

Problem in iOS 13.7 --> Unbekanntes Risiko bei Risikobegegnung

* Adjust tests

Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Co-authored-by: Michael Burwig <michael.burwig@sap.com>

* Update exposure-config.yaml (#765)

patch transmission as requested by Thomas Augsten

Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>

* Resolve merge conflicts

* Resolve github comments

Co-authored-by: Frederico <fred.rbittencourt@gmail.com>
Co-authored-by: EugenM-SAP <67458405+EugenM-SAP@users.noreply.github.com>
Co-authored-by: Hilmar Falkenberg <hilmar.falkenberg@sap.com>
Co-authored-by: Michael Frey <michael.frey@gmx.ch>
Co-authored-by: Evgenii Skrebtcov <67064767+EvgeniiSkrebtcov@users.noreply.github.com>
Co-authored-by: ioangut <67064882+ioangut@users.noreply.github.com>
Co-authored-by: emmetsap <emmet.power@sap.com>
Co-authored-by: Ioan Gut <ioan.gut@sap.com>
Co-authored-by: Eugen M <eugen.madean@sap.com>
Co-authored-by: Pit Humke <pithumke@users.noreply.github.com>
Co-authored-by: KevponSAP <66735382+KevponSAP@users.noreply.github.com>
Co-authored-by: Michael Burwig <michael.burwig@sap.com>

common/persistence/src/main/java/app/coronawarn/server/common/persistence/service/common/KeySharingPoliciesChecker.java

@@ -88,11 +88,12 @@ public LocalDateTime getEarliestTimeForSharingKey(DiagnosisKey diagnosisKey, Exp
 
   /**
    * Returns the end of the rolling time window that a {@link DiagnosisKey} was active for as a {@link LocalDateTime}.
+   * The ".plusDays(1L)" is used as there can be now diagnosis keys with rollingPeriod set to less than 1 day.
    */
   private LocalDateTime getRollingPeriodExpiryTime(DiagnosisKey diagnosisKey) {
     return LocalDateTime
         .ofEpochSecond(diagnosisKey.getRollingStartIntervalNumber() * TEN_MINUTES_INTERVAL_SECONDS, 0, UTC)
-        .plusMinutes(diagnosisKey.getRollingPeriod() * DiagnosisKey.ROLLING_PERIOD_MINUTES_INTERVAL);
+        .plusDays(1L);
   }
 
   /**

services/distribution/src/main/java/app/coronawarn/server/services/distribution/assembly/appconfig/validation/ParameterSpec.java

@@ -52,7 +52,7 @@ private ParameterSpec() {
   /**
    * The allowed maximum value for a risk score.
    */
-  public static final int RISK_SCORE_MAX = 72;
+  public static final int RISK_SCORE_MAX = 9999;
 
   /**
    * The allowed minimum value for an attenuation threshold.

services/distribution/src/main/java/app/coronawarn/server/services/distribution/objectstore/publish/LocalFile.java

@@ -84,12 +84,12 @@ private String loadChecksum() {
 
   protected String createS3Key(Path file, Path rootFolder) {
     Path relativePath = rootFolder.relativize(file);
-    return relativePath.toString().replaceAll("\\\\", "/");
+    return relativePath.toString().replace("\\\\", "/");
   }
 
   /**
    * Value for the <code>content-type</code> header.
-   * 
+   *
    * @return Either <a href="https://www.iana.org/assignments/media-types/application/zip">zip</a> or
    *         <a href="https://www.iana.org/assignments/media-types/application/json">json</a>.
    */
@@ -108,7 +108,7 @@ public String getContentType() {
   /**
    * Indicates if a local file is a Key-file or not. Only the Key files are stored in the Date / Hour tree structure.
    * One file per sub-folder (days: 1-31 / hours: 0-23). The index files are not stored in folders ending with a digit.
-   * 
+   *
    * @return <code>true</code> if and only if the {@link #s3Key} ends with a digit, false otherwise.
    */
   public boolean isKeyFile() {

services/distribution/src/main/resources/master-config/exposure-config.yaml

@@ -18,8 +18,8 @@ days_weight: 20
 # Parameters Section
 
 transmission:
-  app_defined_1: 1
-  app_defined_2: 2
+  app_defined_1: 0
+  app_defined_2: 0
   app_defined_3: 3
   app_defined_4: 4
   app_defined_5: 5

services/distribution/src/main/resources/master-config/risk-score-classification.yaml

@@ -1,5 +1,5 @@
 # This is the Risk Score Classification master file. It partitions the risk
-# score value range (0.000 - 72.000) into 2 distinct risk classes. These are
+# score value range (0.000 - 9999.000) into 2 distinct risk classes. These are
 # used by the respective app in order to present the correct risk classification
 # in a user friendly manner, based on the underlying total risk score.
 #
@@ -9,7 +9,7 @@
 # - "HIGH" class: [min, max]
 #
 # The risk classes must not overlap and
-# cover the full risk score value range (0.000 - 72.000).
+# cover the full risk score value range (0.000 - 9999.000).
 #
 # Change this file with caution!
 
@@ -22,5 +22,5 @@ risk-classes:
   -
     label: HIGH
     min: 15
-    max: 72
+    max: 9999
     url: "https://www.coronawarn.app"

services/distribution/src/test/java/app/coronawarn/server/services/distribution/assembly/diagnosiskeys/ProdDiagnosisKeyBundlerExpiryPolicyTest.java

@@ -86,4 +86,47 @@ void testLastPeriodOfHourAndSubmissionGreaterDistributionDateTime() {
     bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
     assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 4, 0, 0), "DE")).hasSize(10);
   }
+
+  @ParameterizedTest
+  @ValueSource(longs = {0L, 24L, 24L + 2L})
+  void testLastPeriodOfHourAndSubmissionLessThanDistributionDateTimeWithFlexibleRollingPeriod(
+      long submissionTimestamp) {
+    List<DiagnosisKey> diagnosisKeys = Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, submissionTimestamp, 5, 44);
+    diagnosisKeys.addAll(Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, submissionTimestamp, 5, 100));
+    bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
+    assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 3, 0, 0),"DE")).hasSize(10);
+  }
+
+  @Test
+  void testLastPeriodOfHourAndSubmissionEqualsDistributionDateTimeWithFlexibleRollingPeriod() {
+    List<DiagnosisKey> diagnosisKeys = Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, 24L + 3L, 5, 44);
+    diagnosisKeys.addAll(Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, 24L + 3L, 5, 100));
+    bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
+    assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 3, 0, 0),"DE")).hasSize(10);
+  }
+
+  @ParameterizedTest
+  @ValueSource(longs = {0L, 24L, 24L + 2L, 24L + 3L})
+  void testFirstPeriodOfHourAndSubmissionLessThanDistributionDateTimeWithFlexibleRollingPeriod(long submissionTimestamp) {
+    List<DiagnosisKey> diagnosisKeys = Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(6, submissionTimestamp, 5, 44);
+    diagnosisKeys.addAll(Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(6, submissionTimestamp, 5, 100));
+    bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
+    assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 4, 0, 0),"DE")).hasSize(10);
+  }
+
+  @Test
+  void testFirstPeriodOfHourAndSubmissionEqualsDistributionDateTimeWithFlexibleRollingPeriod() {
+    List<DiagnosisKey> diagnosisKeys = Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(6, 24L + 4L, 5, 44);
+    diagnosisKeys.addAll(Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(6, 24L + 4L, 5, 100));
+    bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
+    assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 4, 0, 0),"DE")).hasSize(10);
+  }
+
+  @Test
+  void testLastPeriodOfHourAndSubmissionGreaterDistributionDateTimeWithFlexibleRollingPeriod() {
+    List<DiagnosisKey> diagnosisKeys = Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, 24L + 4L, 5, 44);
+    diagnosisKeys.addAll(Helpers.buildDiagnosisKeysWithFlexibleRollingPeriod(5, 24L + 4L, 5, 80));
+    bundler.setDiagnosisKeys(diagnosisKeys, LocalDateTime.of(1970, 1, 5, 0, 0));
+    assertThat(bundler.getDiagnosisKeysForHour(LocalDateTime.of(1970, 1, 2, 4, 0, 0),"DE")).hasSize(10);
+  }
 }

services/distribution/src/test/java/app/coronawarn/server/services/distribution/common/Helpers.java

@@ -109,6 +109,20 @@ public static List<DiagnosisKey> buildDiagnosisKeys(int startIntervalNumber,
         .collect(Collectors.toList());
   }
 
+  public static List<DiagnosisKey> buildDiagnosisKeysWithFlexibleRollingPeriod(
+      int startIntervalNumber, long submissionTimestamp, int number, int rollingPeriod) {
+    return IntStream.range(0, number)
+        .mapToObj(ignoredValue -> DiagnosisKey.builder()
+            .withKeyData(new byte[16])
+            .withRollingStartIntervalNumber(startIntervalNumber)
+            .withTransmissionRiskLevel(2)
+            .withSubmissionTimestamp(submissionTimestamp)
+            .withRollingPeriod(rollingPeriod)
+            .withVisitedCountries(Collections.singletonList("DE"))
+            .withCountryCode("DE").build())
+        .collect(Collectors.toList());
+  }
+
   public static Set<String> getFilePaths(java.io.File root, String basePath) {
     Set<String> files = Arrays.stream(Objects.requireNonNull(root.listFiles()))
         .filter(File::isFile)

services/distribution/src/test/java/app/coronawarn/server/services/distribution/runner/TestDataGenerationTest.java

@@ -37,7 +37,6 @@
 import app.coronawarn.server.services.distribution.config.DistributionServiceConfig.TestData;
 import java.time.LocalDateTime;
 import java.time.ZoneOffset;
-import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.List;

services/distribution/src/test/resources/configtests/app-config_mrs_oob.yaml

@@ -1,3 +1,3 @@
-min-risk-score: 73
+min-risk-score: 10000
 risk-score-classes: !include risk-score-class_ok.yaml
 exposure-config: !include exposure-config_ok.yaml

services/distribution/src/test/resources/configtests/risk-score-class_ok.yaml

@@ -7,5 +7,5 @@ risk-classes:
   -
     label: HIGH
     min: 15
-    max: 72
+    max: 9999
     url: "https://www.coronawarn.app"

services/submission/src/main/java/app/coronawarn/server/services/submission/controller/ApiExceptionHandler.java

@@ -41,19 +41,23 @@ public class ApiExceptionHandler {
   @ExceptionHandler(Exception.class)
   @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
   public void unknownException(Exception ex, WebRequest wr) {
-    logger.error("Unable to handle {}", wr.getDescription(false), ex);
+    logger.error("Unable to handle {}", getFormattedDescription(wr), ex);
   }
 
   @ExceptionHandler({HttpMessageNotReadableException.class, ServletRequestBindingException.class,
       InvalidProtocolBufferException.class})
   @ResponseStatus(HttpStatus.BAD_REQUEST)
   public void bindingExceptions(Exception ex, WebRequest wr) {
-    logger.error("Binding failed {}", wr.getDescription(false), ex);
+    logger.error("Binding failed {}", getFormattedDescription(wr), ex);
   }
 
   @ExceptionHandler({InvalidDiagnosisKeyException.class, ConstraintViolationException.class})
   @ResponseStatus(HttpStatus.BAD_REQUEST)
   public void diagnosisKeyExceptions(Exception ex, WebRequest wr) {
-    logger.error("Erroneous Submission Payload {}", wr.getDescription(false), ex);
+    logger.error("Erroneous Submission Payload {}", getFormattedDescription(wr), ex);
+  }
+
+  private String getFormattedDescription(WebRequest wr) {
+    return wr.getDescription(false).replaceAll("[\n|\r|\t]", "_");
   }
 }

services/submission/src/test/java/app/coronawarn/server/services/submission/controller/SubmissionControllerTest.java

@@ -51,20 +51,16 @@
 import app.coronawarn.server.services.submission.monitoring.SubmissionMonitor;
 import app.coronawarn.server.services.submission.verification.TanVerifier;
 import com.google.protobuf.ByteString;
-
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Locale;
 import java.util.Set;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 import org.apache.commons.lang3.StringUtils;
-import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.TestInstance;
 import org.junit.jupiter.api.TestInstance.Lifecycle;

services/submission/src/test/java/app/coronawarn/server/services/submission/integration/SubmissionPersistenceIT.java

@@ -57,16 +57,15 @@
 import app.coronawarn.server.services.submission.verification.TanVerifier;
 
 /**
- * This test serves more like a dev tool which helps with debugging production issues.
- * It inserts keys parsed from a proto buf file whos content was captured by the mobile
- * client during requests to the server. The content of the current test resource file
- * can be quickly replaced during the investigation of an issue.
+ * This test serves more like a dev tool which helps with debugging production issues. It inserts keys parsed from a
+ * proto buf file whos content was captured by the mobile client during requests to the server. The content of the
+ * current test resource file can be quickly replaced during the investigation of an issue.
  */
 @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
-@ActiveProfiles({ "disable-ssl-client-verification", "disable-ssl-client-verification-verify-hostname" })
+@ActiveProfiles({"disable-ssl-client-verification", "disable-ssl-client-verification-verify-hostname"})
 @Sql(scripts = {"classpath:db/clean_db_state.sql"},
-                executionPhase = ExecutionPhase.BEFORE_TEST_METHOD)
-public class SubmissionPersistenceIT {
+    executionPhase = ExecutionPhase.BEFORE_TEST_METHOD)
+class SubmissionPersistenceIT {
 
   private static final Logger logger = LoggerFactory.getLogger(SubmissionPersistenceIT.class);
 
@@ -97,8 +96,8 @@ public void setUpMocks() {
   @Disabled("Because the content of the .pb file becomes old and retention time passes, this test will fail. "
       + "Enable when debugging of a new payload is required.")
   @ParameterizedTest
-  @ValueSource(strings = { "src/test/resources/payload/mobile-client-payload.pb" })
-  public void testKeyInsertionWithMobileClientProtoBuf(String testFile) throws IOException {
+  @ValueSource(strings = {"src/test/resources/payload/mobile-client-payload.pb"})
+  void testKeyInsertionWithMobileClientProtoBuf(String testFile) throws IOException {
     Path path = Paths.get(testFile);
     InputStream input = new FileInputStream(path.toFile());
     SubmissionPayload payload = SubmissionPayload.parseFrom(input);