CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning from version 3.0.0 on. This Project entered open source status with version 2.1.0. Older versions are not part of this Project.

Unreleased

7.2.4 - 2024-06-06

Changed

• Apply language switch to variable images
• Clear cache during data import
• Move study import to beginning of imports

Fixed

• search highlighting
• several breaking changes from django and elastic upgrades

7.1.0 - 2023-07-19

Changed

• Switched variable label comparison (label table) from django template generation to a combination of REST-API and Typescript.

7.0.0 - 2023-06-02

Added

• site wide language switch.

Changed

• limit variable and question API.
• several smaller UI elements.

Fixed

• publication import problems.
• several incompatibilities with dependency updates.

6.6.0 - 2022-12-06

Added

• Relation between variables and question items.

Changed

• Improved elasticsearch security.
• Display related question items along with related questions in variable view.
• Update script generators.
• Update images and dependencies.

Fixed

• buttons inside the basket view to add/remove variables.
• variable images display.

6.5.2 - 2022-10-27

Fixed

• some outdated names in script generator.

6.5.1 - 2022-10-27

Changed

• Convert types in script_metadata import.

6.5.0 - 2022-10-26

Changed

• Moved datafiles.json, used for workspace script generation, into the database.

6.4.1 - 2022-10-24

Added

• related question label to statistics view.

Changed

• Sorting of related questions in variable view is now done in Database.

6.4.0 - 2022-09-26

Added

• Relationships between instruments and datasets.

Changed

• sorting of datasets table

6.3.1 - 2022-09-01

Added

• list of dataset attachments to variable view.

Fixed

• error in typescript for question view.

6.3.0 - 2022-08-29

Added

• Handle y-limits for numerically displayed ordinal variables.
• new fields to datasets and view of datasets.
• Studies can be sorted in the UI through the setting of their menu_order.

Changed

• Add more screen dependent css rules to statistics view for better display on smaller screens.
• Migrated most JavaScript modules to TypeScript.
• Updated script generator.

Fixed

• an encoding problem caused by certain names of csv files served by the API.
• statistics feedback link target.

Removed

• splinter dependency.

6.2.0 - 2022-06-24

Added

• Added separate views for dataset and instrument tables.

Changed

• feedback system to allow several feedback forms with separate e-mail targets.
• datatables for instruments and datasets to use API data.

6.1.0 - 2022-06-08

Added

• improved instrument import.
• instrument data API.
• n_gram tokenizer for variable names.
• column search for instrument and dataset DataTables.

Changed

• construction of instrument and dataset DataTables.
• Improved question import performance.
• Made feedback button more visible.
• fields used in instrument DataTable.

Fixed

• faulty loop in question_item import.
• usage of Font Awesome 6 icons.

Removed

• unused django templates.

6.0.0 - 2022-04-08

Removed

• django_filer from dependencies (breaking change)

  • Use migrations from commit tagged as filer-fade-out to prepare the database for breaking changes in the migrations.

• prototype for data-api from project.

• unessesairy logging in import.

5.4.1 - 2022-03-21

Fixed

• Do not display Statistic search when SHOW_STATISTICS=False.
• search result link rendering.

5.4.0 - 2022-03-17

Added

• Integrate shiny transfer.server to display statistical data in plots.

• models and imports to store statistical data for variables.

• search feedback page.

Changed

• menu navigation to work better with smaller Displays.
• Configured weight scores for variable search.

5.3.4 - 2022-02-01

Changed

• Allow empty scale field in questions.csv. (use txt as fallback)
• Enabled quick-search for period search facet.
• Period facet entries are now ordered alphabetically.

5.3.3 - 2022-01-27

Changed

• Small refactoring of question view

5.3.2 - 2022-01-06

Changed

• question comparison button icon

5.3.1 - 2022-01-06

Changed

• fields included in the question comparison

Fixed

• question comparison button did not work properly

5.3.0 - 2022-01-05

Changed

• Refactored question view
• Re-added and updated comparison of related questions

5.2.0 - 2021-10-29

Changed

• database storage and import of question related metadata.
• design of the question view.
• Updated several dependencies.

5.1.4 - 2021-09-14

Changed

• Update dependencies
• Refactor to comply with breaking changes of dependencies

Fixed

• Handle description_de field in variable import

5.1.3 - 2021-07-19

Changed

• Refactor import modules.

Fixed

• dependencies to work with Python 3.9 containers.
• Don't show question scale field when no scale is present.
• scrolling in variable view description modal.

5.1.2 - 2021-06-24

Changed

• Make use of models.AutoField explicit for id fields.

Fixed

• issue where attachments with the same target url overwrite each other.
• Removed use of string representation of database objects in templates.
• routing for study view; to fix relative links.

5.1.1 - 2021-06-18

Fixed

• Restore falsely deleted template.

5.1.0 - 2021-06-17

Changed

• Updated concept view to use new API.

  • Split formerly merged table into variables and questions table

5.0.4 - 2021-06-09

Fixed

• Image import no longer fails if not all questions have images.
• Links to variables and datasets in the "Codes of related variables with the same concept" table are now generated correctly.

5.0.3 - 2021-05-26

Changed

• Upgrade several dependencies.

5.0.2 - 2021-02-03

Fixed

• broken links on question view, caused by recent refactoring.

5.0.1 - 2021-01-20

Changed

• Load outgoing mail address from environment variable.

Fixed

• Limited search fields for variable search to mitigate slow response times.

5.0.0 - 2021-01-18

Added

• new topic related API endpoints.

Changed

• Search UI now runs on latest reactivesearch-vue version.

• 🚀 Topics view frontend JavaScript was completely rewritten.

  • Loading time for related elements was improved drastically.
  • Rendering of related elements was migrated to the front end.

• Variable labels are now imported from variables.csv files, if no data from datasets JSON files was imported.

Removed

• 🚀 old API endpoints that were not implemented with django rest-framework.

4.7.0 - 2020-10-29

Added

• modal elements to the variable view to display larger metadata content in a better way.

Changed

• formatting for gunicorn logs changed for parsability.
• import of django JSONField to address deprecation issues.
• Migrate webpack setup to v5 and optimize building process.
• Refactor import to better work with data specifications.
• Refactor variable view to make content more understandable.
• Removed unique constraint on Topic name field. Unique identifier is now dependent on study name + topic name.

Fixed

• Handle questions without concepts in import.

4.6.3 - 2020-09-03

Changed

• Display of sub menu link Topics for studies now depends on the existence of topics linked to that study.

  • Was dependent on the attribute topic_languages being set before this change.

• Error reporting for attachment import is now more detailed.

Fixed

• Removed unused function call in topics.js that caused errors.
• Users without baskets no longer cause errors during createBasketList calls.

4.6.2 - 2020-08-31

Changed

• Migrate data visualization to d3 v6.

Fixed

• Concepts are no longer implicitly imported from question data.

• Errors during instrument import are no longer silently dropped.

• Handle missing data during variable indexation.

  • Values "none", "unspecified" and None (Python None) for conceptual dataset period and analysis unit are now unified to "Not Categorized".

4.6.1 - 2020-08-11

Fixed

• A part of the concepts import still used only old field names.

4.6.0 - 2020-08-07

Changed

• Import now works with data, as it was specified.
• Made several imports able to fail by replacing catch all try, except blocks that let errors drop basically unhandled.
• Made several imports atomic.

4.5.4 - 2020-07-20

Changed

• Bumped lodash dependency to 4.17.19 for important security update.

Fixed

• 500 HTTP error that occurred with newer soep-core long datasets.

4.5.3 - 2020-07-14

Fixed

• Import no longer fails, when no concepts.csv is present.
• Remove all link in Basket view now works again.
• Search in Study navigational link now works on all pages it occurs.

4.5.2 - 2020-07-03

Changed

• Downgrade reactivesearch-vue dependency to 1.2.0 since everything from 1.3.0 breaks UI components.

• Reimplement clean-update to address data integrity issues.

  • A clean update now creates a backup of user basket data. Then the study data is deleted, reimported and the backup restored. The restored backup can introduce "dead" BasketVariables which are cleaned up at the end.

Fixed

• Unittests that were not up to date.

4.5.1 - 2020-06-24

Changed

• During production docker image build JavaScript and CSS libraries are now moved after building them with webpack.

  • They are moved back through the entrypoint script.
  • The shared static files volume for web and nginx container otherwise masks files in production setup.

• Migrate visualization.js from d3 <= v3 to >= v5.

  • d3 v4 brought in some breaking changes that had to be addressed.

• Recoupled Baskets with BasketVariables and BasketVariables with Variables due to data integrity problems.

Fixed

• Corrected message displayed in base search UI, when no results are found.

• Instantiation of dataTable elements in UI.

• Question model called a non existent period attribute.

  • Period is now set for Questions during save.
  • Question Period == Instrument Period

4.5.0 - 2020-06-09

Added

• "Search in Study" navigation link from a Study view.

  • User is directed to the search interface with the filter for the study already enabled.

Changed

• Update ScriptGenerator in workspace

4.4.3 - 2020-05-26

Added

• Flag to import without redis

Fixed

• Abandoned Concepts (only found in variables.csv) are now handled explicitly.
• Concepts are now imported directly in the ConceptInput class not through an inherited function. The inherited function did no longer seem to work.

4.4.2 - 2020-05-04

Changed

• Dangling BasketVariables are now removed after a full import of all variables instead of after all import jobs.

  • To guarantee that no basket variables are removed because of failed variable imports.
  • To no longer block the study update command since it is now part of a redis job.

• Removal of Dangling BasketVariables is now only performed for the study, that is currently imported to prevent accidental removal of variables when trying to update several studies.

Fixed

• Authentication for the button to add variables to baskets now works in production.

• Added function to instantiate all dataTables on a page.

  • Display of dataTables had stopped working, probably through some dependency update.

4.4.1 - 2020-05-04

Fixed

• Clean import no longer deletes baskets

4.4.0 - 2020-04-23

Added

• update command flag -c --clean-import

  • Clean import deletes a study and all its associated data, except BasketVariables. The import then restores the study itself and continues importing from the studies metadata repository.

4.3.0 - 2020-04-21

Added

• Several API endpoints using Django REST Framework.

Changed

• Refactored javascript libraries.

• Replaced basket handling via old API endpoints with django REST framework calls.

• Replaced django-click dependency with django native command handling.

• Set limit (1000) for variables associated to a basket to improve server performance.

4.2.4 - 2020-02-27

Changed

• Updated django from 2.2.10 to 3.0.3

Fixed

• Added missing migration.
• crash occurring while trying to display faulty dataset data.

4.2.3 - 2020-02-24

Changed

• Badge with news on Homepage is now dynamically filled from the database.
• Limit LabelTable size to fix system overloading.
• Variable detail description is now filled from the database field description. Was formerly filled from description_long.

Removed

• fakeredis from dependencies.

4.2.2 - 2020-01-16

Changed

• Update "legacy" import to work with image import.
• Major refactoring of variable model.

Fixed

• Comply to new names used in ReactiveList of reactivesearch-vue.

• pytest-mock is no longer used as a context manager.

  • Newest release of pytest-mocker raises an error if used as context manager.

• Update workspace.management.commands.restore to work with updates in new (external) tablib version.

4.2.1 - 2019-11-20

Changed

• Image Import

  • Handle urls, where no image can be retrieved.
  • Do not load images larger than 200KB.

• Patch old import to work with questions_images.csv

Fixed

• Instrument import passed wrong object to image import.

4.2.0 - 2019-10-31

Changed

• BasketVariables are no longer deleted when their corresponding variables are deleted.

  • Function was implemented to clean up stale BasketVariables if the need arises.

• Copyright notice to include former contributors.

• dev: NGINX now set up for debugging long requests.

• dev: Setup now better supports commit signing.

• Landing page for successful password reset was changed.

  • It now extends the standard registration template.
  • This extension adds a badge with a success message.

• Switch to Python 3.8 from 3.7.

Fixed

• dev: Move pytest coverage flags from setup.cfg to .travis.yml.

• • Coverage Reports in Pytest versions newer than 5.0.1 did cause issues with the vscode debugger.

• dev: Pylint Pre-commit hook properly uses pylint_django plugin.

• sort_id for variable imports is now incremented for each variable.

  • Before it was only set to 1 for each variable.

4.1.2 - 2019-10-07

Fixed

• Version pin of postfix for docker postfix image is less strict.

  • apk was not able to install pinned version for the newest alpine version.

4.1.1 - 2019-10-01

Added

• Function to alter result text above search results (x results found in yms)

  • Now adds "More than " if result count is higher than the elasticsearch 7 scroll limit of 10000. Meaning it will display "More than 10000 results found in yms".

Changed

• Update several dependencies.

Fixed

• Installation of dependencies for the dev container and travis setup.

  • Setup only installed dev dependencies without the applications dependencies.

4.1.0 - 2019-08-28

Added

• A Warning is now displayed for Internet Explorer users.

  • IE is not supported and the warning suggests the use of a different Browser.

• dev: pre-push hook can be installed to spellcheck commit messages.

Changed

• dev: All pre-commit hooks are now local.
• Switched source for django-elasticsearch-dsl from GitHub to PyPi.
• Python packages are now installed via pip instead of pipenv install --system.
• Update README.md to reflect changes in the Docker setup.

Fixed

• dev: removed leftover volume definition from remote-dev docker-compose file.

4.0.2 - 2019-08-22

Added

• Favicon and webpack setup.

Changed

• Raised rqworker default Timeout.
• Refactored Vue router.
• Revised update section in home.html template.
• Update dependencies.

Fixed

• Creation of answer tables on question views works again.
• Facet checkbox does no longer shrink when facet text gets to long.
• Hyperrefs at the end of the CHANGELOG.md.
• Responsive mobile menu now collapses correctly.

4.0.1 - 2019-08-16

Fixed

• Adjust variable_detail.html django template to Bootstrap 4.

• Further separated dev and production setup to keep content of static/dist in production.

  • Dockerfile and dev.Dockerfile use different entrypoint.sh.
  • Dockerfile entrypoint no longer contains rebuilding section for npm packages.

4.0.0 - 2019-08-13

Added

• container health checks.
• CONTRIBUTING.md.
• Different image setups for development and live systems.
• doi field to Study.
• label_de field to ddionrails.instruments.Instrument.

Changed

• Database and search engine import were decoupled.

  • Search engine import is done via management command search_index --populate.
  • When calling the update management command, search_index --populate is called implicitly after imports are finished.

• Development setup is now focused on remote 'in container' development.

• Django management commands update and upgrade were merged.

  • Now update loads remote data and starts the import.

• Downgrade compose version to 2.4.

  • Version 3 is geared towards use with swarm setups and provides many features only for use in a docker swarm.

• Elasticsearch is now proxied through Nginx instead of Django.

• 🚀 most models now use UUIDs as Primary Keys. [major change]

  • UUIDs are either generated from a base UUID and the content of the fields that make a row unique or from the UUID of the "parent" model and the
    unique fields.

  • Numeric ids which were used until now are incompatible with the new UUID field. This breaking change was used as an opportunity to reset all migrations and start from a clean slate.

• 🚀 Replace ddionrails-elasticsearch with ReactiveSearch. [major change]

• Several dependency updates.

• Slimmed down live docker image.

• Switched base image of postfix to vanilla alpine.

• Update Bootstrap to Bootstrap 4 and subsequent layout adaptations.

• Update Django version to 2.2.4.

• Update Elasticsearch to 7.3.0.

• Update Docker settings related to security.

• Use Python 3.7.4 as base for Django image.

Fixed

• Gunicorn now correctly logs to docker.

Removed

• "Magic at work!".
• Bootstrap 3.
• DatasetRedirectView.
• ddionrails-elasticsearch.
• Elasticsearch 2.
• lowercasing from ModelMixin.
• old migrations. New squashed ones now use UUIDs.
• several code smells identified by codacy.
• StudyRedirectView.
• workspace/angular.html template.

3.3.0 - 2019-06-18

Added

• File server using django-filer.
• Setup for renovate-bot.
• pre-commit hooks for development.
• QuestionImage Model to store Images for Question Objects.
• Tests for paver tasks and api.view.

Changed

• Restructure static file handling.

  • Moved static assets from static/src/ to assets/ directory.
  • Removed static folder from source tracking.
  • Update webpack.config.js.

• Update third party dependencies.

• Renamed media files folder from media_root to media.

• Move context from ddionrails.studies.models into own module.

• Move import_path() method from Study and System Models to new ImportPathMixin.

• Direct use of "/tmp/" replaced with use of tempfile library.

• imports.manager. Repository now uses GitPython.

• imports.manager. SystemImportManager now runs without django_rq.

Removed

• Several unused templates and views.

• Unused or pre-installed packages from dockerfile package installation section.

• Security issues

  • use of preprocessing
  • use of os.system
  • use of eval

• Several outdated paver tasks.

• Forced storing of only lower cased names for Question and Variable names.

3.2.0 - 2019-05-29

Added

• Management commands to work with the elasticsearch index

Changed

• Reword Documentation and LICENSE.md file to let GitHub recognize the Project as AGPL-3.0.
• Combined staging and production settings.
• Improved admin interface.
• Improved admin model functionality.
• Update models of data app.
• Update models of instruments app.

3.1.0 - 2019-05-23

Added

• This Changelog.
• Many redundant commits due to unresolveable merge conflicts.
• Automatic update of stale static files kept in the static_data volume.
• Section to README to explain the currently hackey way the elastic search index is configured.

Changed

• Storage location for data to be imported can now be set via environment variable.

• Update dependencies.

Fixed

• Pavement command to create elasticsearch index.

Removed

• Unused codecov library from Pipfile.

3.0.1 - 2019-05-20

Added

• Install section for Production to README.

Changed

• Update Readme.
• Update Dependencies.
• Refactor dependency files.

Fixed

• Sending of coverage data to codecov.io

Removed

• Fabric from dependencies

3.0.0 - 2019-05-17

Added

• docker setup.
• docker-compose setup.
• docker-compose development setup.
• Example config and ENVIRONMENT files for the use with the docker setup.
• Some configuration can now be set via environment variables.
• Use of PostgreSQL database.
• Use of dedicated mail server service.
• Adherence to Semantic Versioning for this project.

Changed

• Data Model: Question labels type from CharField to TextField.
• Travis setup to work with docker.
• Tests to work with docker;
• Installation Section in Readme to explain docker setup.
• Elasticsearch version to 2.4.6.

Deprecated

• Installation of project without docker
• Use of paver to set up project is phased out in future versions

Removed

• Import: Data is no longer pulled via --depth 1 .

2.1.0 2018-12-05

Changed

• Moved Project into Open Source and onto GitHub.:rocket:
• Codestyle to work with flake8