Django Data Wizard 2.0

Django Data Wizard 2.0 has arrived! This release brings improved customizability, as well as a new React / Material UI interface to supplement the existing Django Admin integration.

Be sure to check out the new documentation site!

All changes by @sheppard.

New Features

• New React/Material UI interface, @wq/wizard, implemented as a plugin for integration with wq 1.3 (f9050fa, 5859f67, da54e2c, 27aee57). The existing Django Admin-themed templates are still supported and maintained (#36).
• Support for custom task flows with new AUTO_IMPORT_TASKS setting and auto_import_tasks serializer option (4621e12, 62c0776, 0067e1a, 10bd328, ece3842, ab9cb73, #36)
• Support S3 storage backends, by updating IterTable to support loading data from arbitrary file-like APIs (#31)
• Support for Python 3.10 and Django 4 (1caa9d8, 84b821b, #37)

Breaking Changes

• Dropped support for Python < 3.6, Django < 2.2, and Django REST Framework < 3.9 (22b36c0, 6f16312)
• Dropped support for wq framework < 1.3 (jQuery Mobile / Mustache templates) (5859f67)
• Changed default IDMAP setting from "data_wizard.idmap.never" to "data_wizard.idmap.existing" (6f16312)
• Always default the BACKEND setting to "data_wizard.backends.threading", even if Celery is installed. If you want to use the "data_wizard.backends.celery" backend, set it explicitly.

Other Minor Improvements

• Avoid making simultaneous requests to the status API (e722eae, d2235b5)
• Use REST_FRAMEWORK["DEFAULT_AUTHENTICATION_CLASSES"], but only when integrating with wq.db (54191ee)
• Update to openpyxl-based IterTable (5cb31c9, 96ac5d2)
• Ignore None values when merging cell data (0b6e627)
• Update lint & CI (9e247a6, 22b36c0)

Django Data Wizard 1.3.0

Django Data Wizard 1.3.0 brings support for Django 3.0, Django REST Framework 3.11, and wq 1.2, together with configuration enhancements to improve integration with the ecosystem.

Improvements

Foreign Key Mapping

Previously, all foreign keys and slugs needed to be mapped manually the first time they were encountered in a file, even if the IDs exactly matched known records in the database (#14, #25). The new DATA_WIZARD['IDMAP'] setting provides more flexibility over this behavior.

• "data_wizard.idmap.never": Require user to manually map all IDs the first time they are found in a file
• "data_wizard.idmap.existing": Automatically map existing IDs, but require user to map unknown ids
• "data_wizard.idmap.always": Always map IDs (skip manual mapping). Unknown IDs will be passed on as-is to the serializer, which will cause per-row errors (unless using natural keys).

For backwards compatibility with the 1.x series, "data_wizard.idmap.never" is the default setting. The default will change to "data_wizard.idmap.existing" in version 2.0.

Django Integration

• New DATA_WIZARD['AUTHENTICATION'] setting (see #24)
  • The default value is "rest_framework.authentication.SessionAuthentication", for smooth integration with the Django Admin.
  • This is important for cases where REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] is overridden for another purpose that does not include SessionAuthentication (see discussion in #24)
• Don't break if "data_wizard" is listed after "wq.db.rest" in INSTALLED_APPS (see #26)
• Don't break if "data_wizard" is listed before"django.contrib.admin" in INSTALLED_APPS (see #27)

wq Framework Integration

The included wq Framework Integration has been updated for compatibility with the Redux and NPM integration introduced in wq.app 1.2 (#28). In addition, the former wq.io dependency has been renamed to IterTable. See the IterTable release notes for more information on the name change.

Upgrade Notes

If you are using Django Data Wizard's built-in admin integration and loaders, then upgrading to 1.3.0 should not affect you. However:

• If you created a custom UI that accesses the Wizard's REST API with something other than SessionAuthentication, update your DATA_WIZARD["AUTHENTICATION"] setting. Django Data Wizard no longer uses the REST_FRAMEWORK["DEFAULT_AUTHENTICATION_CLASSES"] setting.
• If you were using a custom loader based on wq.io, rename the load_io() method to load_iter() and update your import paths as described in the IterTable release notes.
• If you are using the wq framework integration, you will need to install @wq/progress from NPM as it is no longer provided in the wq.app PyPI package.

Django Data Wizard 1.2.0

Django Data Wizard 1.2.0 brings several additional features and bug fixes.

New Features

• Ensure compatibility with Django REST Framework 3.9 (#22)
• Django Integration Improvements (#21/#23 via @Bomba1990)
  • New ImportActionModelAdmin and ImportActionMixin classes for use with custom source models
  • Support translation of "Import via data wizard" action text
  • Use reverse() when redirecting between Data Wizard views
  • Don't assume user is set in request
• If no serializers are registered, show a warning instead of a blank list (#16)
• Test ability to import number-formatted cells without float point (#20)
• Document and extend available serializer configuration options (e51edba, b9640f3)
• Add option to ignore spreadsheet columns during importing (1b7774f)

Bug Fixes

• Explicitly list djangorestframework in dependencies (#16)
• Code cleanup (#17, #18, #23 via @Bomba1990)
• Avoid re-parsing the same data source more than necessary (edd6ccb)
• Fix pagination when integrating with wq.db (c30e180)

Upgrade Notes

In 1.1.0, the "Import via data wizard" action was applied to all models in the admin, rather than only to those that made sense as sources. As noted above, 1.2.0 instead provides ImportActionModelAdmin and ImportActionMixin classes. If you are using custom source models, you just need to update your admin.py:

from django.contrib import admin
from .models import MySource
from data_wizard.admin import `ImportActionModelAdmin`

admin.site.register(MySource, ImportActionModelAdmin)

## To restore old "always available" behavior:
# from data_wizard.admin import start_data_wizard
# admin.site.add_action(start_data_wizard, 'data_wizard')

The built-in data_wizard.sources module already incorporates the new usage.

Django Data Wizard 1.1.0

Django Data Wizard 1.1.0 brings a number of exciting improvements while (mostly) maintaining backwards compatibility with 1.0.

New Features

• Full integration with the Django admin, including:
  • Admin-styled Django templates (#7), complementing the existing wq/Mustache templates.
  • "Import via data wizard" admin action
  • Optional ready-to-use data source models to streamline setup (FileSource and URLSource) (#8)
  • Admin view for identifier mappings (#6)
• Removed hard dependency on Celery/Redis
  • Async processing now uses a simple threading backend, but can still use celery if desired
  • This is configurable via the new DATA_WIZARD["BACKEND'] setting.
• Compatibility with Django 2.0/2.1 and Python 3.7 (#11)
• CLI for automated processing (#12)
• Fixed issues with content type id serialization (#9, #13)

Upgrade Notes

The Django Data Wizard 1.1 API is designed to be backwards compatible with 1.0, but there are two breaking configuration changes:

1. The provided data_wizard.urls no longer include the datawizard/ prefix, so you need to add it yourself.

 urlpatterns = [
     # ...
-    path('', include('data_wizard.urls')),
+    path('datawizard/', include('data_wizard.urls')),
 ]

2. The permission to create, view, and execute Run instances via the API is now restricted by default to rest_framework.permissions.IsAdminUser. Previously the permissions were whatever you had configured for REST_FRAMEWORK['DEFAULT_PERMISSION_CLASSES']. To override the new default, set DATA_WIZARD['PERMISSION'] to your desired permissions setting.

Django Data Wizard 1.0.1

Fixes a couple of issues with the README that were preventing installation and display on PyPI.

Django Data Wizard 1.0.0

Django Data Wizard 1.0.0 is finally here! This is the first stable release of Django Data Wizard 1.0, which is now ready for production use.

Changes since Django Data Wizard 1.0.0 RC1

• Fix generation of empty queryset (71bf665)
• Add a simple list view (6a36cbc)
• Incorporate Code of Conduct and Contributing Guidelines
• Documentation updates

Other changes since vera 0.6.2

• Pre-Beta Changes
  • Extract from wq.db.contrib.dbio into standalone package with dependencies on wq.db and vera (wq/wq.db#29)
  • See the release notes for wq.db 0.7 and vera 0.7
• Changes in Beta 1
  • Rename package module from dbio to data_wizard; publish as data-wizard on PyPI
  • Use package-specific models (Identifier, Range, and Record) instead of wq.db's identify and relate patterns (#4)
• Changes in Beta 2
  • Incorporate default wq-compatible templates for user interface
• Changes in RC 1
  • Create a registration API for importing arbitrary structured data via Django REST Framework model serializers; completely factor out vera dependency and minimize reliance on wq.db. (#1, #3, #5)

Django Data Wizard 1.0 RC

This is the release candidate for Django Data Wizard 1.0. The dependency on vera has been completely factored out, in favor of the ability to register Django REST Framework serializers to define the mapping between spreadsheet columns and database fields. This means that the wizard now works with any data model (#3) and also is much less likely to have database-level validation errors as long as the serializer correctly validates the data beforehand (#1). See the updated README for details on how to use these new features.

Django Data Wizard 1.0 Beta 2

This version bring updates for compatibility with wq.db 1.0.0b3 and vera 1.0.0b2, as well as (for the first time) a set of default mustache templates for each of the wizard steps to make it easier to get up and running.

Django Data Wizard 1.0 Beta

This is the beta 1.0 release of the Django Data Wizard. Django Data Wizard was previously released as wq.db.contrib.dbio, which was removed in wq.db 0.7.0.