The batch-processing
quickstart shows how to use chunk oriented batch jobs to import a file to a database.
This quickstart simulates a file importation using batch jobs. To make it easy, this quickstart offers the user a way to generate files. The generated file can have its name and the number of records customized. The user can also specify if the file contains an error or not.
The job contains two tasks:
-
It imports the file using a chunk oriented approach.
-
The chunk size is set to
3
. -
The
RecordsReader
is responsible for parsing the file and create an instance ofContact
. -
The
ContactsFormatter
applies the proper case to the contact name and it also applies a mask to the phone number. -
Finally,
ContactsPersister
sends the contact instance to the database.
-
-
It logs the number of records imported.
The database schema defines that the column for name is unique. For that reason, any attempt to persist a duplicate value will throw an exception. On the second attempt to run the job, the ChunkCheckpoint
provides information to skip the contacts that were already persisted.
- H2 Database
-
This quickstart uses the H2 database included with WildFly Application Server 35. It is a lightweight, relational example datasource that is used for examples only. It is not robust or scalable, is not supported, and should NOT be used in a production environment.
The application this project produces is designed to be run on WildFly Application Server 35 or later.
All you need to build this project is Java SE 17.0 or later, and Maven 3.6.0 or later. See Configure Maven to Build and Deploy the Quickstarts to make sure you are configured correctly for testing the quickstarts.
In the following instructions, replace WILDFLY_HOME
with the actual path to your WildFly installation. The installation path is described in detail here: Use of WILDFLY_HOME and JBOSS_HOME Variables.
When you see the replaceable variable QUICKSTART_HOME, replace it with the path to the root directory of all of the quickstarts.
-
Open a terminal and navigate to the root of the WildFly directory.
-
Start the WildFly server with the default profile by typing the following command.
$ WILDFLY_HOME/bin/standalone.sh
NoteFor Windows, use the WILDFLY_HOME\bin\standalone.bat
script.
-
Make sure WildFly server is started.
-
Open a terminal and navigate to the root directory of this quickstart.
-
Type the following command to build the quickstart.
$ mvn clean package
-
Type the following command to deploy the quickstart.
$ mvn wildfly:deploy
This deploys the batch-processing/target/batch-processing.war
to the running instance of the server.
You should see a message in the server log indicating that the archive deployed successfully.
Access the running application in a browser at the following URL: http://localhost:8080/batch-processing/
You are presented with a simple form that allows you to generate sample files to be imported.
-
Click on Generate a new file and start import job button. This generates a new file with 10 unique records to be imported. The import job starts after the file is generated.
-
A table is displayed containing information about the task that was started. Click on the Update jobs list button and verify that the job was completed.
-
Investigate the console output. It shows that files with 10 records were processed, 3 records at a time.
INFO [org.jboss.as.quickstarts.batch.controller.BatchController] (default task-3) Starting to generate 10 in file /var/folders/j8/63sgdmbn5tqdkyw0tz6df53r0000gn/T/temp-file.txt INFO [org.jboss.as.quickstarts.batch.controller.BatchController] (default task-3) File generated at /var/folders/j8/63sgdmbn5tqdkyw0tz6df53r0000gn/T/temp-file.txt INFO [org.jboss.as.quickstarts.batch.job.listener.JobListener] (Batch Thread - 1) Job import-file - Execution #1 starting. INFO [org.jboss.as.quickstarts.batch.job.ContactsPersister] (Batch Thread - 1) No checkpoint detected. Cleaning the Database INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #1 - Changing name ZIqYKITxiM -> Ziqykitxim | phone 978913851 -> (978)-913-851 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #2 - Changing name JbHjnaThps -> Jbhjnathps | phone 095108018 -> (095)-108-018 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #3 - Changing name FJTlXRtCdR -> Fjtlxrtcdr | phone 286847939 -> (286)-847-939 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Preparing to persist 3 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Persisting 3 contacts INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #4 - Changing name mlmBABWzfL -> Mlmbabwzfl | phone 744478648 -> (744)-478-648 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #5 - Changing name jVlTYiBRMP -> Jvltyibrmp | phone 135063841 -> (135)-063-841 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #6 - Changing name DwEFbSjfQE -> Dwefbsjfqe | phone 404572175 -> (404)-572-175 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Preparing to persist 3 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Persisting 3 contacts INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #7 - Changing name niDXWwGJuQ -> Nidxwwgjuq | phone 949448390 -> (949)-448-390 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #8 - Changing name VZBArfowSe -> Vzbarfowse | phone 902370961 -> (902)-370-961 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #9 - Changing name aSpyWCWwje -> Aspywcwwje | phone 246977695 -> (246)-977-695 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Preparing to persist 3 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Persisting 3 contacts INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 1) Register #10 - Changing name TofTfbRBzI -> Toftfbrbzi | phone 868339088 -> (868)-339-088 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Preparing to persist 1 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 1) Persisting 1 contacts INFO [org.jboss.as.quickstarts.batch.job.ReportBatchelet] (Batch Thread - 1) Imported 10 to Database INFO [org.jboss.as.quickstarts.batch.job.listener.JobListener] (Batch Thread - 1) Job import-file - Execution #1 finished. Status: COMPLETED
Now you will simulate a file with duplicate records. This will raise an exception and stop the processing. After that, you will fix the file and continue the importing where it stopped.
-
Mark the Generate a duplicate record checkbox and click on Generate a new file and start import job button. If you click on Update jobs list button, you will see that the job failed with the following Exit Status:
Error : org.hibernate.exception.ConstraintViolationException: could not execute statement
. This was caused because the job tried to insert a duplicate record at the Database. You will also seeorg.h2.jdbc.JdbcSQLException: Unique index or primary key violation
exception stacktraces in the server log. -
Next, you will fix the file and restart that job execution. Uncheck the Generate a duplicate record checkbox and click on Generate a new file button. This will generate the file without errors.
-
Click on Restart button in the last column for that job instance in the List of Jobs table. If you click on Update jobs list button, you will see that the job was completed.
-
Review the server logs and verify that the job started from the last checkpoint.
INFO [org.jboss.as.quickstarts.batch.job.RecordsReader] (Batch Thread - 3) Skipping to line 3 as marked by previous checkpoint
-
Investigate the console output.
INFO [org.jboss.as.quickstarts.batch.job.listener.JobListener] (Batch Thread - 3) Job import-file - Execution #3 starting. INFO [org.jboss.as.quickstarts.batch.job.RecordsReader] (Batch Thread - 3) Skipping to line 3 as marked by previous checkpoint INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #4 - Changing name HdeqwzEjbA -> Hdeqwzejba | phone 686417040 -> (686)-417-040 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #5 - Changing name veEEbtpYTJ -> Veeebtpytj | phone 367981821 -> (367)-981-821 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #6 - Changing name bQIKTUyqMW -> Bqiktuyqmw | phone 103363182 -> (103)-363-182 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Preparing to persist 3 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Persisting 3 contacts INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #7 - Changing name KVLIGXhCry -> Kvligxhcry | phone 117327691 -> (117)-327-691 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #8 - Changing name PBAZgernHy -> Pbazgernhy | phone 066203468 -> (066)-203-468 INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #9 - Changing name DGtNZdteGB -> Dgtnzdtegb | phone 908779587 -> (908)-779-587 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Preparing to persist 3 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Persisting 3 contacts INFO [org.jboss.as.quickstarts.batch.job.ContactsFormatter] (Batch Thread - 3) Register #10 - Changing name mhmIHhZMhv -> Mhmihhzmhv | phone 094518410 -> (094)-518-410 INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Preparing to persist 1 contacts INFO [org.jboss.as.quickstarts.batch.job.listener.PersistListener] (Batch Thread - 3) Persisting 1 contacts WARN [org.jberet] (Batch Thread - 3) JBERET000018: Could not find the original step execution to restart. Current step execution id: 0, step name: reportBatchelet INFO [org.jboss.as.quickstarts.batch.job.ReportBatchelet] (Batch Thread - 3) Imported 10 to Database INFO [org.jboss.as.quickstarts.batch.job.listener.JobListener] (Batch Thread - 3) Job import-file - Execution #3 finished. Status: COMPLETED
-
Check the Generate a duplicate record checkbox and click on Generate a new file ans start import job button. If you click on Update jobs list button, you will see that the job failed with the following Exit Status:
Error : org.hibernate.exception.ConstraintViolationException: could not execute statement
. This was caused because you tried to insert a duplicate record at the Database. -
This time you will not fix the file. Just click on Restart button again. If you click on Update jobs list button, you will see that the job was marked as
ABANDONED
this time because it was restarted once. Notice that there is a new parameter:restartedOnce=true
. This behavior was implemented inJobListener
for demonstration purposes, to prevent aFAILED
job from being restarted twice.
This quickstart includes integration tests, which are located under the src/test/
directory. The integration tests verify that the quickstart runs correctly when deployed on the server.
Follow these steps to run the integration tests.
-
Make sure WildFly server is started.
-
Make sure the quickstart is deployed.
-
Type the following command to run the
verify
goal with theintegration-testing
profile activated.$ mvn verify -Pintegration-testing
Instead of using a standard WildFly server distribution, you can alternatively provision a WildFly server to deploy and run the quickstart. The functionality is provided by the WildFly Maven Plugin, and you may find its configuration in the quickstart pom.xml
:
<profile>
<id>provisioned-server</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<discover-provisioning-info>
<version>${version.server}</version>
</discover-provisioning-info>
<add-ons>...</add-ons>
</configuration>
<executions>
<execution>
<goals>
<goal>package</goal>
</goals>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
</profile>
When built, the provisioned WildFly server can be found in the target/server
directory, and its usage is similar to a standard server distribution, with the simplification that there is never the need to specify the server configuration to be started.
Follow these steps to run the quickstart using the provisioned server.
-
Make sure the server is provisioned.
$ mvn clean package
-
Start the WildFly provisioned server, using the WildFly Maven Plugin
start
goal.$ mvn wildfly:start
-
Type the following command to run the integration tests.
$ mvn verify -Pintegration-testing
-
Shut down the WildFly provisioned server.
$ mvn wildfly:shutdown
On OpenShift, the S2I build with Apache Maven uses an openshift
Maven profile to provision a WildFly server, deploy and run the quickstart in OpenShift environment.
The server provisioning functionality is provided by the WildFly Maven Plugin, and you may find its configuration in the quickstart pom.xml
:
<profile>
<id>openshift</id>
<build>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<discover-provisioning-info>
<version>${version.server}</version>
<context>cloud</context>
</discover-provisioning-info>
<add-ons>...</add-ons>
</configuration>
<executions>
<execution>
<goals>
<goal>package</goal>
</goals>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
</profile>
You may note that unlike the provisioned-server
profile it uses the cloud context which enables a configuration tuned for OpenShift environment.
The plugin uses WildFly Glow to discover the feature packs and layers required to run the application, and provisions a server containing those layers.
If you get an error or the server is missing some functionality which cannot be auto-discovered, you can download the WildFly Glow CLI and run the following command to see more information about what add-ons are available:
wildfly-glow show-add-ons
This section contains the basic instructions to build and deploy this quickstart to WildFly for OpenShift or WildFly for OpenShift Online using Helm Charts.
-
You must be logged in OpenShift and have an
oc
client to connect to OpenShift -
Helm must be installed to deploy the backend on OpenShift.
Once you have installed Helm, you need to add the repository that provides Helm Charts for WildFly.
$ helm repo add wildfly https://docs.wildfly.org/wildfly-charts/
"wildfly" has been added to your repositories
$ helm search repo wildfly
NAME CHART VERSION APP VERSION DESCRIPTION
wildfly/wildfly ... ... Build and Deploy WildFly applications on OpenShift
wildfly/wildfly-common ... ... A library chart for WildFly-based applications
Log in to your OpenShift instance using the oc login
command.
The backend will be built and deployed on OpenShift with a Helm Chart for WildFly.
Navigate to the root directory of this quickstart and run the following command:
$ helm install batch-processing -f charts/helm.yaml wildfly/wildfly --wait --timeout=10m0s
NAME: batch-processing
...
STATUS: deployed
REVISION: 1
This command will return once the application has successfully deployed. In case of a timeout, you can check the status of the application with the following command in another terminal:
oc get deployment batch-processing
The Helm Chart for this quickstart contains all the information to build an image from the source code using S2I on Java 17:
build:
uri: https://github.com/wildfly/quickstart.git
ref: main
contextDir: batch-processing
deploy:
replicas: 1
This will create a new deployment on OpenShift and deploy the application.
If you want to see all the configuration elements to customize your deployment you can use the following command:
$ helm show readme wildfly/wildfly
Get the URL of the route to the deployment.
$ oc get route batch-processing -o jsonpath="{.spec.host}"
Access the application in your web browser using the displayed URL.
The integration tests included with this quickstart, which verify that the quickstart runs correctly, may also be run with the quickstart running on OpenShift.
Note
|
The integration tests expect a deployed application, so make sure you have deployed the quickstart on OpenShift before you begin. |
Run the integration tests using the following command to run the verify
goal with the integration-testing
profile activated and the proper URL:
$ mvn verify -Pintegration-testing -Dserver.host=https://$(oc get route batch-processing --template='{{ .spec.host }}')
Note
|
The tests are using SSL to connect to the quickstart running on OpenShift. So you need the certificates to be trusted by the machine the tests are run from. |
For Kubernetes, the build with Apache Maven uses an openshift
Maven profile to provision a WildFly server, suitable for running on Kubernetes.
The server provisioning functionality is provided by the WildFly Maven Plugin, and you may find its configuration in the quickstart pom.xml
:
<profile>
<id>openshift</id>
<build>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<discover-provisioning-info>
<version>${version.server}</version>
<context>cloud</context>
</discover-provisioning-info>
<add-ons>...</add-ons>
</configuration>
<executions>
<execution>
<goals>
<goal>package</goal>
</goals>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
</profile>
You may note that unlike the provisioned-server
profile it uses the cloud context which enables a configuration tuned for Kubernetes environment.
The plugin uses WildFly Glow to discover the feature packs and layers required to run the application, and provisions a server containing those layers.
If you get an error or the server is missing some functionality which cannot be auto-discovered, you can download the WildFly Glow CLI and run the following command to see more information about what add-ons are available:
wildfly-glow show-add-ons
This section contains the basic instructions to build and deploy this quickstart to Kubernetes using Helm Charts.
In this example we are using Minikube as our Kubernetes provider. See the Minikube Getting Started guide for how to install it. After installing it, we start it with 4GB of memory.
minikube start --memory='4gb'
The above command should work if you have Docker installed on your machine. If, you are using Podman instead of Docker, you will also need to pass in --driver=podman
, as covered in the Minikube documentation.
Once Minikube has started, we need to enable its registry since that is where we will push the image needed to deploy the quickstart, and where we will tell the Helm charts to download it from.
minikube addons enable registry
In order to be able to push images to the registry we need to make it accessible from outside Kubernetes. How we do this depends on your operating system. All the below examples will expose it at localhost:5000
# On Mac:
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"
# On Linux:
kubectl port-forward --namespace kube-system service/registry 5000:80 &
# On Windows:
kubectl port-forward --namespace kube-system service/registry 5000:80
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:host.docker.internal:5000"
-
Helm must be installed to deploy the backend on Kubernetes.
Once you have installed Helm, you need to add the repository that provides Helm Charts for WildFly.
$ helm repo add wildfly https://docs.wildfly.org/wildfly-charts/
"wildfly" has been added to your repositories
$ helm search repo wildfly
NAME CHART VERSION APP VERSION DESCRIPTION
wildfly/wildfly ... ... Build and Deploy WildFly applications on OpenShift
wildfly/wildfly-common ... ... A library chart for WildFly-based applications
The backend will be built and deployed on Kubernetes with a Helm Chart for WildFly.
Navigate to the root directory of this quickstart and run the following commands:
mvn -Popenshift package wildfly:image
This will use the openshift
Maven profile we saw earlier to build the application, and create a Docker image containing the WildFly server with the application deployed. The name of the image will be batch-processing
.
Next we need to tag the image and make it available to Kubernetes. You can push it to a registry like quay.io
. In this case we tag as localhost:5000/batch-processing:latest
and push it to the internal registry in our Kubernetes instance:
# Tag the image
docker tag batch-processing localhost:5000/batch-processing:latest
# Push the image to the registry
docker push localhost:5000/batch-processing:latest
In the below call to helm install
which deploys our application to Kubernetes, we are passing in some extra arguments to tweak the Helm build:
-
--set build.enabled=false
- This turns off the s2i build for the Helm chart since Kubernetes, unlike OpenShift, does not have s2i. Instead, we are providing the image to use. -
--set deploy.route.enabled=false
- This disables route creation normally performed by the Helm chart. On Kubernetes we will use port-forwards instead to access our application, since routes are an OpenShift specific concept and thus not available on Kubernetes. -
--set image.name="localhost:5000/batch-processing"
- This tells the Helm chart to use the image we built, tagged and pushed to Kubernetes' internal registry above.
$ helm install batch-processing -f charts/helm.yaml wildfly/wildfly --wait --timeout=10m0s --set build.enabled=false --set deploy.route.enabled=false --set image.name="localhost:5000/batch-processing"
NAME: batch-processing
...
STATUS: deployed
REVISION: 1
This command will return once the application has successfully deployed. In case of a timeout, you can check the status of the application with the following command in another terminal:
kubectl get deployment batch-processing
The Helm Chart for this quickstart contains all the information to build an image from the source code using S2I on Java 17:
build:
uri: https://github.com/wildfly/quickstart.git
ref: main
contextDir: batch-processing
deploy:
replicas: 1
This will create a new deployment on Kubernetes and deploy the application.
If you want to see all the configuration elements to customize your deployment you can use the following command:
$ helm show readme wildfly/wildfly
To be able to connect to our application running in Kubernetes from outside, we need to set up a port-forward to the batch-processing
service created for us by the Helm chart.
This service will run on port 8080
, and we set up the port forward to also run on port 8080
:
kubectl port-forward service/batch-processing 8080:8080
The server can now be accessed via http://localhost:8080
from outside Kubernetes. Note that the command to create the port-forward will not return, so it is easiest to run this in a separate terminal.
The integration tests included with this quickstart, which verify that the quickstart runs correctly, may also be run with the quickstart running on Kubernetes.
Note
|
The integration tests expect a deployed application, so make sure you have deployed the quickstart on Kubernetes before you begin. |
Run the integration tests using the following command to run the verify
goal with the integration-testing
profile activated and the proper URL:
$ mvn verify -Pintegration-testing -Dserver.host=http://localhost:8080