iobeam supports Apache Spark apps for data analysis. Writing and deploying iobeam Spark apps allows users to process their data in a variety of ways, from simple rule-based models to more complex prediction, anomaly detection, classification, and other ML-driven models. The output of an iobeam Spark app can be a new stream of derived data, or an event to trigger an action using our Trigger service.
This repo describes how to write an iobeam Spark app. Some developers might be using iobeam for their entire data pipeline, while others' first introduction to iobeam will be for "device ops" and telemetry.
The first step to develop a custom iobeam Spark app is to set up up a Scala app with the right Spark and iobeam dependencies. To automate this, iobeam provides a Maven archetype that gives you framework code and dependencies such as iobeam libraries and a unit test framework.
- A recent Maven installation (3.3+)
- Scala 2.10
- JDK 8
To set up the iobeam Spark app, you run the following command from your command-line:
mvn archetype:generate \
-DarchetypeArtifactId=iobeam-spark-scala-maven-archetype \
-DarchetypeGroupId=com.iobeam \
-DarchetypeCatalog=https://assets.iobeam.com/libs/archetype-catalog.xml
Maven will ask for information unique to your app. At the minimum, you should set groupId,
artifactId.
Define value for property 'groupId': : com.mycompany
Define value for property 'artifactId': myApp
Define value for property 'version': 1.0-SNAPSHOT: : [Enter for default]
Define value for property 'package': com.mycompany: : [Enter for default]
Confirm properties configuration:
groupId: com.mycompany
artifactId: myApp
version: 1.0-SNAPSHOT
package: com.mycompany
Y: : [Enter]
When those properties are set, maven will create an app directory in the current directory. In this case, a directory named ./myApp/.
To test that the app works, try to build the example app.
cd myApp/
mvn package
If the build is successful, the end of the maven output will look similar to
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25.664 s
[INFO] Finished at: 2015-12-22T14:00:52+01:00
[INFO] Final Memory: 42M/1211M
[INFO] ------------------------------------------------------------------------
The mvn package command will build a JAR file that can be uploaded to iobeam. In this example, the JAR file is
target/myApp-1.0-SNAPSHOT.jar
Now, it's time to modify and/or deploy your app on iobeam.
- Device Ops. If you are interested in iobeam's native support for device ops, follow these device ops instructions
- App Development. For more general instructions on writing apps on iobeam, follow these app development instructions
Questions? Please reach out to us at support@iobeam.com.