Replies: 6 comments 9 replies
-
The Spline developement and testing is done primarly on Apache Spark we try to fix issues with other flavours of Spark as well, but we don't have resources to do extensive tests on other platforms/flavours.
currenlty not, but we welcome any contribution even guides like that. |
Beta Was this translation helpful? Give feedback.
-
In short, yes, it does. Like Adam said, there could be some rough edges, and caveats, depending on your particular setup and job, but many people do use Spline on Glue successfully.
For example this - an older AWS article. |
Beta Was this translation helpful? Give feedback.
-
|
thanks- I have read the article and put Dependent JARs path parameters: spark.spline.mode=ENABLED
spark.spline.lineageDispatcher.http.producer.url=https://databricks.spline.octopai.com/producer
spark.spline.lineageDispatcher=http
spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListeneralso tried spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener
spark.spline.producer.url=https://databricks.spline.octopai.com/producerbut didnt got any lineage on the arangoDB the code of the job is simple- write from csv to other csv import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
# Read the CSV file from the source S3 location
source_path = "s3://johnny-chivers-glue-full-course-a/data/customers_source/Changes.csv"
df = spark.read.csv(source_path, header=True, inferSchema=True)
# Write the DataFrame to the target S3 location
target_path = "s3://johnny-chivers-glue-full-course-a/data/customers_source/ChangesTgt.csv"
df.write.csv(target_path, mode="overwrite", header=True)
# Commit the job
job.commit()what am i missing? how to trouble shoot it? |
Beta Was this translation helpful? Give feedback.
-
|
Is there anything in logs? |
Beta Was this translation helpful? Give feedback.
-
|
yes - the dirve logs 24/03/26 14:11:32 INFO ShutdownHookManager: Shutdown hook called |
Beta Was this translation helpful? Give feedback.
-
|
You have posted a link to a CloudWatch. I do not have access to your AWS account. |
Beta Was this translation helpful? Give feedback.
-
|
I mean this is the log |
Beta Was this translation helpful? Give feedback.
-
|
I do not see any logs related to Spline. Please find the relevant logs. They must be among driver logs. |
Beta Was this translation helpful? Give feedback.
-
|
programicly means put |
Beta Was this translation helpful? Give feedback.
-
|
yes |
Beta Was this translation helpful? Give feedback.
-
|
also not working |
Beta Was this translation helpful? Give feedback.
-
|
Hi seems to work when i put --extra-jars Dependent JARs path But has issues with spark 3.3 LAUNCH ERROR | Error downloading from S3 for bucket: johnny-chivers-glue-full-course-a, key: data/spark-3.3-spline-agent-bundle_2.12-2.0.0.jar;s3://johnny-chivers-glue-full-course-a/data/snakeyaml-1.33.jar.The specified key does not exist. (Service: Amazon S3; Status Code: 404; Please refer logs for details. |
Beta Was this translation helpful? Give feedback.
-
|
I believe you used an incorrect delimiter for your JAR URLs. But the issue doesn't seem to be related to Spline at this point. |
Beta Was this translation helpful? Give feedback.
-
|
i get an error |
Beta Was this translation helpful? Give feedback.
-
|
It might be caused by Glue 4.0 missing |
Beta Was this translation helpful? Give feedback.
-
Hi
does Spline support AWS Glues that runs jobs on Spark?
if so do you have any manual how to configure it?
Zacay
Beta Was this translation helpful? Give feedback.
All reactions