categories.md

Auditing Properties

The following job properties are essential for auditing function:

• ms.audit.enabled
• ms.kafka.brokers
• ms.kafka.clientId
• ms.kafka.schema.registry.url
• ms.kafka.audit.topic.name

Monitoring Properties

The following job properties are essential for monitoring:

• ms.metrics.enabled
• ms.kafka

Authentication Properties

The following are related to authentication:

• ms.authentication
• ms.http.request.headers
• ms.secondary.input
• ms.parameters
• source.conn.username
• source.conn.password

Connection Properties

The following are related to connections:

• ms.connection.client.factory
• ms.source.uri
• ms.source.s3.parameters
• ms.sftp.conn.timeout.millis
• ms.ssl
• source.conn.username
• source.conn.password

The following could also be related to connections as they define variables:

• ms.secondary.input
• ms.parameters

Conversion Properties

The following are related to conversion (converters):

• ms.data.explicit.eof
• ms.derived.fields
• ms.enable.schema.based.filtering
• ms.encryption.fields
• ms.extract.preprocessors
• ms.extract.preprocessor.parameters
• ms.extractor.class
• ms.extractor.target.file.name
• ms.extractor.target.file.permission
• ms.normalizer.batch.size
• ms.output.schema
• ms.source.schema.urn
• ms.target.schema
• ms.target.schema.urn
• ms.validation.attributes

Extractor Properties

• ms.csv

Execution Properties

• ms.enable.dynamic.full.load

HTTP Properties

The following are related to HTTP sources:

• ms.authentication
• ms.http.request.headers
• ms.http.request.method
• ms.http.response.type
• ms.http.statuses
• ms.http.conn.max
• ms.http.conn.per.route.max
• ms.http.conn.ttl.seconds

Pagination Properties

• ms.call.interval.millis
• ms.pagination
• ms.session.key.field
• ms.wait.timeout.seconds

Schema Properties

The following are related to schema:

• ms.data.default.type
• ms.output.schema
• ms.source.schema.urn
• ms.target.schema
• ms.target.schema.urn
• ms.schema.cleansing
• ms.enable.cleansing
• ms.enable.schema.based.filtering
• ms.jdbc.schema.refactor
• ms.kafka.schema.registry.url
• ms.converter.keep.null.strings

Source Properties

• ms.data.field
• ms.jdbc.statement
• ms.parameters
• ms.s3.list.max.keys
• ms.session.key.field
• ms.source.data.character.set
• ms.source.files.pattern
• ms.source.s3.parameters
• ms.source.schema.urn
• ms.source.uri
• ms.total.count.field
• ms.wait.timeout.seconds

Watermark Work Unit Properties

The following are related to watermarks and work units:

• ms.abstinent.period.days
• ms.grace.period.days
• ms.secondary.input
• ms.watermark
• ms.work.unit.min.records
• ms.work.unit.min.units
• ms.work.unit.pacing.seconds
• ms.work.unit.parallelism.max
• ms.work.unit.partial.partition
• ms.work.unit.partition

Gobblin Properties

The following properties are inherited from Gobblin and enhanced with explicitly validation rules.

• converter.avro.date.format
• converter.avro.time.format
• converter.avro.timestamp.format
• extract.table.name
• job.commit.policy
• source.class
• converter.class