Skip to content

Latest commit

 

History

History
228 lines (152 loc) · 6.07 KB

IoTDB.md

File metadata and controls

228 lines (152 loc) · 6.07 KB
Raw

IoTDB

IoTDB source connector

Description

Read external data source data through IoTDB.

Key features

supports query SQL and can achieve projection effect.

Options

name type required default value
host string no -
port int no -
node_urls string no -
username string yes -
password string yes -
sql string yes -
schema config yes -
fetch_size int no -
lower_bound long no -
upper_bound long no -
num_partitions int no -
thrift_default_buffer_size int no -
enable_cache_leader boolean no -
version string no -
common-options no -

single node, you need to set host and port to connect to the remote data source.

host [string] the host of the IoTDB when you select host of the IoTDB

port [int] the port of the IoTDB when you select

multi node, you need to set node_urls to connect to the remote data source.

node_urls [string] the node_urls of the IoTDB when you select

e.g.

127.0.0.1:8080,127.0.0.2:8080

other parameters

sql [string] execute sql statement e.g.

select name,age from test

schema [config]

fields [Config]

The schema of the IoTDB that you want to generate

e.g.

schema {
    fields {
        name = string
        age = int
    }
  }

option parameters

fetch_size [int]

the fetch_size of the IoTDB when you select

username [string]

the username of the IoTDB when you select

password [string]

the password of the IoTDB when you select

lower_bound [long]

the lower_bound of the IoTDB when you select

upper_bound [long]

the upper_bound of the IoTDB when you select

num_partitions [int]

the num_partitions of the IoTDB when you select

thrift_default_buffer_size [int]

the thrift_default_buffer_size of the IoTDB when you select

enable_cache_leader [boolean]

enable_cache_leader of the IoTDB when you select

version [string]

Version represents the SQL semantic version used by the client, which is used to be compatible with the SQL semantics of 0.12 when upgrading 0.13. The possible values are: V_0_12, V_0_13.

split partitions

we can split the partitions of the IoTDB and we used time column split

num_partitions [int]

split num

upper_bound [long]

upper bound of the time column

lower_bound [long]

lower bound of the time column

     split the time range into numPartitions parts
     if numPartitions is 1, use the whole time range
     if numPartitions < (upper_bound - lower_bound), use (upper_bound - lower_bound) partitions
     
     eg: lower_bound = 1, upper_bound = 10, numPartitions = 2
     sql = "select * from test where age > 0 and age < 10"
     
     split result

     split 1: select * from test  where (time >= 1 and time < 6)  and (  age > 0 and age < 10 )
     
     split 2: select * from test  where (time >= 6 and time < 11) and (  age > 0 and age < 10 )

common options

Source plugin common parameters, please refer to Source Common Options for details

Examples

Case1

Common options:

source {
  IoTDB {
    node_urls = "localhost:6667"
    username = "root"
    password = "root"
  }
}

When you assign sql、fields、partition, for example:

sink {
  IoTDB {
    ...
    sql = "SELECT temperature, moisture FROM root.test_group.* WHERE time < 4102329600000 align by device"
    lower_bound = 1
    upper_bound = 4102329600000
    num_partitions = 10
    fields {
      ts = bigint
      device_name = string

      temperature = float
      moisture = bigint
    }
  }
}

Upstream IoTDB data format is the following:

IoTDB> SELECT temperature, moisture FROM root.test_group.* WHERE time < 4102329600000 align by device;
+------------------------+------------------------+--------------+-----------+
|                    Time|                  Device|   temperature|   moisture|
+------------------------+------------------------+--------------+-----------+
|2022-09-25T00:00:00.001Z|root.test_group.device_a|          36.1|        100|
|2022-09-25T00:00:00.001Z|root.test_group.device_b|          36.2|        101|
|2022-09-25T00:00:00.001Z|root.test_group.device_c|          36.3|        102|
+------------------------+------------------------+--------------+-----------+

Loaded to SeaTunnelRow data format is the following:

ts device_name temperature moisture
1664035200001 root.test_group.device_a 36.1 100
1664035200001 root.test_group.device_b 36.2 101
1664035200001 root.test_group.device_c 36.3 102

Changelog

2.2.0-beta 2022-09-26

  • Add IoTDB Source Connector

2.3.0-beta 2022-10-20

  • [Improve] Improve IoTDB Source Connector (2917)
    • Support extract timestamp、device、measurement from SeaTunnelRow
    • Support TINYINT、SMALLINT
    • Support flush cache to database before prepareCommit