IoConfig

IoConfig用于适配实时/离线的数据，比如kafka实时数据和文件批数据。通常需要与firhose组合配置

• IoConfig：
  • luceneIndexIOConfig
  • kafkaSupervisorIOConfig
• FireHose：
  • localFirehose
  • hdfsFirehose
  • parquetFirehose
  • jdbcFirehose

1. IoConfig

1.1 luceneIndexIOConfig

luceneIndexIOConfig一般用于适配离线数据

"ioConfig":{
      "type": "lucene_index",
     "firehose": FirehoseFactory firehoseFactory
}

1.2 kafkaSupervisorIOConfig

kafkaSupervisorIOConfig一般用于适配kafka实时数据

"ioConfig":{
     "topic": String topic,
      "replicas": Integer replicas,
      "taskCount": Integer taskCount,
      "taskDuration": Period taskDuration,
      "consumerProperties": Map<String, String> consumerProperties,
      "startDelay" Period startDelay,
      "period": Period period,
      "useEarliestOffset": Boolean useEarliestOffset,
      "completionTimeout": Period completionTimeout,
      "lateMessageRejectionPeriod": Period lateMessageRejectionPeriod,
      "earlyMessageRejectionPeriod": Period earlyMessageRejectionPeriod,
      "skipOffsetGaps": Boolean skipOffsetGaps
}

1.2.1 Period配置

Preiod 一般用于时间周期配置，通过P/PT与时间符号来设定具体的周期。支持以下配置

2. FireHose

FireHose用于配置不同数据源的连接器

2.1 localFirehose

localFirehose用于采集本地文件

"firehose":{
      "type": "local",
      "baseDir": String baseDir,
      "filter": String filter,
      "dirFilter": String dirFilter,
      "recursive": Boolean recursive
}

2.2 hdfsFirehose

hdfsFirehose用于采集Hdfs的文本文件

"firehose":{
      "type": "hdfs",
      "baseDir": String baseDir,
      "defaultFS": String defaultFS,
      "filter": String filter,
      "dirFilter": String dirFilter,
      "recursive": Boolean recursive,
      "readThreads": Integer readThreads,
      "dataQueueThreshold": Integer dataQueueThreshold

}

2.3 parquetFirehose

parquetFirehose用于采集Hdfs的parquet文件

"firehose":{
      "type": "parquet",
      "baseDir": String baseDir,
      "defaultFS": String defaultFS,
      "filter": String filter,
      "dirFilter": String dirFilter,
      "recursive": Boolean recursive,
      "readThreads": Integer readThreads,
      "dataQueueThreshold": Integer dataQueueThreshold

}

2.4 jdbcFirehose

jdbcFirehose用于采集Hdfs的parquet文件

 "firehose":{
       "type": "jdbc",
        "provider": JdbcProvider provider,
        "sqls": String[] sqls,
        "readThreads":  Integer readThreads,
        "dataQueueThreshold": Integer dataQueueThreshold
 }

2.4.1 JdbcProvider配置

提供Jdbc连接信息  

"provider": {
    "type": String type,
    "url": String url,
    "user": String user,
    "password": String password,
    "encrypt": Boolean encrypt,
    "driverClassName":String driverClassName,
    "driverPath":String driverPath
}