Parser

parser需要与parseSpec共同组合配置完成接入数据不同格式的转换功能。不同的数据需要进行不同parser/parseSpec组合

• Parser：

  • stringParser
  • standardParser
  • parquetParser

• ParseSpec：

  • json
  • csv
  • tsv
  • jsonLowercase
  • timeAndDims
  • regex
  • javascript
  • parquet
  • nested

• 维度说明

  • timestampSpec
  • dimensionsSpec

1. Parser

1.1 stringParser

"parser":{
      "type": "string",
      "parseSpec": ParseSpec parseSpec,
      "encoding": String encoding
}

1.2 standardParserSpec

"parser":{
      "type": "standard",
      "parseSpec": ParseSpec parseSpec,
      "encoding": String encoding,
      "excludeHead": Boolean excludeHead
}

1.3 parquetParser

"parser":{
      "type": "parquet",
      "parseSpec": ParseSpec parseSpec,
      "binaryAsString": Boolean excludeHead
}

2. ParseSpec

parseSpec用于配置说明接入数据的维度名称/类型/格式等信息

2.1 json

"parseSpec":{
  "format":"json",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
  "flattenSpec":{
    "useFieldDiscovery": Boolean useFieldDiscovery,
    "fields": List<JSONPathFieldSpec> fields
  }
  "featureSpec":Map<String, Boolean> featureSpec
}

example

"parseSpec":{
    "format":"json",
    "dimensionsSpec":{
        "dynamicDimension":true,
        "dimensions":[]
    },
    "timestampSpec":{
        "column":"d|place_time",
        "format":"millis"
    }
}

这个例子用动态维来接入json类型数据源，其中时间戳的维度名为 d|place_time，格式为 millis。

2.1.1 flattenSpec

{
  "useFieldDiscovery": boolean,
   "fields": {
        "type": "PATH|ROOT",
        "name": string,
        "expr": string
    }

}

example

json数据为：
{
    "fromhost-ip":"192.168.0.1",
    "hostname":"test1.sugo.io",
    "msg":{
        "EventTime":"2019-01-01 12:30:00",
        "Hostname":"ubuntu"
    }
}
====================
对应的配置：
"flattenSpec": {
    "useFieldDiscovery": false,
    "fields": [{
            "type": "ROOT",
            "name": "fromhost-ip"
        },{
            "type": "PATH",
            "name": "hostname",
            "expr": "$.hostname"
        },{
            "type": "PATH",
            "name": "EventTime",
            "expr": "$.msg.EventTime"
        },{
            "type": "PATH",
            "name": "Hostname1",
            "expr": "$.msg.Hostname"
        }
    ]
}

2.2 csv

"parseSpec":{
  "format":"csv",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
  "listDelimiter"： String listDelimiter,
  "multiValueDelimiter": String multiValueDelimiter,
  "columns": List<String> columns
}

example

"parser": {
  "parseSpec": {
    "format": "csv",
    "timestampSpec":{
      "column": "ts",
      "format": "yy-MM-dd HH:mm:ss.SSS"
    },
    "dimensionsSpec": {
      "dimensions": [
        {
          "name": "cardNum",
          "type": "string"
        },
        {
          "name": "level",
          "type": "int"
        },
        {
          "name": "create_time",
          "type": "date", 
          "format":"yy-MM-dd HH:mm:ss.SSS"
        }
      ]
    },
    "listDelimiter": ",",
    "columns": ["ts","cardNum","level","create_time"]
  }
}

这个例子接入csv类型数据，这个csv文件有四个维度分别和columns对应。

2.3 tsv

"parser": {
  "parseSpec":{
    "format":"tsv",
    "timestampSpec":TimestampSpec timestampSpec,
    "dimensionsSpec":DimensionsSpec dimensionsSpec,
    "delimiter": String delimiter,
    "listDelimiter"： String listDelimiter,
    "columns": List<String> columns,
    "nullFormat": String nullformat
  }
}

example

"parseSpec": {
  "format": "tsv",
  "timestampSpec": {"column": "myDate","format": "millis"},
  "dimensionsSpec": {
      "dimensions": [
          {"name": "myStr1","type": "string"},
          {"name": "myStr2","type": "string","hasMultipleValues":true }                                  
      ]
  },
  "delimiter":"abc",
  "listDelimiter": "!",
  "columns": ["myDate","myStr1","myStr2"]
}

注意这个例子的行数据分隔符为多个字符"abc"，不是常见的单字符分隔符如","
该例子对应的部分数据为:

1484839807587abcqq4hwfhh03abccvv904doqj!632124152
1490986198551abchaig2sabczufi!9733

对应的结果为：

2.4 jsonLowercase

"parseSpec":{
  "format":"jsonLowercase",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
}

2.5 timeAndDims

"parseSpec":{
  "format":"timeAndDims",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
}

2.6 regex

"parseSpec":{
  "format":"regex",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
  "listDelimiter"： String listDelimiter,
  "columns": List<String> columns,
  "pattern": String pattern
}

2.7 javascript

"parseSpec":{
  "format":"javascript",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
  "function": String function
}

2.8 parquet

"parseSpec":{
  "format":"parquet",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec
}

2.9 nested

"parseSpec":{
  "format":"nested",
  "timestampSpec":TimestampSpec timestampSpec,
  "dimensionsSpec":DimensionsSpec dimensionsSpec,
  "decollator":String decollato,
  "parseSpec": ParseSpec parseSpec,
  "dimensionParseSpec":  Map<String, ParseSpec> dimensionParseSpec
}

3. 维度说明

维度说明用于定义Tindex的schema，包括维度的名称和类型。

3.1 timestampSpec

时间列定义

timestampSpec:{
  "column": String timestampColumn,
  "format": String format,
  "timeZone": String timeZone,
  "missingValue": DateTime missingValue,
  "reNewTime": Boolean reNewTime,
  "excludeTimeColumn": Boolean excludeTimeColumn,
}

3.1.1 时间格式支持

• yy-MM-dd HH:mm:ss.SSS: 自定义的时间格式
• auto: 自动识别时间，支持iso和millis格式
• iso: iso标准时间格式，如2016-08-03T12:53:51.999Z
• posix: 从1970年1月1日开始所经过的秒数,10位的数字
• millis: 从1970年1月1日开始所经过的毫秒数，13位数字

3.2 dimensionsSpec

事件维度列定义

dimensionsSpec:{
  "dynamicDimension": Boolean dynamicDimension,
  "defaultDynamicType": String defaultDynamicType,
  "enableStored": Boolean enableStored,
  "dimensions": List<DimensionSchema> dimensions,
  "dimensionExclusions": List<String> dimensionExclusions,
  "spatialDimensions": List<SpatialDimensionSchema> spatialDimensions
}

3.2.1 DimensionSchema

• 基本维度
• 时间维度
• 聚合维度

3.2.1.1 基本维度：

{
  "type": String type,
  "name": String name,
  "hasMultipleValues": Boolean hasMultipleValues
}

3.2.1.2 时间维度：

{
  "type": "date",
  "name": String name,
  "hasMultipleValues": Boolean hasMultipleValues,
  "format": String format,
  "timeZone": String timeZone
}

3.2.1.3 聚合维度

{
  "type": "aggregator",
  "name": String name,
  "aggType": String aggType
}