# Log Collection

The Unified Log Service (ULS) uses ULogAgent for log collection. This guide covers how to configure log collection rules in the console and deliver logs to ULogService.

## Text Log Collection

ULogAgent supports the following extraction modes for text log collection:

| Extraction Mode | Description |
| --- | --- |
| Single-line Full Text | Treats each line as a complete log entry, separated by newlines (`\n`). No structured processing applied. |
| Multi-line Full Text | Matches multi-line logs using a first-line regex pattern. No structured processing applied. |
| JSON | Parses Object-type JSON logs, extracting first-level keys as field names and values as field values. |
| Delimiter | Splits each log line by a specified delimiter and maps each segment to a custom field name (key). |
| Single-line Full Regex | Uses a single regex expression to extract structured fields from single-line logs. |
| Multi-line Full Regex | Matches multi-line logs by a first-line regex, then extracts structured fields using a regex or delimiter. |

### Step 1: Create/Select a Log Topic

Refer to Topic and Topic Set Management.

### Step 2: Configure Collection

#### Configure Log File Path

**File Path** specifies the directory and file name of the logs. ULogAgent matches directories according to the path pattern and monitors log files that match the rules. You can set up to 10 different collection paths.

The collection path can specify an exact directory and file name, or use wildcards for fuzzy matching.

Common configuration examples:

| Configuration Method | Directory Prefix | File Name Pattern | Description |
| --- | --- | --- | --- |
| Exact file name | /var/log/nginx | access.log | The path is configured as `/var/log/nginx/**/access.log`. ULogAgent monitors all `access.log` files in subdirectories under `/var/log/nginx`. |
| File name suffix match | /var/log/nginx | *.log | The path is configured as `/var/log/nginx/**/*.log`. ULogAgent monitors all `.log` files in subdirectories under `/var/log/nginx`. |
| File name fuzzy match | /var/log/nginx | error* | The path is configured as `/var/log/nginx/**/error*`. ULogAgent monitors all files starting with `error` in subdirectories under `/var/log/nginx`. |

#### Configure Collection Strategy

- **Full**: ULogAgent reads from the beginning of the file.
- **Incremental**: ULogAgent only collects newly added content in the file.

#### Configure Encoding Mode

- **UTF-8**: Select this option if your log file encoding is UTF-8.
- **GBK**: Select this option if your log file encoding is GBK.

#### Configure Extraction Mode

Choose one of the following extraction modes based on your log format:

##### Single-line Full Text

Suitable when each line is a complete log entry and no structured processing is needed.

Example log:
```
123.123.123.123 - - [17/Sep/2025:15:24:30 +0800] "GET /index.html HTTP/1.1" 200 612 "https://example.com/referer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" "-"
```

Processed result:
```
"message": "123.123.123.123 - - [17/Sep/2025:15:24:30 +0800] \"GET /index.html HTTP/1.1\" 200 612 \"https://example.com/referer\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\" \"-\""
```

On the collection configuration page, set **Extraction Mode** to **Single-line Full Text**.

##### Multi-line Full Text

Suitable when a log entry spans multiple lines and no structured processing is needed. Uses a first-line regex to identify the start of each log entry.

1. On the collection configuration page, set **Extraction Mode** to **Multi-line Full Text**.
2. Enter the first-line regex pattern. ULogService uses this pattern to match the beginning of each log entry.

##### JSON

Suitable for Object-type JSON logs. ULogAgent extracts first-level keys as field names and first-level values as field values.

1. On the collection configuration page, set **Extraction Mode** to **JSON**.

**Upload Parse Failure Logs**: Controls whether logs that fail to parse are uploaded. Enabled by default.
- **Enabled**: All logs that fail to parse are uploaded with a specified key name (default: `LogParseFailure`) and the raw log content as the value.
- **Disabled**: Logs that fail to parse are not uploaded.

##### Delimiter

Suitable for structured logs with clear delimiters.

1. Set **Extraction Mode** to **Delimiter**.
2. Select a delimiter (supports space, tab, vertical bar, semicolon, comma), enter a log sample, and click Extract.
3. The system splits the log sample by the delimiter and displays the extracted fields. Assign a unique key to each field.

##### Single-line Full Regex

Suitable for single-line logs that need structured field extraction via regex.

1. Set **Extraction Mode** to **Single-line Full Regex** and enter a log sample.
2. Enter the regex expression to extract and verify key-value pairs.

##### Multi-line Full Regex

Suitable for multi-line logs that need structured field extraction. Supports both regex and delimiter as the processing plugin.

1. Set **Extraction Mode** to **Multi-line Regex** and enter a log sample.
2. Enter the first-line regex pattern and click Verify.
3. Select the processing plugin:
   - **Regex**: Enter a regex expression to extract and verify key-value pairs.
   - **Delimiter**: Select or enter a delimiter to extract and verify key-value pairs.

For Single-line Full Regex, Multi-line Full Regex, and JSON modes, the **Upload Parse Failure Logs** option is also available (default: enabled).

### Step 3: Configure Index

After completing the collection configuration, go to the **Index Configuration** page to set up index configuration based on your business needs.

---

## UK8S Cluster Log Collection

This section describes how to configure log collection rules for UK8S clusters in the console and deliver logs to ULogService.

### Prerequisites

A cluster has been created in the [UK8S Console](https://console.feb.sg/uk8s/manage).

### Step 1: Select Log Topic and Cluster

1. Log in to the [ULogService Console](https://console.feb.sg/ulogservice/topic).
2. Select the target topic and click **Log Access**.
3. Select the access log type to enter the UK8S selection page.
4. Select the UK8S cluster instance to access.

### Step 2: Configure Collection Rules

After selecting the cluster, click **Next** to configure collection rules:

- **Log Source Configuration**:
  - **Collection Type**: Currently supports container file paths. You can specify the log source by workload or Pod labels.

- **Metadata Configuration**:
  In addition to the raw log content, ULogService can also attach container or Kubernetes metadata (e.g., container ID) when reporting to ULogService, making it easier to trace log sources or search by container identifiers (e.g., container name, labels). You can choose whether to upload this metadata.

- **Parse Rule Configuration**:
  - **Collection Strategy**: Full or Incremental.
    - **Full**: Collect from the beginning of the log file.
    - **Incremental**: Only collect newly added content.
  - **Encoding Mode**: Supports UTF-8 and GBK.
  - **Extraction Mode**: Supports multiple types including JSON, Delimiter, Single-line Full Text, Multi-line Full Text, Single-line Full Regex, and Multi-line Full Regex. See **Text Log Collection** above for details.

### Step 3: Configure Index

1. Click **Create** to go to the **Index Configuration** page.
2. Set the index configuration according to your business requirements.

### Step 4: Search Logs

At this point, the UK8S cluster log collection setup is complete. You can go to [Console > Search & Analysis](https://console.feb.sg/ulogservice/ulogservice) to view the collected logs.