What's the difference between the tail and forward source types?

tail reads an existing log file from disk (you give it a path, a position file to track how far it's read, and a tag), which is the typical setup for reading an application's own log files. forward listens on a TCP port for events sent by other Fluentd or Fluent Bit agents, which is the typical setup for a central aggregator collecting logs from many hosts.

Why does the filter and match block use a tag like `**` for forward sources?

A tail source has one specific tag you choose, so the filter and match blocks can target that exact tag. A forward source receives events tagged by whichever upstream agent sent them, so there's no single fixed tag to match, and this generator uses Fluentd's `**` wildcard so the filter and match apply to everything the forward input receives.

What do flush_interval and retry_max_times actually control?

Both live inside the output's block. flush_interval sets how often Fluentd flushes buffered events to the destination (shorter means lower latency but more requests); retry_max_times caps how many times Fluentd retries a failed flush before giving up on that chunk, which matters for how long a downstream outage can be tolerated without losing data.

Does this validate that Elasticsearch, Loki, Kafka, or S3 is actually reachable?

No. It only validates that the fields each output type requires (like host and index_name for Elasticsearch, or bucket for S3) are filled in; it doesn't open a network connection to check that the destination exists or is reachable, since this runs entirely in your browser.

Fluentd Config Generator - Build fluent.conf Visually

Overview

Introduction

A working fluent.conf needs a source block, at least one filter, and an output (match) block, all nested with Fluentd's tag-based directive syntax, and it's easy to end up with a filter or match block whose tag doesn't line up with the source that's supposed to feed it.

This tool builds that file from a form: pick a source type, add a static field via a record_transformer filter, choose an output destination, and set buffer flush/retry behavior, and the resulting fluent.conf keeps every tag consistent automatically.

What Is Fluentd Config Generator?

A visual builder for fluent.conf files covering the three directives every Fluentd pipeline needs: <source> (tail or forward), <filter> (record_transformer), and <match> (elasticsearch, loki, kafka, or s3), plus the <buffer> settings nested inside the match block.

It mirrors Fluentd's own configuration model rather than inventing a new one, so the output can be dropped directly into a real fluent.conf or td-agent.conf.

How Fluentd Config Generator Works

The source's tag (either the one you set for a tail source, or the `**` wildcard for a forward source) is reused for both the <filter> and <match> block headers, so events flowing through the source are guaranteed to hit the filter and land in the configured output.

The record_transformer filter always emits a <record> block; if you don't add a custom static field, it defaults to a single example field so the block is never emitted empty. The output's specific fields (host/index_name for Elasticsearch, url for Loki, brokers/topic for Kafka, bucket/path for S3) are only included for the output type you selected.

When To Use Fluentd Config Generator

Use it when standing up a new Fluentd pipeline and you want a syntactically correct starting fluent.conf without memorizing each output plugin's exact field names.

It's also useful for quickly comparing what an Elasticsearch versus a Loki versus a Kafka output block looks like for the same source and filter setup.

Often used alongside Fluent Bit Config Generator.

Features

Advantages

Keeps the source, filter, and match tags consistent automatically, avoiding the classic mistake of a filter or match block that silently never fires because its tag doesn't match the source.
Covers four of the most common Fluentd output plugins (Elasticsearch, Loki, Kafka, S3) with each one's actual required fields, not a generic placeholder.
Generates a real <buffer> block with flush_interval and retry_max_times, a section many hand-written quick configs skip entirely.

Limitations

Only models one source, one filter, and one output; real production pipelines often chain multiple filters or route to multiple outputs with <label>, which this generator doesn't build.
Doesn't validate connectivity to the destination service or verify credentials/authentication blocks that Elasticsearch, Kafka, or S3 outputs often need in production (like user/password or aws_key_id).
The record_transformer filter only supports adding one static field; more advanced transformations (ruby expressions, nested records) need manual editing after generating.

Examples

Tailing an app log into Elasticsearch

Input

source: tail, path /var/log/app.log, tag app.log; output: elasticsearch, host localhost, index_name app-logs

Output

<source>
  @type tail
  path /var/log/app.log
  pos_file /var/log/td-agent/buffer.pos
  tag app.log
  <parse>
    @type none
  </parse>
</source>

<filter app.log>
  @type record_transformer
  <record>
    environment production
  </record>
</filter>

<match app.log>
  @type elasticsearch
  host localhost
  port 9200
  index_name app-logs
  <buffer>
    flush_interval 5s
    retry_max_times 5
  </buffer>
</match>

The tail source's tag (app.log) is reused for both the filter and match block headers, and the buffer defaults to a 5-second flush with 5 retries.

Best Practices & Notes

Best Practices

Give tail sources a specific, descriptive tag (like app.access or app.error) rather than a generic one, so downstream filters and matches can be scoped precisely.
Keep retry_max_times high enough to survive a brief downstream outage, but not so high that a persistently broken output silently accumulates unbounded buffered chunks on disk.
Add authentication fields (user/password for Elasticsearch, SASL config for Kafka, IAM credentials for S3) manually after generating, since production destinations almost always require them.

Developer Notes

The generator emits plain Fluentd configuration text (no YAML/JSON library involved) using a 2-space indent() helper identical in shape to the one used by other config generators in this app. Tag consistency is enforced by computing the tag once (the tail tag, or the literal `**` for forward sources) and reusing that single value for the <filter> and <match> headers, so the three directives can never drift out of sync.

Fluentd Config Generator Use Cases

Scaffolding a new Fluentd pipeline for a service that needs to ship logs to Elasticsearch, Loki, Kafka, or S3
Producing a reference fluent.conf to compare against an existing hand-written one for drift
Quickly prototyping buffer flush/retry settings before tuning them against real traffic

Common Mistakes

Using a forward source but expecting the filter/match tag to be a specific value, forward inputs receive whatever tag the upstream sender used, which is why this generator falls back to the `**` wildcard.
Forgetting that S3 and Kafka outputs in production almost always need authentication fields this generator doesn't include, and shipping the generated config as-is without adding them.
Setting retry_max_times very high without also thinking about disk space, since Fluentd buffers unflushed chunks to disk while retrying.

Tips

Generate the Elasticsearch variant first if you're unsure which output to use, it's the most widely deployed Fluentd destination and easiest to test locally.
Use the Fluent Bit Config Generator instead if you need a lighter-weight agent for an edge node or sidecar container rather than a full Fluentd/td-agent install.

Fluentd Config Generator