uken/fluent-plugin-elasticsearch

## Changelog [[tags]](https://github.com/uken/fluent-plugin-elasticsearch/tags)

### [Unreleased]

### 5.4.3
- add data_stream_template\_use\_index\_patterns\_wildcard in elasticsearch\_data\_stream (#1040)

### 5.4.2
- in\_elasticsearch: Avoid to use deprecated endpoint for clear\_scroll (#1039)

### 5.4.1
- Adjust ilm endpoint (#1036)

### 5.4.0
- Fix support for host/hosts placeholders for ipv6 addresses (#1030)
- Handle newer es library (#1032)

### 5.3.0

- Unpin `faraday` from v1, upgrade to v2. (#1012)
  Note that if you can't migrate other plugins from `faraday` v1 yet, need to keep
  fluent-plugin-elasticsearch v5.2.5.

### 5.2.5

- Update the name of parameter about `thread_pool`.
  It should be `thread_pool.write.queue_size` instead of `thread_pool.bulk.queue_size` (#1000)
- Replace obsolete `File.exists?` with `File.exist?` (#998)
- Support `compression_level` for `elasticsearch_data_stream` (#995)
- Honor `hosts` parameter for `elasticsearch_data_stream` (#1008)
- Drop outdated coveralls (#1009)

### 5.2.4
- Pin Faraday 1.10 (#987)
- Increase errors metric on error response in data stream (#986)

### 5.2.3
- Bump actions/checkout from 2 to 3 (#978)
- chore: Included githubactions in the dependabot config (#977)
- chore: Set permissions for GitHub actions (#972)
- Remove nested msgpack\_each in handle\_error (#970)
- do not overwrite @timestamp in data stream if it already exists in the record (#968)

### 5.2.2
-  Add missing top level class markers (#961)
- Ensure use_record_as_seed for same records (#960)

### 5.2.1
- respect include\_tag\_key and tag\_key setting when using data streams (#936)
- Handle unsupported version error (#956)
- Display deprecated warning on ES dynamic plugin (#955)

### 5.2.0
-  Migrate to handle Elasticsearch 8 (#949)

### 5.1.5
- Make retryable DataStreams creation at configure phase (#943)
- Handle @hosts parameter on data_stream plugin (#942)
- allow specifying custom ILM policies for data streams (#933)

### 5.1.4
- Handle ES8 or above more strictly (#931)
- fixing double "\_policy" in index lifecycle management policy for elasticsearch\_data\_stream output (#930)

### 5.1.3
-  fixing execution order for dynamic data stream creation (#928)

### 5.1.2
-  Fix default values of datastream parameters (#926)

### 5.1.1
- Report appropriate error for data_stream parameters (#922)
- Add ILM and template parameters for data streams (#920)
- Support Buffer in Data Stream Output (#917)

### 5.1.0
- Correct default target bytes value (#914)
- Handle elasticsearch-ruby 7.14 properly (#913)

### 5.0.5
- Drop json_parse_exception messages for bulk failures (#900)
- GitHub Actions: Drop Ruby 2.5 due to EOL (#894)

### 5.0.4
- test: out_elasticsearch: Remove a needless headers from affinity stub (#888)
- Target Index Affinity (#883)

### 5.0.3
- Fix use_legacy_template documentation (#880)
- Add FAQ for dynamic index/template (#878)
- Handle IPv6 address string on host and hosts parameters (#877)

### 5.0.2
- GitHub Actions: Tweak Ruby versions on test (#875)
- test: datastreams: Set nonexistent datastream as default (#874)
- Fix overwriting of index template and index lifecycle policy on existing data streams (#872)

### 5.0.1
- Use elasticsearch/api instead of elasticsearch/xpack (#870)

### 5.0.0
- Support #retry_operate on data stream (#863)
- Support placeholder in @data\_stream\_name for @type elasticsearch\_data\_stream (#862)
- Extract troubleshooting section (#861)
- Fix unmatched `<source>` close tag (#860)
- Initial support for Elasticsearch Data Stream (#859)

### 4.3.3
- Handle invalid Elasticsearch::Client#info response (#855)

### 4.3.2
- Use log.debug to dump ES 8.x _type information (#852)

### 4.3.1
- Unsplit huge records by default (#851)

### 4.3.0
- Add cloud_id & cloud_auth settings (#850)
- Fix failing tests on es6 client (#848)
- feat: retry on transport errors (#846)
- fix typo (#838)
- Remove unnecessary nil check (#826)

### 4.2.2
- Remove unnecessary nil check (#826)
- Fix ILM rollover index template pattern to apply index_separator (#825)
- Fix ILM rollover alias creation when a placeholder is used in index_name (#823)
- Add a note about the pitfalls of per-date indexes used with ILM (#822)

### 4.2.1
- Update a broken link (#821)
- Include chunk_id in records (#820)
- Fix a failing testcase (#818)

### 4.2.0
- ci: Add Ruby 2.7 jobs (#812)
- Support Elasticsearch new style template (#810)

### 4.1.4
- ElasticsearchGenID update docs for hash_type (#809)
- Handle api key header (#808)
- index_template: Handle error object entirely on index creation failure (#807)

### 4.1.3
- Load multiple templates even if template_name and template_file given (#799)
- Handle elasticsearch-ruby 7.9.0 using HTTP method changes (#795)

### 4.1.2
- Use Hash#dig instead of Hash#[] on retrieving version information from Elasticsearch info API (#793)

### 4.1.1
- Correct ILM explain on logstash_format case (#786)

### 4.1.0
- Implement Fallback selector and configurable selector class (#782)

### 4.0.11
- Add custom and time placeholders combination testcase for ILM (#781)
- Really support ILM on `logstash_format` enabled environment (#779)

### 4.0.10
- filter_elasticsearch_genid: Use entire record as hash seed (#777)
- Suppress type in meta with suppress_type_name parameter (#774)
- filter\_elasticsearch\_genid: Add hash generation mechanism from events (#773)
- Clean up error text (#772)
- Use GitHub Actions badges instead of Travis' (#760)
- Add issue auto closer workflow (#759)
- Document required permissions (#757)

### 4.0.9
- Add possibility to configure multiple ILM policies (#753)
- Document required permissions (#757)

### 4.0.8
- Handle compressable connection usable state (#743)
- Use newer tls protocol versions (#739)
- Add GitHub Actions file (#740)

### 4.0.7
- Added http_backend_excon_nonblock config in out_elasticsearch (#733)

### 4.0.6
- Add fallback mechanism for handling to detect es version (#730)
- Remove needless section (#728)
- Handle exception if index already exists (#727)
- Tweak test cases (#726)

### 4.0.5
-  add logstash_dateformat as placeholder (#718)
- Tweak travis.yml for suppressing validator warnings and add CI for Linux Arm64 architecture and macOS 10.14 (#724)
- Elasticsearch ruby v7.5 (#723)
- Add Oj serializer testcases for all job (#722)
- Update documentation for ILM (#721)

### 4.0.4
- Provide clearing caches timer (#719)

### 4.0.3
-  Use http.scheme settings for persisent scheme setup (#713)

### 4.0.2
- Support TLSv1.3 (#710)

### 4.0.1
- Placeholders for template name and customize template (#708)
- Add overwriting ilm policy config parameter (#707)
- Fix a failing ILM config testcase (#706)

### 4.0.0
- Restructuring ILM related features (#701)
- Extract placeholders in pipeline parameter (#695)
- fix typo in `README.md` (#698)
- Reduce log noise when not using rollover_index (#692)

### 3.8.0
- Add FAQ for specifying index.codec (#679)
- Add FAQ for connect_write timeout reached error (#687)
- Unblocking buffer overflow with block action (#688)

### 3.7.1
- Make conpatible for Fluentd v1.8 (#677)
- Handle flatten_hashes in elasticsearch_dynamic (#675)
- Handle empty index_date_pattern parameter (#674)

### 3.7.0
- Tweak for cosmetic change (#671)
- Fix access to Elasticsearch::Transport::VERSION with explicit top level class path (#670)
- Implement Elasticsearch Input plugin (#669)

### 3.6.1
- retry upsert on recoverable error. (#667)
- Allow `_index` in chunk_keys (#665)
- Support compression feature (#664)

### 3.6.0
- Set order in newly created templates (#660)
- Merge Support index lifecycle management into master (#659)
- Support template installation with host placeholder (#654)
- Support index lifecycle management (#651)

### 3.5.6
- Support elasticsearch8 removal of mapping types (#656)
- Upgrade webmock to 3 (#652)
- Suppress `ruby -c` warnings (#649)
- Add tips for sniffer class (#644)
- Make `client_key_pass` secret (#637)
- Validate `user` and `password` early (#636)

### 3.5.5
- Fix arguments order of `assert_equal` (#635)
- Add description for `port` option (#634)
- Fix description position and add examples for `hosts` option (#633)
- Use upper-case to compare before and after conversion (#630)
- Validate `max_retry_get_es_version` (#629)
- Remove unused getting plugin instance (#628)
- Fix error message for `max_retry_putting_template` (#627)
- Fix DST-breaking unit test (#623)
- Handle ClusterBlockException (#621)
- Add contribution guideline document (#618)

### 3.5.4
- Add FAQ for Fluentd seems to hang if it unable to connect Elasticsearch (#617)
- Check bulk_message size before appending (#616)
- Add FAQ for Elasticsearch index mapping glitch (#614)
- Display retry counts and interval (#613)

### 3.5.3
- Handle nil items response (#611)

### 3.5.2
- Fix `@meta_config_map` creation timing (#592)

### 3.5.1
- Configurable split request size threshold (#586)

### 3.5.0
- Adopt Elasticsearch ruby client v7 loggable class (#583)

### 3.4.3
- Add fail_on_putting_template_retry_exceed config (#579)

### 3.4.2
- Comparing DEFAULT_TYPE_NAME_ES_7x to target_type instead of type_name (#573)

### 3.4.1
- Handle non-String value on parse_time (#570)

### 3.4.0
- Check exclusive feature on #configure (#569)
- Modify FAQ for highly load k8s EFK stack (#566)
- Add FAQ for high load k8s EFK stack (#564)

### 3.3.3
- Add unit test for exception message (#563)
- Add ignore_exceptions config (#562)

### 3.3.2
- Fix support for host, hosts placeholders (#560)
- typo fixes in README.md (#559)

### 3.3.1
- add new option to suppress doc wrapping (#557)
- Include 2 (#555)

### 3.3.0
- Support builtin placeholders for host and hosts parameter (#554)

### 3.2.4
- Pass chunk for built in placeholders (#553)

### 3.2.3
- Expose exception backtrace for typhoeus gem loading error (#550)

### 3.2.2
- Don't validate ES cliuent version under dry-run mode (#547)

### 3.2.1
- Don't attempt to connect to Elasticsearch in dry run mode (#543)
- Add FAQ for typhoeus gem installation (#544)

### 3.2.0
- Split huge record requests (#539)

### 3.1.1
- Add document for custom_headers (#538)
- out_elasticsearch: Add custom_headers parameter (#529)
- Bundle irb on Ruby 2.6 or later (#537)

### 3.1.0
- Retry obtaining Elasticsearch version (#532)
- Fix broken id links (#530)

### 3.0.2
-  appveyor: Remove Ruby 2.1 CI targets on AppVeyor (#524)
- Follow removal of _routing field change on recent Elasticsearch (#523)
- Travis: Tweak to use Ruby versions (#522)

### 3.0.1
- Remove needless Elasticsearch version detection (#520)

### 3.0.0
- Use fluentd core retry mechanism (#519)
- Depends on builtin retrying mechanism (#518)
- Loosen ConnectionRetryFailure condition when flush_thread_count > 1 and depends on Fluentd core retrying mechanism (#516)

### 2.12.5
- Ensure sniffer class constants definition before calling #client (#515)

### 2.12.4
- #506 Rollover index will be in effect in case of template overwrite also. (#513)

### 2.12.3
- Added log_es_400_reason configuration item (#511)
- Allow a user to specify the rollover index date pattern (#510)

### 2.12.2
- Verify connection at startup (#504)
- Add faq for glob pattern tag routing (#502)

### 2.12.1
- Make configurable unrecoverable types (#501)
- Add FAQ for TLS enabled nginx proxy TLS version incompatibility trouble (#496)
- Add FAQs (#492)
- Remove issuestats.com badges (#489)

### 2.12.0
- Decoupling the custom template and rollover index creation #485 (#486)

### 2.11.11
- Handle error not to acquire version information (#479)

### 2.11.10
- Verbose error reason output (#469)

### 2.11.9
- Use ConnectionRetryFailure in plugin specific retrying for consistency (#468)
- Remove outdated generating hash_id_key code (#466)
- Tweak behavior for UnrecoverableError and #detect_es_major_version (#465)

### 2.11.8
- Serialize requests with Oj (#464)

### 2.11.7
- Add mechanism to detect ES and its client version mismatch (#463)

### 2.11.6
- 355 customize template (#431)

### 2.11.5
- Uplift Merge pull request #459 from richm/v0.12-simple-sniffer (#461)

### 2.11.4
- Persistent backend (#456)

### 2.11.3
- Implement the `include_index_in_url` option for out_elasticsearch (#451)
- Add an option `include_index_in_url` to allow URL-based conrtrols (#450)

### 2.11.2
- Strictness scheme (#445)

### 2.11.0
- Uplift Merge pull request #437 from jcantrill/fix_bulk_count (#438)

### 2.10.5
- Uplift Merge pull request #435 from jcantrill/add_trace_logging (#436)

### 2.10.4
- Use Fluent::UnrecoverableError as unrecoverable error class ancestors (#433)
- Add parameter validation for retrying template installation (#429)

### 2.10.3
- Add retry mechanism for template installation (#428)

### 2.10.2
- Use logstash_prefix_separator on elasticsearch_dynamic (#427)

### 2.10.1
- Uplift Merge pull request #419 from jcantrill/retry_prefix (#421)
- Uplift Merge pull request #418 from jcantrill/emit_exception (#420)

### 2.10.0
- Uplift Merge pull request #405 from jcantrill/sanitize_bulk (#414)

### 2.9.2
- Uplift Merge pull request #410 from richm/v0.12-consistent-errors-and-tests (#411)
- Add correct link for include_timestamp (#408)

### 2.9.1
- Uplift Merge pull request #406 from richm/v0.12-successes-duplicates-no-retry (#407)

### 2.9.0
- DLQ revisited v1 uplifted #398, #402 (#404)

### 2.8.6
- auth: Fix missing auth tokens after reloading connections (#394)

### 2.8.5
- Add deprecated option into content_type parameter (#391)

### 2.8.4
- Use nanosecond precision in elasticsearch_dynamic (#387)

### 2.8.3
- Specify SSL/TLS version in out_elasticsearch_dynamic (#385)

### 2.8.2
- Revert content type header default value (#383)

### 2.8.1
- Restore default value of type name #(377)

### 2.8.0
- Remove outdated generating hash id support module (#373)
- Check Elasticsearch major version (#371)

### 2.7.0
- Configureable content type (#367)

### 2.6.1
- Prevent error when using template in elasticsearch_dynamic for elementally use case (#363)

### 2.6.0
- Handle high precision time format when using custom time_key (#360)

### 2.5.0
- Using nested record in `id_key`, `parent_key`, and `routing_key` (#351)
- Fix inverted case of a proper noun "Elasticsearch" (#349)

### 2.4.1
- Add config parameter to enable elasticsearch-ruby's transporter logging (#342)

### 2.4.0
- Add built-in placeholders support against type_name parameter (#338)

### 2.3.0
- Allow overwriting existing index template (#239)

### 2.2.0
- GA release 2.2.0.

### 2.2.0.rc.1
- Separate generate hash id module and bundled new plugin for generating unique hash id (#331)

### 2.1.1
- Raise ConfigError when specifying different @hash_config.hash_id_key and id_key configration (#327)
- Small typo fix in README.md (#325)

### 2.1.0
- Retry on certain errors from Elasticsearch (#322)

### 2.0.1
- Releasing generating hash id mechanism to avoid records duplication feature.

### 2.0.1.rc.1
- Add generating hash id mechanism to avoid records duplication (#318)

### 2.0.0
- Release for Fluentd v0.14 stable.

### 2.0.0.rc.7
- Add `include_timestamp` option (#310)

### 2.0.0.rc.6
- Improve documentation (#304)
- Handle dynamic_config misconfigurations (#305)
- Escape basic authentication user information placeholders (#306)

### 2.0.0.rc.5
- make configurable with `ssl_version` parameter (#299)
- add `logstash_prefix_separator` config parameter (#297)

### 2.0.0.rc.4
- fix license identifier in gemspec (#294)

### 2.0.0.rc.3
- add built-in placeholders support (#288, #293)
- permit multi workers feature (#291)

### 2.0.0.rc.2
- add pipeline parameter (#290)

### 2.0.0.rc.1
- Use v0.14 API to support nanosecond precision (#223)

### 1.9.5
- sub-second time precision [(#249)](https://github.com/uken/fluent-plugin-elasticsearch/pull/249)

### 1.9.4
- Include 'Content-Type' header in `transport_options`

### 1.9.3
- Use latest elasticsearch-ruby (#240)
- Log ES response errors (#230)

### 1.9.2
- Fix elasticsearch_dynamic for v0.14 (#224)

### 1.9.1
- Cast `reload_*` configs in out_elasticsearch_dynamic to bool (#220)

### 1.9.0
- add `time_parse_error_tag` (#211)
- add `reconnect_on_error` (#214)

### 1.9.0.rc.1
- Optimize output plugins (#203)

### 1.8.0
- fix typo in defaults for ssl_verify on elasticsearch_dynamic (#202)
- add support for `templates` (#196)
- rename `send` method to `send_bulk` (#206)

### 1.7.0
- add support for `template_name` and `template_file` (#194)

### 1.6.0
- add support for dot separated `target_index_key` and `target_type_key` (#175)
- add `remove_keys_on_update` and `remove_keys_on_update_key` (#189)
- fix support for fluentd v0.14 (#191)
- remove support for elasticsearch v2 for now (#177)

### 1.5.0
- add `routing_key` (#158)
- add `time_key_exclude_timestamp` to exclude `@timestamp` (#161)
- convert index names to lowercase (#163)
- add `remove_keys` (#164)
- add `flatten_hashes` (#168)
- add `target_type_key` (#169)

### 1.4.0
- add `target_index_key` to specify target index (#153)
- add `time_key_format` for faster time format parsing (#154)

### 1.3.0
- add `write_operation`

### 1.2.1
- fix `resurrect_after` in out_elasticsearch_dynamic

### 1.2.0
- out_elasticsearch_dynamic get memory improvement and fix for race condition (#133)
- Add `resurrect_after` option (#136)

### 1.1.0
- Support SSL client verification and custom CA file (#123)
- Release experimental `type elasticsearch_dynamic` (#127)

### 1.0.0
- password config param is now marked as secret and won't be displayed in logs.

### 0.9.0
- Add `ssl_verify` option (#108)

### 0.8.0
- Replace Patron with Excon HTTP client (#93)

### 0.7.0
- Add new option `time_key` (#85)

### 0.6.1
- 0.10.43 is minimum version required of fluentd (#79)

### 0.6.0
- added `reload_on_failure` and `reload_connections` flags (#78)

### 0.5.1
- fix legacy hosts option, port should be optional (#75)

### 0.5.0
- add full connection URI support (#65)
- use `@timestamp` for index (#41)
- add support for elasticsearch gem version 1 (#71)
- fix connection reset & retry when connection is lost (#67)

### 0.4.0
- add `request_timeout` config (#59)
- fix lockup when non-hash values are sent (#52)

### 0.3.1
- force using patron (#46)
- do not generate @timestamp if already part of message (#35)

### 0.3.0
- add `parent_key` option (#28)
- have travis-ci build on multiple rubies (#30)
- add `utc_index` and `hosts` options, switch to using `elasticsearch` gem (#26, #29)

### 0.2.0
- fix encoding issues with JSON conversion and again when sending to elasticsearch (#19, #21)
- add logstash_dateformat option (#20)

### 0.1.4
- add logstash_prefix option

### 0.1.3
- raising an exception on non-success response from elasticsearch

### 0.1.2
- add id_key option

### 0.1.1
- fix timezone in logstash key

### 0.1.0
 - Initial gem release.