{"id":242,"date":"2021-10-27T18:48:07","date_gmt":"2021-10-27T18:48:07","guid":{"rendered":"https:\/\/www.huerter.me\/?p=242"},"modified":"2021-11-11T20:34:42","modified_gmt":"2021-11-11T20:34:42","slug":"howto-amazon-opensearch-als-elk-ersatz-mit-logstash","status":"publish","type":"post","link":"https:\/\/www.huerter.me\/index.php\/2021\/10\/27\/howto-amazon-opensearch-als-elk-ersatz-mit-logstash\/","title":{"rendered":"HowTo: Amazon OpenSearch Cluster als ELK Ersatz mit Logstash"},"content":{"rendered":"<p>Vorg\u00e4nger: <a href=\"https:\/\/www.huerter.me\/index.php\/2019\/02\/28\/howto-install-elk-stack-unter-debian-9\/\">HowTo: Install ELK-Stack unter Debian 9<\/a><\/p>\n<p>Das folgende HowTo zeigt wie ein Amazon OpenSearch Cluster installiert werden kann mithilfe von Docker Containern bzw. Docker-Compose.<\/p>\n<p>OpenSearch ist ein auf dem ELK-Stack (Elasticsearch Version 7.10.2), der letzten &#8222;freien&#8220; Version, aufbauender Ersatz welcher das Sammeln, Auswerten, Anreichern und Darstellen von Logs bzw. Logeintr\u00e4gen erm\u00f6glicht.<\/p>\n<p>&nbsp;<\/p>\n<h2>1. Vorbereitung:<\/h2>\n<ul>\n<li>fertig installierter Ubuntu 20.04 LTS Server (&gt;=4GB RAM empfohlen)<\/li>\n<li>Dokumentation OpenSearch (<a href=\"https:\/\/opensearch.org\/docs\/latest\">https:\/\/opensearch.org\/docs\/latest<\/a>)<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h3>2. Update<\/h3>\n<pre class=\"aphph-container line-numbers language-bash\"><code class=\" language-bash\"><span class=\"token function\">apt-get<\/span> update <span class=\"token operator\">&amp;&amp;<\/span>apt-get -y dist-upgrade\r\n<span class=\"token function\">apt-get<\/span> <span class=\"token function\">install<\/span><\/code><code class=\" language-bash\">apt-transport-https software-properties-common <span class=\"token function\">wget<\/span> telnet net-tools psmisc <span class=\"token function\">curl<\/span> <span class=\"token function\">ca-certificates curl gnupg lsb-release vim <\/span><\/code><code class=\" language-bash\"><\/code><\/pre>\n<p>&nbsp;<\/p>\n<h3>3. Vorbereitung Installation Docker und Docker-Compose<\/h3>\n<p>Installation von Docker f\u00fcr die Bereitstellung von Container und die Installation von Docker-Compose. Letzteres ist eine Verwaltung von Docker Container mithilfe von Konfigurations-Dateien (yml files).<\/p>\n<p>Hinzuf\u00fcgen des Docker GPG Schl\u00fcssel f\u00fcr die Nutzung des Repository:<\/p>\n<pre class=\"highlight\"><code>curl <span class=\"nt\">-fsSL<\/span> https:\/\/download.docker.com\/linux\/ubuntu\/gpg | <span class=\"nb\">sudo <\/span>gpg <span class=\"nt\">--dearmor<\/span> <span class=\"nt\">-o<\/span> \/usr\/share\/keyrings\/docker-archive-keyring.gpg<\/code><\/pre>\n<p>Erg\u00e4nzen des Docker Repository in der lokalen Konfiguration:<\/p>\n<pre class=\"highlight\">echo \"deb [arch=$(dpkg --print-architecture) signed-by=\/usr\/share\/keyrings\/docker-archive-keyring.gpg] https:\/\/download.docker.com\/linux\/ubuntu $(lsb_release -cs) stable\" | sudo tee \/etc\/apt\/sources.list.d\/docker.list &gt; \/dev\/null<\/pre>\n<p>Installation von Docker:<\/p>\n<pre class=\"highlight\">apt-get update &amp;&amp; apt-get install docker-ce docker-ce-cli containerd.io<\/pre>\n<p>Installation von Docker-Compose:<\/p>\n<pre class=\"highlight\"><code>curl <span class=\"nt\">-L<\/span> <span class=\"s2\">\"https:\/\/github.com\/docker\/compose\/releases\/download\/1.29.2\/docker-compose-<\/span><span class=\"k\">$(<\/span>uname <span class=\"nt\">-s<\/span><span class=\"k\">)<\/span><span class=\"s2\">-<\/span><span class=\"k\">$(<\/span>uname <span class=\"nt\">-m<\/span><span class=\"k\">)<\/span><span class=\"s2\">\"<\/span> <span class=\"nt\">-o<\/span> \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n<p>Ausf\u00fchrrechte setzen, damit docker-compose ausgef\u00fchrt werden kann:<\/p>\n<pre class=\"highlight\"><code>chmod +x \/usr\/local\/bin\/docker-compose \/usr\/local\/bin\/docker-compose<\/code><\/pre>\n<p>Verlinkung von docker-compose, damit dieses systemweit genutzt werden kann, ohne den absoluten Pfad nutzen zu m\u00fcssen:<\/p>\n<pre class=\"highlight\"><code>ln <span class=\"nt\">-s<\/span> \/usr\/local\/bin\/docker-compose \/usr\/bin\/docker-compose<\/code><\/pre>\n<p>&nbsp;<\/p>\n<h3>4. Installation &amp; Konfiguration OpenSearch und OpenSearch-Dashboards<\/h3>\n<p>Der folgende Abschnitt beschreibt wie OpenSearch (Ersatz Elasticsearch) und OpenSearch Dashboards (Kibana Ersatz) installiert wird. Generell kann man OpenSearch per Tar-Archive (Empfohlen f\u00fcr Test- und Entwicklungssysteme) standalone betreiben oder mittels Docker Container. Allgemein sind die Docker Container zu empfehlen, da diese einfach erweiterbar (Skalierbarkeit), leichter zu updaten und komfortabler zu verwalten sind.<\/p>\n<p>Erstellung von Unterordnern f\u00fcr den besseren \u00dcberblick<\/p>\n<pre class=\"highlight\"><span style=\"color: #222222; font-family: monospace;\"><span style=\"background-color: #e9ebec;\">mkdir -p \/opt\/docker\/opensearch\/docker-compose &amp;&amp; cd \/opt\/docker\/opensearch\/docker-compose<\/span><\/span><\/pre>\n<p>Beispiel docker-compose.yml Inhalt, zu finden unter <a href=\"https:\/\/opensearch.org\/docs\/latest\/opensearch\/install\/docker\/#start-a-cluster\u00a0:\">Sample Docker Compose File<\/a><\/p>\n<pre class=\"lang:bash\">version: '3'\r\nservices:\r\n  opensearch-node1:\r\n    image: opensearchproject\/opensearch:1.1.0\r\n    container_name: opensearch-node1\r\n    environment:\r\n      - cluster.name=opensearch-cluster\r\n      - node.name=opensearch-node1\r\n      - discovery.seed_hosts=opensearch-node1,opensearch-node2\r\n      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2\r\n      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping\r\n      - \"OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m\" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM\r\n    ulimits:\r\n      memlock:\r\n        soft: -1\r\n        hard: -1\r\n      nofile:\r\n        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems\r\n        hard: 65536\r\n    volumes:\r\n      - opensearch-data1:\/usr\/share\/opensearch\/data\r\n    ports:\r\n      - 9200:9200\r\n      - 9600:9600 # required for Performance Analyzer\r\n    networks:\r\n      - opensearch-net\r\n  opensearch-node2:\r\n    image: opensearchproject\/opensearch:1.1.0\r\n    container_name: opensearch-node2\r\n    environment:\r\n      - cluster.name=opensearch-cluster\r\n      - node.name=opensearch-node2\r\n      - discovery.seed_hosts=opensearch-node1,opensearch-node2\r\n      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2\r\n      - bootstrap.memory_lock=true\r\n      - \"OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m\"\r\n    ulimits:\r\n      memlock:\r\n        soft: -1\r\n        hard: -1\r\n      nofile:\r\n        soft: 65536\r\n        hard: 65536\r\n    volumes:\r\n      - opensearch-data2:\/usr\/share\/opensearch\/data\r\n    networks:\r\n      - opensearch-net\r\n  opensearch-dashboards:\r\n    image: opensearchproject\/opensearch-dashboards:1.1.0\r\n    container_name: opensearch-dashboards\r\n    ports:\r\n      - 5601:5601\r\n    expose:\r\n      - \"5601\"\r\n    environment:\r\n      OPENSEARCH_HOSTS: '[\"https:\/\/opensearch-node1:9200\",\"https:\/\/opensearch-node2:9200\"]' # must be a string with no spaces when specified as an environment variable\r\n    networks:\r\n      - opensearch-net\r\n\r\nvolumes:\r\n  opensearch-data1:\r\n  opensearch-data2:\r\n\r\nnetworks:\r\n  opensearch-net:<\/pre>\n<p>&nbsp;<\/p>\n<p>Bevor der Container gestartet wird m\u00fcssen noch Limits angehoben werden, da ansonsten Fehlermeldungen wie die folgende angezeigt werden:<\/p>\n<pre class=\"highlight\">opensearch-dashboards | {\"type\":\"log\",\"@timestamp\":\"2021-10-20T18:21:53Z\",\"tags\":[\"error\",\"opensearch\",\"data\"],\"pid\":1,\"message\":\"[ConnectionError]: getaddrinfo ENOTFOUND opensearch-node1 opensearch-node1:9200\"}<\/pre>\n<p>Die L\u00f6sung: Erg\u00e4nzen des Parameters &#8222;vm.max_map_count=262144&#8220; in der &#8222;\/etc\/sysctl.conf&#8220; und anschlie\u00dfendes neuladen der Parameter per &#8222;sysctl -p&#8220;.<\/p>\n<p>Der Container kann wie folgt gestartet werden:<\/p>\n<pre class=\"highlight\"><span style=\"color: #222222; font-family: monospace;\"><span style=\"background-color: #e9ebec;\">docker-compose up -d<\/span><\/span><\/pre>\n<p>Sofern man den Parameter &#8222;-d&#8220; wegl\u00e4sst, startet der Container in der aktuellen Sitzung und man erh\u00e4lt alle Debug\/Log Informationen<\/p>\n<p>Die letzten Zeiles des Outputs (ohne &#8222;-d&#8220; Parameter) sollten in etwa wie folgt aussehen:<\/p>\n<pre class=\"lang:bash\">opensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:57Z\",\"tags\":[\"info\",\"savedobjects-service\"],\"pid\":1,\"message\":\"Starting saved objects migrations\"}\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:57Z\",\"tags\":[\"info\",\"savedobjects-service\"],\"pid\":1,\"message\":\"Creating index .kibana_1.\"}\r\nopensearch-node1         | [2021-11-11T19:53:57,737][DEPRECATION][o.o.d.c.m.MetadataCreateIndexService] [opensearch-node1] index name [.kibana_1] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices\r\nopensearch-node1         | [2021-11-11T19:53:57,760][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch-node1] [.kibana_1] creating index, cause [api], templates [], shards [1]\/[1]\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:57Z\",\"tags\":[\"info\",\"savedobjects-service\"],\"pid\":1,\"message\":\"Pointing alias .kibana to .kibana_1.\"}\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:58Z\",\"tags\":[\"info\",\"savedobjects-service\"],\"pid\":1,\"message\":\"Finished in 312ms.\"}\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:58Z\",\"tags\":[\"info\",\"plugins-system\"],\"pid\":1,\"message\":\"Starting [45] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,notebooksDashboards,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,traceAnalyticsDashboards,reportsDashboards,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]\"}\r\nopensearch-node1         | [2021-11-11T19:53:58,080][INFO ][o.o.c.r.a.AllocationService] [opensearch-node1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]]]).\r\nopensearch-node1         | [2021-11-11T19:53:58,589][INFO ][o.o.c.m.MetadataIndexTemplateService] [opensearch-node1] adding template [tenant_template] for index patterns [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*]\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:58Z\",\"tags\":[\"listening\",\"info\"],\"pid\":1,\"message\":\"Server running at http:\/\/0:5601\"}\r\nopensearch-dashboards    | {\"type\":\"log\",\"@timestamp\":\"2021-11-11T19:53:59Z\",\"tags\":[\"info\",\"http\",\"server\",\"OpenSearchDashboards\"],\"pid\":1,\"message\":\"http server running at http:\/\/0:5601\"}<\/pre>\n<p>Nun werden drei Container gestartet: OpenSearch-Node1 sowie OpenSearch-Node2 (jeweils &#8222;Elasticsearch&#8220;) sowie OpenSearch-Dashboard (Kibana). Sinnvoll w\u00e4re eine Verteilung der Container auf verschiedene virtuelle Maschinen (VMs) oder direkt auf mehrere Hardware Nodes. Zu beachten ist, dass zus\u00e4tzlich Port 9300 freigegeben wird.<\/p>\n<p>Nachdem alle Container korrekt gestartet sind, kann man mittels der IP\/DNS Adresse und dem Protokoll https per Port 9200 auf das Opensearch Backend zugreifen:<\/p>\n<p><a href=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-257\" src=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch.png\" alt=\"\" width=\"390\" height=\"360\" srcset=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch.png 642w, https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch-300x277.png 300w\" sizes=\"auto, (max-width: 390px) 100vw, 390px\" \/><\/a><\/p>\n<p>Ebenso muss nun Opensearch-Dashboard erreichbar sein via &#8222;http:\/\/%ip%:5601&#8220;. Einloggen kann man sich mit den Default-Logindaten &#8222;admin&#8220; und gleichnamigen Passwort:<\/p>\n<p><a href=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-258\" src=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch-1.png\" alt=\"\" width=\"371\" height=\"346\" srcset=\"https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch-1.png 529w, https:\/\/www.huerter.me\/wp-content\/uploads\/2021\/10\/opensearch-1-300x280.png 300w\" sizes=\"auto, (max-width: 371px) 100vw, 371px\" \/><\/a><\/p>\n<h3>5. Installation &amp; Konfiguration Logstash<\/h3>\n<p>Erstellung von Unterordnern f\u00fcr den besseren \u00dcberblick<\/p>\n<pre class=\"highlight\"><span style=\"color: #222222; font-family: monospace;\"><span style=\"background-color: #e9ebec;\">mkdir -p \/opt\/docker\/logstash\/docker-compose &amp;&amp; cd \/opt\/docker\/logstash\/docker-compose<\/span><\/span><\/pre>\n<p>Logstah docker-compose.yml:<\/p>\n<pre class=\"lang:bash\">version: '3'\r\nservices:\r\n  logstash-node01:\r\n    image: opensearchproject\/logstash-oss-with-opensearch-output-plugin:7.13.2\r\n    container_name: logstash\r\n    volumes:\r\n#      - opensearch-logstash1\r\n      - \/opt\/logstash\/data\/config\/pipeline:\/usr\/share\/logstash\/pipeline\r\n      - \/opt\/logstash\/data\/config\/patterns:\/usr\/share\/logstash\/patterns\r\n    ports:\r\n        - \"5044:5044\"\r\n        - \"514:514\"\r\n        - \"10514:10514\"\r\n    networks:\r\n\r\n        - opensearch-net\r\n#volumes:\r\n#  opensearch-logstash1:\r\n\r\nnetworks:\r\n  opensearch-net:<\/pre>\n<p>In der Sektion &#8222;ports&#8220; m\u00fcssen alle Ports hinterlegt werden, welche auch f\u00fcr Inputs genutzt werden.<\/p>\n<p>Datei: \/opt\/logstash\/data\/config\/pipeline\/logstash-opensearch.conf<\/p>\n<pre class=\"lang:bash\">input {\r\n  beats {\r\n    port =&gt; 5044\r\n  }\r\n}\r\n\r\noutput {\r\n  opensearch {\r\n    hosts =&gt; [\"https:\/\/opensearch-node1:9200\"]\r\n    index =&gt; \"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}\"\r\n    ssl =&gt; true\r\n    ssl_certificate_verification =&gt; false\r\n    user =&gt; \"admin\"\r\n    password =&gt; \"admin\"\r\n  }\r\n}<\/pre>\n<p>Die Konfigurationen k\u00f6nnen und sollten bei umfangreicheren Einstellungen in verschiedene &#8222;*.conf&#8220; Dateien ausgelagert werden. Ebenso ist ein sinniges Index-Namensschema zu empfehlen, da Elasticsearch per Default auf 1.000 Indices begrenzt ist. In der o.g. Datei sind auch keine Filter hinterlegt.<\/p>\n<p>&nbsp;<\/p>\n<p>Weitere Hinweise zur Installation von Logstash folgen&#8230;<\/p>\n<p>TODOs:<\/p>\n<ul>\n<li>Zertifikate selbst generieren<\/li>\n<li>Adminzugang f\u00fcr Logstash austauschen<\/li>\n<li>Passwort admin f\u00fcr GUI \u00e4ndern<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Vorg\u00e4nger: HowTo: Install ELK-Stack unter Debian 9 Das folgende HowTo zeigt wie ein Amazon OpenSearch Cluster installiert werden kann mithilfe von Docker Containern bzw. Docker-Compose. OpenSearch ist ein auf dem ELK-Stack (Elasticsearch Version 7.10.2), der letzten &#8222;freien&#8220; Version, aufbauender Ersatz welcher das Sammeln, Auswerten, Anreichern und Darstellen von Logs bzw. Logeintr\u00e4gen erm\u00f6glicht. &nbsp; 1. Vorbereitung: [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-242","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/posts\/242","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/comments?post=242"}],"version-history":[{"count":13,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/posts\/242\/revisions"}],"predecessor-version":[{"id":263,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/posts\/242\/revisions\/263"}],"wp:attachment":[{"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/media?parent=242"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/categories?post=242"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.huerter.me\/index.php\/wp-json\/wp\/v2\/tags?post=242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}