{"id":1933,"date":"2018-07-08T22:23:20","date_gmt":"2018-07-08T20:23:20","guid":{"rendered":"http:\/\/van-maanen.com\/?p=1933"},"modified":"2018-07-08T22:23:20","modified_gmt":"2018-07-08T20:23:20","slug":"flume-used-for-logs","status":"publish","type":"post","link":"http:\/\/archief.van-maanen.com\/?p=1933","title":{"rendered":"Flume used for logs"},"content":{"rendered":"<p>In an earlier post, <a href=\"http:\/\/van-maanen.com\/index.php\/2016\/12\/18\/flume-sending-data-via-stream\/\">I showed how<\/a> one may send a stream via netcat to hdfs using flume. Another possibility is to set up a stream that is received by a server whereby the data are directly shown. The idea is that the client starts with telnet or netcat whereby data are sent. On the server side, the data are shown directly.<br \/>\nLet us set up the flume. The config file is:<\/p>\n<pre>\n# example.conf: A single-node Flume configuration\n# Name the components on this agent\na1.sources = r1\na1.sinks = k1\na1.channels = c1\n# Describe\/configure the source\na1.sources.r1.type = netcat\na1.sources.r1.bind = 192.168.2.25\na1.sources.r1.port = 12345\n# Describe the sink\na1.sinks.k1.type = logger\n# Use a channel which buffers events in memory\na1.channels.c1.type = memory\na1.channels.c1.capacity = 1000\na1.channels.c1.transactionCapacity = 100\n# Bind the source and sink to the channel\na1.sources.r1.channels = c1\na1.sinks.k1.channel = c1\n<\/pre>\n<p>Flume is then started with flume-ng agent &#8211;conf \/etc\/flume-ng\/conf &#8211;conf-file \/home\/training\/netcat_log.conf &#8211;name a1 -Dflume.root.logger=INFO,console.<\/p>\n<p>On the client, we start either telnet: telnet 192.168.2.25 12345. Or we start netcat with: netcat 192.168.2.25 12345. This looks like:<br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/van-maanen.com\/wp-content\/uploads\/2018\/07\/client.png\" alt=\"\" width=\"402\" height=\"72\" class=\"alignnone size-full wp-image-1938\" \/><br \/>\nOn the client, we started a stream with &#8220;tom&#8221;.<br \/>\nWhereas on the serverside, we see:<br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/van-maanen.com\/wp-content\/uploads\/2018\/07\/server.png\" alt=\"\" width=\"645\" height=\"112\" class=\"alignnone size-full wp-image-1939\" \/><br \/>\nOn the server, this &#8220;tom&#8221; is received.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In an earlier post, I showed how one may send a stream via netcat to hdfs using flume. Another possibility is to set up a stream that is received by a server whereby the data are directly shown. The idea is that the client starts with telnet or netcat whereby data are sent. On the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1933","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"_links":{"self":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1933","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1933"}],"version-history":[{"count":0,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1933\/revisions"}],"wp:attachment":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1933"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1933"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1933"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}