{"id":1887,"date":"2018-05-15T23:02:13","date_gmt":"2018-05-15T21:02:13","guid":{"rendered":"http:\/\/van-maanen.com\/?p=1887"},"modified":"2018-05-15T23:02:13","modified_gmt":"2018-05-15T21:02:13","slug":"sending-avro-file-via-http","status":"publish","type":"post","link":"http:\/\/archief.van-maanen.com\/?p=1887","title":{"rendered":"Sending Avro file via HTTP"},"content":{"rendered":"

It is possible to send an AVRO file via HTTP. The idea is that one sets up a server process. Once the server process runs, a client call is made.
\nI found a neat scheme how such process works.
\n\"\"
\nWe see on the server side, a socket must be set up. This set-up must also be undertaken with an indication which type of network is utilized. The most common network in the IP4 system: 4 numbers, indicating the position in a network. This is indicated with “AF_INET”. Once the socket is set up, it must be linked to an (internal) IP address. This link is indicated with the port. After that the server can be instructed to start listening to a particular port.
\nThe system will be instructed to receive AVRO files. Therefore the avro modules are imported.<\/p>\n

One is then able to see how the server reacts upon the client call. We use Python3 to work this out. The server process runs as;<\/p>\n

import avro.datafile\nimport avro.io\nimport io\nimport socket\n\ndef handle_client(connection, address):\n    data = connection.recv(1024)\n    message_buf = io.BytesIO(data)\n    reader = avro.datafile.DataFileReader(message_buf, avro.io.DatumReader())\n    for thing in reader:\n        print(thing)\n    reader.close()\n\ndef main():\n    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n    sock.bind(('van-maanen.com\/', 54321))\n    sock.listen(10)\n\n    while True:\n        conn, addr = sock.accept()\n        handle_client(conn, addr)\n        conn.close()\n\nif __name__ == '__main__':\n    main()\n<\/pre>\n
A more  extended programme can be seen here:  <\/a><\/p>\n
The client programme then looks like:<\/p>\n
\nimport io\nimport json\nimport socket\nimport avro.datafile\nimport avro.schema\nimport avro.io\nimport avro.ipc\n\nSCHEMA = avro.schema.Parse(json.dumps({\n \"namespace\"    : \"example.avro\",\n \"type\"         : \"record\",\n \"name\"         : \"User\",\n \"fields\"       : [\n     {\"name\": \"name\"            , \"type\": \"string\"},\n     {\"name\": \"favorite_number\" , \"type\": [\"int\", \"null\"]},\n     {\"name\": \"favorite_color\"  , \"type\": [\"string\", \"null\"]}\n ]\n}))\n\ndef send_message(connection, message):\n    buf = io.BytesIO()\n    writer = avro.datafile.DataFileWriter(buf, avro.io.DatumWriter(), SCHEMA)\n    writer.append(message)\n    writer.flush()\n    buf.seek(0)\n    data = buf.read()\n    connection.send(data)\n\ndef main():\n    connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n    connection.connect(('192.168.2.25', 54321))\n    send_message(connection, {'name': 'Eli', 'favorite_number': 42, 'favorite_color': 'black'})\n\nif __name__ == '__main__':\n    main()\n<\/pre>\n
A more  extended programme can be seen here:  <\/a><\/p>\n
Another  elaborate programme can be seen here:  <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
It is possible to send an AVRO file via HTTP. The idea is that one sets up a server process. Once the server process runs, a client call is made. I found a neat scheme how such process works. We see on the server side, a socket must be set up. This set-up must also […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1887","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"_links":{"self":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1887"}],"version-history":[{"count":0,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1887\/revisions"}],"wp:attachment":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1887"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}