curl 'localhost:9200\/_cat\/indices?v'<\/code><\/p>\nThere are a lot of ways in which the JSON file can be imported from that URL and put into elasticsearch. For example, you can make a script like this one:<\/p>\n
\nimport urllib, json\nimport elasticsearch\nimport time<\/code>\n\n#data = json.loads(response.read())\n\nf = open("ttnsample.json","r")\n#data = f.read()\n\nes = elasticsearch.Elasticsearch()\n\ndef lemain():\n\nwhile True:\nurl = "http:\/\/thethingsnetwork.org\/api\/v0\/nodes\/02D1DC09\/"\nresponse = urllib.urlopen(url)\njsonobject = json.loads(response.read())\nfor x in range (0,50):\nes.index(index = 'lora_messages', doc_type='node_message',body = jsonobject[x])\ntime.sleep(15)\n\nif __name__ == "__main__":\nlemain();\n<\/pre>\nAs in the previous steps, make sure to use your own devaddr in the url.<\/p>\n
Visualising the data in Kibana<\/h2>\n
The last step is to make the data readable and easy to understand. Make sure both Kibana and elasticsearch are running. Open the browser and go to 127.0.0.1:5601<\/code>. Add the new index LoRa_message*<\/code>. Kibana should auto detect the time filed. Below you can see how should the node data look like:<\/p>\n![\"kibana_lora_sensor_data\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/kibana_lora_sensor_data-1024x667.png\")
<\/p>\n
Each message contains the following fields :<\/p>\n
\n- \u201cnode_eui\u201d – the node’s devaddr<\/li>\n
- \u201cdatarate\u201d<\/li>\n
- \u201cdata_plain\u201d – the data sent from the node<\/li>\n
- \u201cgateway_eui\u201d – the unique mac address of the gateway<\/li>\n
- \u201cdata_raw\u201d – data before decryption<\/li>\n
- \u201csnr\u201d – Signal to noise ratio (it indicates the quality of the signal)<\/li>\n
- \u201ctime\u201d – the time at which the packet was received<\/li>\n
- \u201crssi\u201d – Received signal strength indication<\/li>\n
- \u201cdata_json\u201d – the same data, but in a number format.<\/li>\n
- \u201cdata\u201d – data in chars<\/li>\n
- \u201cfrequency\u201d<\/li>\n<\/ul>\n
Kibana allows to read this data in various ways. For this example I created three simple line charts that would show the sensor data, the signal to noise ration and RSSI over time.<\/p>\n
![\"kibana_lora_sensor_data_2\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/kibana_lora_sensor_data_2-1024x551.png\")
<\/p>\n
In here you can see the data from a sound sensor in the Trifork office in Eindhoven over an hour, together with the signal to noise ratio and RSSI. The dashboard can be customized in any way possible, and multiple nodes can be added.<\/p>\n
Setting up a personal LoRaWAN network without the open TTN cloud<\/h2>\n
The main benefit of using the free TTN cloud server is that you don’t need a personal server with static IP address that has to run 24\/7. On the other hand, by using their server, the data becomes public, and can be accessed by anyone. In addition, anyone can use the same devaddr for the nodes and fill your buffer with unnecessary data.
\nIn order to redirect your data from the gateway to a personal server you can use Logstash, which can parse and filter the messages.\u00a0The network architecture in this case would look like this:<\/p>\n
![\"lorawan_private_gateway\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/lorawan_private_gateway-1024x670.png\")
<\/p>\n
Setting up the\u00a0gateway<\/h3>\n
In order to configure the gateway, you have to access the local_conf.json<\/code> file which can be fond in the \/opt\/ttn-gateway\/bin<\/code> folder. You have to change the parameters under the index \u201cserver\u201d to the IP address of your server’s address and the UTP ports used for sending the data to\/from the gateway, as in the example below:<\/p>\n![\"lora_gateway_configuration\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/lora_gateway_configuration-1024x181.png\")
<\/p>\n
And the run the .\/start.sh<\/code> from the same folder. Now the gateway\u00a0is sending JSON\u00a0packages to your server.<\/p>\nSetting up Logstash<\/h3>\n
The next step is to capture the data from the UDP port and add it to Elasticsearch. In the logstash-x.x.x<\/code> folder, create a config file, lora.conf<\/code> for example. In this file you have to specify where does the data come from, how should it be filtered, and where it should be sent to.\u00a0The following example is the minimal setup for simply redirecting all the data from the UDP port to elasticsearch:<\/p>\n![\"logstash_lora_configuration\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/logstash_lora_configuration-300x167.png\")
<\/p>\n
As you can see there is no filter applied, which means that\u00a0all the information will be redirected. The gateway sends json packages about the gateway’s\u00a0status, periodical acknowledgements and the data received from the nodes.\u00a0More information can be found in the PROTOCOL.TXT file which can be found in the Rpi’s folder \/opt\/ttn_gateway\/packet_forwarder<\/code>.<\/p>\nIn order to get only the node’s data, you need to apply a filter in logstash, or choose a specific search index in Kibana. In order to visualize the data in kibana, simply choose logstash*<\/code> as the search index.<\/p>\nThe last step is to decode the node data. In the previous setup, the data was decoded by TTN framework, but in this case you have to decode it using a base64 decoder. The encryption keys can be set up in the Arduino code by changing the variables netwskey<\/code> and appskey<\/code>. The decryption can be done with logstash using the cipher plugin.<\/p>\n","protected":false},"excerpt":{"rendered":"Introduction to LoRaWAN and ELK Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc \u2026 ? LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that […]<\/p>\n","protected":false},"author":55,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[15,337,111,65],"tags":[61,421,92,93,422],"class_list":["post-14724","post","type-post","status-publish","format-standard","hentry","category-enterprise-search","category-from-the-trenches","category-internet-of-things","category-big_data_search","tag-elasticsearch","tag-elk","tag-kibana","tag-logstash","tag-lorawan"],"acf":[],"yoast_head":"\n
Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog\" \/>\n<meta property=\"og:description\" content=\"Introduction to LoRaWAN and ELK Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc \u2026 ? LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/\" \/>\n<meta property=\"og:site_name\" content=\"Trifork Blog\" \/>\n<meta property=\"article:published_time\" content=\"2016-05-20T07:48:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png\" \/>\n<meta name=\"author\" content=\"Ion Iuncu\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ion Iuncu\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/\",\"url\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/\",\"name\":\"Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog\",\"isPartOf\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png\",\"datePublished\":\"2016-05-20T07:48:05+00:00\",\"author\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/ef71e30c92c8ecb6e8eb14049ea69f95\"},\"breadcrumb\":{\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage\",\"url\":\"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png\",\"contentUrl\":\"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/trifork.nl\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Collecting data from a private LoRaWAN sensor network into Elastic\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/trifork.nl\/blog\/#website\",\"url\":\"https:\/\/trifork.nl\/blog\/\",\"name\":\"Trifork Blog\",\"description\":\"Keep updated on the technical solutions Trifork is working on!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/trifork.nl\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/ef71e30c92c8ecb6e8eb14049ea69f95\",\"name\":\"Ion Iuncu\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c92fb406f9e4f40b4fc90c835590bd7b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c92fb406f9e4f40b4fc90c835590bd7b?s=96&d=mm&r=g\",\"caption\":\"Ion Iuncu\"},\"sameAs\":[\"http:\/\/www.trifork.com\"],\"url\":\"https:\/\/trifork.nl\/blog\/author\/ion-iuncutrifork-nl\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/","og_locale":"en_US","og_type":"article","og_title":"Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog","og_description":"Introduction to LoRaWAN and ELK Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc \u2026 ? LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that […]","og_url":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/","og_site_name":"Trifork Blog","article_published_time":"2016-05-20T07:48:05+00:00","og_image":[{"url":"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png","type":"","width":"","height":""}],"author":"Ion Iuncu","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ion Iuncu","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/","url":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/","name":"Collecting data from a private LoRaWAN sensor network into Elastic - Trifork Blog","isPartOf":{"@id":"https:\/\/trifork.nl\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage"},"image":{"@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage"},"thumbnailUrl":"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png","datePublished":"2016-05-20T07:48:05+00:00","author":{"@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/ef71e30c92c8ecb6e8eb14049ea69f95"},"breadcrumb":{"@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#primaryimage","url":"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png","contentUrl":"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png"},{"@type":"BreadcrumbList","@id":"https:\/\/trifork.nl\/blog\/collecting-data-from-a-private-lorawan-sensor-network-into-elastic\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/trifork.nl\/blog\/"},{"@type":"ListItem","position":2,"name":"Collecting data from a private LoRaWAN sensor network into Elastic"}]},{"@type":"WebSite","@id":"https:\/\/trifork.nl\/blog\/#website","url":"https:\/\/trifork.nl\/blog\/","name":"Trifork Blog","description":"Keep updated on the technical solutions Trifork is working on!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/trifork.nl\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/ef71e30c92c8ecb6e8eb14049ea69f95","name":"Ion Iuncu","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c92fb406f9e4f40b4fc90c835590bd7b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c92fb406f9e4f40b4fc90c835590bd7b?s=96&d=mm&r=g","caption":"Ion Iuncu"},"sameAs":["http:\/\/www.trifork.com"],"url":"https:\/\/trifork.nl\/blog\/author\/ion-iuncutrifork-nl\/"}]}},"_links":{"self":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/14724","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/users\/55"}],"replies":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/comments?post=14724"}],"version-history":[{"count":0,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/14724\/revisions"}],"wp:attachment":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media?parent=14724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/categories?post=14724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/tags?post=14724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![](\"https:\/\/trifork.com\/wp-content\/uploads\/2016\/07\/LoRaWAN-1-300x201.png\")
<\/p>\n![\"LoRa_TTN_network_architecture\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/LoRa_TTN_network_architecture-1024x670.png\")
<\/p>\n![\"ic880a_gateway_wiring_raspberry_pi\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/ic880a_gateway_wiring_raspberry_pi.png\")
<\/th>\n<\/tr>\n![\"lora_gateway_util_tx_test_succesful\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/lora_gateway_util_tx_test_succesful-1024x407.png\")
<\/p>\n![\"the_thing_network_output\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2016\/05\/the_thing_network_output-1024x467.png\")
<\/p>\n