{"id":13210,"date":"2015-02-09T09:05:14","date_gmt":"2015-02-09T08:05:14","guid":{"rendered":"https:\/\/blog.trifork.com\/?p=13210"},"modified":"2015-02-09T09:05:14","modified_gmt":"2015-02-09T08:05:14","slug":"anwb-big-data-proof-of-concept","status":"publish","type":"post","link":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/","title":{"rendered":"ANWB Big data Proof of Concept"},"content":{"rendered":"

At the ANWB<\/a>\u00a0people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW<\/a>) provides an enormous database of both real-time and historic traffic data.<\/p>\n

This data comes from many different sources and is available as open data. Wouldn\u2019t it be great if the ANWB could\u00a0use this open data to provide more accurate traffic information, either in real-time or as a prediction for a certain period? In a proof of concept we have collected and analysed the real-time traffic information to calculate the traffic intensity on the roads\u00a0using elasticsearch. We also used weather information to see if the weather has influence on the need of roadside assistance.<\/p>\n

<\/p>\n

Data<\/h3>\n

The data is available as XML files on a FTP server and is refreshed every minute.\u00a0 We are processing two files, one with measurements and one with traffic speed information. The file with measurements contains 25494 records and the file with traffic speed information contains 15767 records. The traffic speed records are combined with the measurements. So this means that we index 25494 documents every minute.<\/p>\n

Because the data consists a lot of different measure points (like cameras or lasers) we only used the measure points that are bedded in the tarmac.\u00a0<\/span> A road can be divided in multiple lanes and each of those lanes\u00a0can have a measure point.\u00a0 This image displays the different measure points that are available.<\/p>\n

\"measurepoints\"<\/a><\/p>\n

Each measure point contains a lot of information such as the location, type of vehicle that passed, type of carriage way, direction but also how many vehicles passed that point. In our proof of concept we have used the measured speed and vehicle flow rate for a specific location or route to display the data for a given period. If a road contains a measure location\u00a0with multiple measure points on different lanes, we summed\u00a0the values of all lanes to keep the data\u00a0a little bit simple.<\/p>\n

\"example-measure-locations\"<\/a><\/p>\n

In the image above you see a graph that displays two road directions (blue and green lines) and their corresponding traffic speed (orange and red lines). On the map you can see all the measure points that have been used to calculate the data. With this data you could possibly get a real-time overview of routes where the traffic speed becomes less and the number of cars passing a measure point will decrease. These locations could be an indication that something is happening and could be the start of a traffic jam.<\/span><\/p>\n

Making better predictions for roadside assistance<\/h2>\n

The number of cars that need to be available for roadside assistance are predicted each year based on historic information. On a regular base\u00a0the ANWB checks if the predictions\u00a0needs to be adapted by checking a number of different possible causes that could increase the traffic like events or sudden weather changes. To make this forecast even better we have added\u00a0the daily temperature (which is also available as open data provided by KNMI<\/a>). With the temperature\u00a0and maybe the precipitation in the future they might be able to see if there is a correlation between the numbers of assistance cars that were needed and the weather.<\/p>\n

\"roadside-assistance\"<\/a><\/p>\n

Here you can see the number of cases that needed assistance (green bar), long term forecast (orange bar), short term forecast (purple bar) and the temperature (blue line).<\/p>\n

Data enrichment<\/h2>\n

In the image above you see a list of regions and areas. Because we want to be able to filter the data\u00a0on a specific region, we needed a way to store the region and area with the measurement document. Elasticsearch provides a way to match an document (which is a measurement in our case) against stored queries. This is called the percolator<\/a>.<\/p>\n

As said before, each measure point contains the location. The location consists of a latitude and longitude. We created a separate percolator type <\/span>that contains all regions and areas.<\/p>\n

So before we index a measurement document we percolate the index with the location to see which region and area matches and store these values with the measurement document. This allows us to execute a query that filters on a region and provides us correct\u00a0data.<\/p>\n

To be continued..<\/h2>\n

This project was only a proof of concept, but provided the ANWB enough input to talk about.\u00a0With open data and the power of elasticsearch to search in near real-time we created a simple application, but this could lead to something very useful. Hopefully this project will continue and gives us better traffic information in the future!<\/p>\n","protected":false},"excerpt":{"rendered":"

At the ANWB\u00a0people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW) provides an enormous database of both real-time and historic traffic data. This data comes from many different sources and is available as open […]<\/p>\n","protected":false},"author":102,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[15,31,65,10],"tags":[72,395,61,11,401],"class_list":["post-13210","post","type-post","status-publish","format-standard","hentry","category-enterprise-search","category-java","category-big_data_search","category-development","tag-big-data","tag-c3js","tag-elasticsearch","tag-java","tag-open-data"],"acf":[],"yoast_head":"\nANWB Big data Proof of Concept - Trifork Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ANWB Big data Proof of Concept - Trifork Blog\" \/>\n<meta property=\"og:description\" content=\"At the ANWB\u00a0people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW) provides an enormous database of both real-time and historic traffic data. This data comes from many different sources and is available as open […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/\" \/>\n<meta property=\"og:site_name\" content=\"Trifork Blog\" \/>\n<meta property=\"article:published_time\" content=\"2015-02-09T08:05:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png\" \/>\n<meta name=\"author\" content=\"Roberto van der Linden\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Roberto van der Linden\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/\",\"url\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/\",\"name\":\"ANWB Big data Proof of Concept - Trifork Blog\",\"isPartOf\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png\",\"datePublished\":\"2015-02-09T08:05:14+00:00\",\"author\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/037974cf3e24a7b09a93770b190d6e35\"},\"breadcrumb\":{\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage\",\"url\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png\",\"contentUrl\":\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/trifork.nl\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ANWB Big data Proof of Concept\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/trifork.nl\/blog\/#website\",\"url\":\"https:\/\/trifork.nl\/blog\/\",\"name\":\"Trifork Blog\",\"description\":\"Keep updated on the technical solutions Trifork is working on!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/trifork.nl\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/037974cf3e24a7b09a93770b190d6e35\",\"name\":\"Roberto van der Linden\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/afe49faf7ef8dd3753baefb334568b10?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/afe49faf7ef8dd3753baefb334568b10?s=96&d=mm&r=g\",\"caption\":\"Roberto van der Linden\"},\"url\":\"https:\/\/trifork.nl\/blog\/author\/roberto\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"ANWB Big data Proof of Concept - Trifork Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/","og_locale":"en_US","og_type":"article","og_title":"ANWB Big data Proof of Concept - Trifork Blog","og_description":"At the ANWB\u00a0people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW) provides an enormous database of both real-time and historic traffic data. This data comes from many different sources and is available as open […]","og_url":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/","og_site_name":"Trifork Blog","article_published_time":"2015-02-09T08:05:14+00:00","og_image":[{"url":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png","type":"","width":"","height":""}],"author":"Roberto van der Linden","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Roberto van der Linden","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/","url":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/","name":"ANWB Big data Proof of Concept - Trifork Blog","isPartOf":{"@id":"https:\/\/trifork.nl\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage"},"image":{"@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage"},"thumbnailUrl":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png","datePublished":"2015-02-09T08:05:14+00:00","author":{"@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/037974cf3e24a7b09a93770b190d6e35"},"breadcrumb":{"@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#primaryimage","url":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png","contentUrl":"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2015\/01\/measurepoints-1024x424.png"},{"@type":"BreadcrumbList","@id":"https:\/\/trifork.nl\/blog\/anwb-big-data-proof-of-concept\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/trifork.nl\/blog\/"},{"@type":"ListItem","position":2,"name":"ANWB Big data Proof of Concept"}]},{"@type":"WebSite","@id":"https:\/\/trifork.nl\/blog\/#website","url":"https:\/\/trifork.nl\/blog\/","name":"Trifork Blog","description":"Keep updated on the technical solutions Trifork is working on!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/trifork.nl\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/037974cf3e24a7b09a93770b190d6e35","name":"Roberto van der Linden","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/afe49faf7ef8dd3753baefb334568b10?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/afe49faf7ef8dd3753baefb334568b10?s=96&d=mm&r=g","caption":"Roberto van der Linden"},"url":"https:\/\/trifork.nl\/blog\/author\/roberto\/"}]}},"_links":{"self":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/13210","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/users\/102"}],"replies":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/comments?post=13210"}],"version-history":[{"count":0,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/13210\/revisions"}],"wp:attachment":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media?parent=13210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/categories?post=13210"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/tags?post=13210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}