Finding potential #Ebola destinations with #OpenData

Air travel makes it easy for any one of us to hop from country to country, for most of the time that’s fine. When the H1N1 flu virus started to spread though it wasn’t much of a surprise how the far and wide reach of reports was caused by, travel. With data being readily available it was easy to see how it was moving, a good example of this was the “Just Landed In” post which used Twitter data, Metacarta and Processing to visualise the movements (


The Ebola virus has the potential to cause an awful lot of problems. So the question is where are the first connection points?

Gathering Route Data

The actual query to find out what flights goes where it easy enough and I’ll show you that in a moment. First though we need data, mainly airports, airlines and routes.

OpenFlights has all this data to hand in CSV format.

Airports: airports.dat

Airlines: airlines.dat

Routes: routes.dat

As they are CSV files they can be imported into a database such as MySQL easily. You can download the sql files here.

Crafting The Query

As it stands there are only four countries in West Africa that the Ebola outbreak is concentrated on: Guinea, Liberia, Sierra Leone and Nigeria.  The United States and Senegal have travel related cases but I’m not looking at those just yet, one step at a time.

The SQL query is basic but it gives us good info.

 a.longname as depairport, as depcountry, 
 a.iatacode as depcode, 
 b.longname as arrairport, 
 b.iatacode as arrcode, as arrcountry 
 FROM airports a, airports b, routes r, airlines al 
 WHERE r.depaircode 
 IN (SELECT iatacode FROM airports WHERE country="NIGERIA") 
 AND a.iatacode=r.depaircode
 AND b.iatacode=r.arraircode
 AND al.iatacode=r.airlinecode

This query will give us the departure and destination airports available for Nigeria, the country can be changed to anything.

Inspecting The Results

The output we get is the airline information, departure airport and IATA code and the destination airport.

"AF","Air France","Nnamdi Azikiwe Intl","Nigeria","ABV","Charles De Gaulle","CDG","France"
"AF","Air France","Nnamdi Azikiwe Intl","Nigeria","ABV","Port Harcourt Intl","PHC","Nigeria"
"AF","Air France","Murtala Muhammed","Nigeria","LOS","Charles De Gaulle","CDG","France"
"AF","Air France","Port Harcourt Intl","Nigeria","PHC","Nnamdi Azikiwe Intl","ABV","Nigeria"
"AT","Royal Air Maroc","Murtala Muhammed","Nigeria","LOS","Mohammed V Intl","CMN","Morocco"
"AZ","Alitalia - Compagnia Aerea Italiana Spa","Murtala Muhammed","Nigeria","LOS","Kotoka Intl","ACC","Ghana"
"BA","British Airways","Nnamdi Azikiwe Intl","Nigeria","ABV","Heathrow","LHR","United Kingdom"
"BA","British Airways","Murtala Muhammed","Nigeria","LOS","Heathrow","LHR","United Kingdom"
"DL","Delta Air Lines","Nnamdi Azikiwe Intl","Nigeria","ABV","Charles De Gaulle","CDG","France"
"DL","Delta Air Lines","Nnamdi Azikiwe Intl","Nigeria","ABV","Port Harcourt Intl","PHC","Nigeria"
"DL","Delta Air Lines","Port Harcourt Intl","Nigeria","PHC","Nnamdi Azikiwe Intl","ABV","Nigeria"
"E2","Eagle Atlantic Airlines","Murtala Muhammed","Nigeria","LOS","Kotoka Intl","ACC","Ghana"
"EK","Emirates","Murtala Muhammed","Nigeria","LOS","Dubai Intl","DXB","United Arab Emirates"
"ET","Ethiopian Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Bole Intl","ADD","Ethiopia"
"ET","Ethiopian Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Cadjehoun","COO","Benin"
"ET","Ethiopian Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Gnassingbe Eyadema Intl","LFW","Togo"
"ET","Ethiopian Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Ndjamena Hassan Djamous","NDJ","Chad"
"ET","Ethiopian Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Diori Hamani","NIM","Niger"
"ET","Ethiopian Airlines","Enugu","Nigeria","ENU","Bole Intl","ADD","Ethiopia"
"ET","Ethiopian Airlines","Murtala Muhammed","Nigeria","LOS","Abidjan Felix Houphouet Boigny Intl","ABJ","Cote d'Ivoire"
"ET","Ethiopian Airlines","Murtala Muhammed","Nigeria","LOS","Bole Intl","ADD","Ethiopia"
"ET","Ethiopian Airlines","Murtala Muhammed","Nigeria","LOS","Leon M Ba","LBV","Gabon"
"ET","Ethiopian Airlines","Murtala Muhammed","Nigeria","LOS","Gnassingbe Eyadema Intl","LFW","Togo"
"EY","Etihad Airways","Murtala Muhammed","Nigeria","LOS","Abu Dhabi Intl","AUH","United Arab Emirates"
"IB","Iberia","Murtala Muhammed","Nigeria","LOS","Barajas","MAD","Spain"
"KL","KLM-Royal Dutch Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Charles De Gaulle","CDG","France"
"KL","KLM-Royal Dutch Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Port Harcourt Intl","PHC","Nigeria"
"KL","KLM-Royal Dutch Airlines","Murtala Muhammed","Nigeria","LOS","Schiphol","AMS","Netherlands"
"KL","KLM-Royal Dutch Airlines","Port Harcourt Intl","Nigeria","PHC","Nnamdi Azikiwe Intl","ABV","Nigeria"
"KQ","Kenya Airways","Murtala Muhammed","Nigeria","LOS","Cadjehoun","COO","Benin"
"KQ","Kenya Airways","Murtala Muhammed","Nigeria","LOS","Jomo Kenyatta International","NBO","Kenya"
"LH","Lufthansa German Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Frankfurt Main","FRA","Germany"
"LH","Lufthansa German Airlines","Nnamdi Azikiwe Intl","Nigeria","ABV","Malabo","SSG","Equatorial Guinea"
"LH","Lufthansa German Airlines","Murtala Muhammed","Nigeria","LOS","Frankfurt Main","FRA","Germany"
"LH","Lufthansa German Airlines","Murtala Muhammed","Nigeria","LOS","Port Harcourt Intl","PHC","Nigeria"
"LH","Lufthansa German Airlines","Port Harcourt Intl","Nigeria","PHC","Murtala Muhammed","LOS","Nigeria"
"ME","Middle East Airlines","Mallam Aminu Intl","Nigeria","KAN","Rafic Hariri Intl","BEY","Lebanon"
"ME","Middle East Airlines","Murtala Muhammed","Nigeria","LOS","Abidjan Felix Houphouet Boigny Intl","ABJ","Cote d'Ivoire"
"ME","Middle East Airlines","Murtala Muhammed","Nigeria","LOS","Rafic Hariri Intl","BEY","Lebanon"
"MS","Egyptair","Nnamdi Azikiwe Intl","Nigeria","ABV","Cairo Intl","CAI","Egypt"
"MS","Egyptair","Mallam Aminu Intl","Nigeria","KAN","Cairo Intl","CAI","Egypt"
"MS","Egyptair","Murtala Muhammed","Nigeria","LOS","Cairo Intl","CAI","Egypt"
"QC","Camair-Co","Murtala Muhammed","Nigeria","LOS","Abidjan Felix Houphouet Boigny Intl","ABJ","Cote d'Ivoire"
"QC","Camair-Co","Murtala Muhammed","Nigeria","LOS","Douala","DLA","Cameroon"
"QR","Qatar Airways","Murtala Muhammed","Nigeria","LOS","Doha Intl","DOH","Qatar"
"RJ","Royal Jordanian","Murtala Muhammed","Nigeria","LOS","Kotoka Intl","ACC","Ghana"
"RJ","Royal Jordanian","Murtala Muhammed","Nigeria","LOS","Queen Alia Intl","AMM","Jordan"
"SA","South African Airways","Murtala Muhammed","Nigeria","LOS","Johannesburg Intl","JNB","South Africa"
"SD","Sudan Airways","Mallam Aminu Intl","Nigeria","KAN","Khartoum","KRT","Sudan"
"SN","Brussels Airlines","Murtala Muhammed","Nigeria","LOS","Frankfurt Main","FRA","Germany"
"SN","Brussels Airlines","Murtala Muhammed","Nigeria","LOS","Port Harcourt Intl","PHC","Nigeria"
"SN","Brussels Airlines","Port Harcourt Intl","Nigeria","PHC","Murtala Muhammed","LOS","Nigeria"
"TK","Turkish Airlines","Murtala Muhammed","Nigeria","LOS","Ataturk","IST","Turkey"
"UA","United Airlines","Murtala Muhammed","Nigeria","LOS","Frankfurt Main","FRA","Germany"
"UA","United Airlines","Murtala Muhammed","Nigeria","LOS","George Bush Intercontinental","IAH","United States"
"UA","United Airlines","Murtala Muhammed","Nigeria","LOS","Port Harcourt Intl","PHC","Nigeria"
"UA","United Airlines","Port Harcourt Intl","Nigeria","PHC","Murtala Muhammed","LOS","Nigeria"
"VS","Virgin Atlantic Airways","Murtala Muhammed","Nigeria","LOS","Heathrow","LHR","United Kingdom"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Kotoka Intl","ACC","Ghana"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Benin","BNI","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Calabar","CBQ","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Enugu","ENU","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Ibadan","IBA","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Ilorin","ILR","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Mallam Aminu Intl","KAN","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Maiduguri","MIU","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Port Harcourt Intl","PHC","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Imo Airport","QOW","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Warri Airport","QRW","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Sadiq Abubakar Iii Intl","SKO","Nigeria"
"W3","Arik Air","Nnamdi Azikiwe Intl","Nigeria","ABV","Yola","YOL","Nigeria"
"W3","Arik Air","Benin","Nigeria","BNI","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Benin","Nigeria","BNI","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Calabar","Nigeria","CBQ","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Calabar","Nigeria","CBQ","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Enugu","Nigeria","ENU","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Enugu","Nigeria","ENU","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Ibadan","Nigeria","IBA","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Ilorin","Nigeria","ILR","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Yakubu Gowon","Nigeria","JOS","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Kaduna","Nigeria","KAD","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Mallam Aminu Intl","Nigeria","KAN","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Mallam Aminu Intl","Nigeria","KAN","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Kotoka Intl","ACC","Ghana"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Benin","BNI","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Calabar","CBQ","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Leopold Sedar Senghor Intl","DKR","Senegal"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Douala","DLA","Cameroon"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Enugu","ENU","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Freetown Lungi","FNA","Sierra Leone"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","John F Kennedy Intl","JFK","United States"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Johannesburg Intl","JNB","South Africa"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Yakubu Gowon","JOS","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Kaduna","KAD","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Mallam Aminu Intl","KAN","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Luanda 4 De Fevereiro","LAD","Angola"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Heathrow","LHR","United Kingdom"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Port Harcourt Intl","PHC","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Imo Airport","QOW","Nigeria"
"W3","Arik Air","Murtala Muhammed","Nigeria","LOS","Warri Airport","QRW","Nigeria"
"W3","Arik Air","Maiduguri","Nigeria","MIU","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Port Harcourt Intl","Nigeria","PHC","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Port Harcourt Intl","Nigeria","PHC","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Imo Airport","Nigeria","QOW","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Imo Airport","Nigeria","QOW","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Warri Airport","Nigeria","QRW","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Warri Airport","Nigeria","QRW","Murtala Muhammed","LOS","Nigeria"
"W3","Arik Air","Sadiq Abubakar Iii Intl","Nigeria","SKO","Nnamdi Azikiwe Intl","ABV","Nigeria"
"W3","Arik Air","Yola","Nigeria","YOL","Nnamdi Azikiwe Intl","ABV","Nigeria"
"WB","Rwandair Express","Murtala Muhammed","Nigeria","LOS","Kotoka Intl","ACC","Ghana"
"WB","Rwandair Express","Murtala Muhammed","Nigeria","LOS","Kigali Intl","KGL","Rwanda"

So we know the first point airports, next time we’ll take things a bit further…..



Posted in Uncategorized

Seeing who’s going to @FNMashup via @GetInvitedTo with #UNIX / #CultureTech

GetInvitedTo has a handy “see who’s attending” list for certain events. As I’m not the mixing type I like to make sure that networking events are going to be worth my while before booking anything. The list on the page can get a bit long and it’s not in any order so, with a bit of copy/paste and one line of unix.


1. Grab Thy Data!

Go the event page (I’m using as the example). And select all the names right down to the bottom. Do a quick CTRL+C to save it to the clipboard.

2. Create a Text File!

In your favourite text editor paste the contents (CTRL+V) and this will dump all the names and also some blank lines. Don’t worry about that right now, we’ll sort that out in a minute. Save the file as names.txt

3. Run Thy Command!

Open up a terminal window (if you are running Windows then you’ll need something like Cygwin to run Unix commands). In the same directory as the names.txt file run the following command:

cat names.txt | egrep "\w+" |  sort -k2 | uniq

So we dump the contents of names.txt to the console, pipe that through egrep looking for all lines with words in them, sort by the second column (the surname in the majority of cases) and then show only the unique occurrences.

...Charlotte Bradberry Axten
Gavin Bradley
Karen Bradley
Lauren Bradley
Nikki Bradle
Reuben Bradley
Ryan Bradley
Tom Bradley
Ben Breslin
Trish Breslin
Rebecca Brown
Fiona Browne
Mura Browne
Tim Brundle
Anji Bull
Connor Bunting
David Bunting
Harry Burke
Brian Burns
Neil Burns
Derek Bustard
Declan Byrne
Tom Byrne....

Job done.

(There’s a whole section on manipulating data in my new book, “Machine Learning – Hand On for Developers and Technical Professionals“, published in  November. You can pre-order on Amazon if you wish.)

Posted in Uncategorized

Who is Your Support Network? #startups #sanity

It’s a simple question…. to be honest the answer is simple as well.


Startups are all consuming beasts that never leave you alone. Fine if there’s a team of 20 but a real challenge if it’s a team of one. If the service runs into trouble and customers are giving you a hard time then it’s the worst job in the world.

Yes this is where cofounders come in useful, you have a sound board. Some founders don’t want cofo’s, it makes life harder but it’s still not impossible to do. If that’s the route you want to go down though then you’ve got to make sure that you’ve got a support network in the industry as well as close family.

And a quick note, if you see a startup struggling, just a wee phone call to see if they’re okay makes all the difference. It’s not about being an entrepreneur or a mentor, it’s about being a human being.


Posted in Startups/Business

Ranum: My new #startup funding system.

The old adage runs true, you need money to make money. Now whether that’s someone else’s money or your own doesn’t really matter, it just alters how the communication and reporting is done.


I have to admit my system is not exactly new among entrepreneurs but I believe that I’ve managed to remove all that writing business plans, doing pitches and so on. I don’t even have to attend things I can just wheel the system out on a regular basis and see what transpires, for a  minimum investment of £2 a week (I can obviously increase that) I could stand to make a 1.6mx return on my investment or there abouts. I mean there’s no guarantees but there isn’t in much else either.

Here’s my startup formula:


I’m not just limited to six numbers, I could add more but I’d need to up my initial investment by £2 for every set of six numbers I pick. There’s a UK board meeting on a Wednesday and a Saturday (I know the board meets on a Saturday) and sometimes the bring a band on and do a quiz. It’s crazy.

The downside to this startup is my return varies from 1 in 56 where I get a 5x per week invested return to the unicorn finding probability of 1 in 13,983,816. If more than one investor has the same startup formula as me then the return is split amongst us.

The great thing is that these startup investment pools are all over the world with varying probabilities but they all seem to have that board meeting. Here’s the European board meeting :)

Posted in Startups/Business

Entrepreneurship: The Belfast Goldrush #Startups


Welcome To The Modern Goldrush

We have a modern gold rush on our hands folks, the Belfast startup scene is exploding! These things work in cycles and I can only compare it to the 2009/2010 explosion of barcamps, bizcamps and open coffees. People getting together and attempting to make stuff happen. There’s gold in them there hills and if you listen to the right people and read the right tech sites and books then you have as a good a chance as any of making it.

Let’s roll back a second though. There’s been a lot of pedalling of the startup way, “anyone with an idea can make it”, and to be fair there are those who have. So entrepreneurs are the new gold prospectors and there be the problem….

1848–1855 – Who Made The Money?

The question is this, in the California Gold Rush who made the money? The guy who sold the shovel. The prospectors themselves made less than the folk who were supplying the services to the prospectors. After expenses prospectors made a modest profit but the people who sold (among other things): retail, shipping, entertainment, lodging, transportation, boarding houses, food preparation, sewing, laundry, saloons, brothels and gambling – they made the money.

The prospectors fuelled an entire business ecosystem and they all did very well out of it, well everyone apart from the prospectors, only a few actually “made it”.

2014 – Who’s Making The Money?

Look at things today, for the startup just entering we’ve got the likes of mobile app development, web design houses, graphic designers, lawyers, accountants, consultants, public relations, marketing companies, social media gurus and the rest of it. The modern day prospector is fuelling a an entire business ecosystem. And you know what that’s fine.

We attend prospector events with more prospectors and we trade our war stories over free beer and food. The only thing it can be compared to is a wake, it’s the only other time when free alcohol and food seem to attract volumes of numbers. A tip for you, if you know over 25% of the people in the room then don’t bother going. We’ll be invited to pay for weekends where distilled knowledge is passed to us regardless of who’s telling us have actually made a prospectors payback or not (let’s be honest, they probably haven’t).

The Realisation

The reality is stark reader: While I concentrate on Belfast as that’s where I live (sort of) it could apply to anywhere in the world that pushing the gold rush as a way to keep the minds of good busy. For some who were there before over the last two cycles we’ve become weary and know that the game’s up. The same mantra reworded doesn’t wash anymore.

The old American Dream … was the dream of the Puritans, of Benjamin Franklin’s “Poor Richard”… of men and women content to accumulate their modest fortunes a little at a time, year by year by year. The new dream was the dream of instant wealth, won in a twinkling by audacity and good luck.

Luck is where the balance of success and failure lies. You have to create your own luck.



Posted in Startups/Business

An AppStore for #Hadoop ?

It’s been staring me in the face for a while, all the training, advising and talking about this Hadoop thing. It needs an AppStore, a seamless platform for one click installing and running of Hadoop applications within the framework and cluster.

Up until now most MapReduce work has been hand coded by in house development teams working to a specific requirement. Hadoop2 and it’s Yarn component extends on that and makes applications easier to distribute across the cluster, like I said in yesterday’s post about deploying point of sale software across kiosks.

The issue is that Hadoop takes an amount of skill to get things running, thinking of getting data in and running scripts at so on. We’re a way off from a one touch solution for app and data deployment and processing.

It’s coming, I’m sure it is. The next two years will be very interesting.

MapR got the ball rolling with an application gallery, not an AppStore par se but a great start.

Posted in Hadoop, Java, Sentiment Analysis, Startups/Business

Could #Hadoop Aid The Point Of Sale Deployment Problem? #POS

Point Of Sale (POS) is one of the hotly contested areas of startups. While many feel that point of sale should shift from the desktop/kiosk based Windows machine to something a bit more funky (iOS and Android, think AirPOS) there is a legacy of hundreds of thousands of kiosks out there in the field.

One of the major headaches for POS vendors is updating the software on the kiosk. The majority is build on Windows, run locally and using local storage. Only recently the advent of broadband connectivity means that stock updates and deployments can happen as required.

Updates are tiresome especially when you have large numbers of kiosks deployed within a store.

With web based POS this isn’t really much of an issue as the software is essentially a SaaS implementation. iOS and Android update their own operating systems via their own ecosystems primarily but this then puts emphasis on the user to update their own and not the vendor to monitor measure deployments across the site. This can become an issue from a customer support point of view when there are multiple versions of the software in use across the retail field.

Hadoop 2 and YARN

Most people think of Hadoop as the BigData thing, crunching volumes of data and spitting out the answers. For Hadoop 1.x this was true, with the advent of Hadoop 2.x it’s not so much the case now.

The differences between Hadoop version 1 and version 2 is vast and it all boils down to the YARN element performing resource management control. This means that Hadoop is no longer about crunch data in the MapReduce paradigm but more a cluster control tool where the YARN element (Yet Another Resource Negotiator) can play a key role in deploying software across the whole cluster.

One Master, Many Slaves

The diagram below shows six POS kiosks in operation. Traditionally updates would happen manually or via some remote software like TeamViewer, remotely copying the files in and restarting the POS application on the kiosk. Introducing Hadoop 2.x into the mix changes the landscape slightly. The Hadoop master is on another server, this could be within the same data centre, or within the control of the POS company for example. The POS kiosks now become slaves to the Hadoop master.




In Hadoop terms every POS kiosk becomes a Node with a Container. With YARN we can make a deployment to all the kiosks at the same time. Ultimately YARN isn’t concerned about what it is running, just that it can be published to the required containers across the available nodes.  The software required for the POS would be stored on the master’s HDFS or a the master local filesystem. When the YARN client is run it’s then copied across to the addressable nodes.

This means the master deployment of the updated software only need happen once. Then it’s up to YARN to distribute the software to the other kiosks and get it started. The beauty of this means that the output and error logs are stored on the master server so they are all in one place.

Taking It Further

As YARN acts as our resource manager it’s also easier to carry data functions, heartbeat monitoring (no one likes a POS kiosk that’s down, you can’t sell) and end of day functions with the data stored on the master. From there it would be a fairly trivial task to perform some batch Hadoop jobs on the day’s trading data. Insight comes quick then the nodes are connected together.



Posted in Android Development, Data and Statistics, Hadoop, iOS Development, Retail and Loyalty, Startups/Business

Get every new post delivered to your Inbox.

Join 975 other followers