Search results for: “sparkling”
-
Processing JSON with Sparkling – #sparkling #spark #bigdata #clojure
While many developers crave the loveliness and simplicity of JSON data it can come with its own set of problems. This is very true when using tools like Spark for consuming data as you cannot guarantee that one line of the text file contains one complete block of a JSON object for processing. Resilient Distributed…
-
Getting clj-kafka consumer example to work. #clojure #kafka #streaming #data
As a data engineer and a software developer I’m spending a lot of time working out things over a number of different technologies. It might be Spark, Sparkling and S3 one day and Cassandra, Clojure and ElasticSearch the next. Historically I’ve been a huge advocate of RabbitMQ, I still am, but more and more I’m…
-
Apache SparkML’s Biggest Pain Point – #MachineLearning #Spark #SparkML #Data #BigData
One of the highlights of my job as a Data Engineer (I’m not a data scientist) is that I get to do some very cool stuff with text mining and all that data schizz. So to that end I’m using Apache Spark, Clojure and Sparkling a lot. With that in mind I do a lot…
-
Creating St Vincent Lyrics And Northern Ireland Assembly Questions With Markov Chains. #clojure @st_vincent #spark #opendata
The Story So Far In previous posts I’ve covered basically loading data in Spark (with Sparkling in Clojure) and doing some half funky stuff with it. That’s all very well and a good point for starting with, but it’s a touch limiting. Ultimately it’s very easy to get some numbers out, crack some percentages and…
-
NI Open Data – Mining Prescription Data Part 2. – #opendata #spark #clojure
The Story So Far…. You can read part 1 here. A few weeks ago I started on finding out which were the most popular items that a GP practice would prescribe. Once again I turned to Sparkling and Clojure to do the grunt work for me. The Practice List What I didn’t have at the…
-
NIAssembly Open Data – Part 2 – Sankey Diagrams #opendata #clojure #spark #sankey
In the first part of this walk through I showed you how to use the excellent NI Assembly open data platform to find out the frequency of departments members were asking questions to. A picture speaks a thousand words so they say, so it makes sense to attempt to visualise a diagram of the data…
-
NIAssembly Open Data – #opendata #ni #spark #clojure
Data From Stormont The Northern Ireland Assembly has opened up a fair chunk of data as a web service, returning results in either XML or JSON format. And from first plays with it, it’s rather well put together. What I’ve also learned is that the team listen, a small suggestion was implemented no sooner had…
-
About
Jason Bell has been working with customer centric data since 2002 and been involved in software development for over 30 years. A curious problem solver he has created many open source projects based on Hadoop, Machine Learning and Recommendation Systems. He has also contributed to Sparkling, the Spark wrapper for the Clojure language. During…