Missing Spark History logs

Problem We have multiple Spark jobs running during ETL phase of our application. Some of them are really large and complicated jobs which takes hours to complete. One of such job, which uses GraphX & Pregal libraries, started failing after running for sometime. We use Spark History server to debug any jobs after they are… Read More Missing Spark History logs

Dynamic fields to Json in Scala

We have faced a situation where we needed to create Json representation for a dynamic set of fields & their values which represented by a Map[String, String]. Example: Scala class case class Person(name: String, age: String, customFields: Map[String,String]) Default Json representation of above class will be: { “name”: “anil”, “age”: “30”, “customFields”: { “field1”: “value1”,… Read More Dynamic fields to Json in Scala

Golang functions & methods simplified

If you are exploring Golang (aka Go) you must have faced some difficulty in understanding the method concepts, especially if you are coming from a Java background which is more statically typed. It gets more complicated when you get into methods with pointers. Tour of Go gives step by step guide with example programs and an environment to edit… Read More Golang functions & methods simplified