For the purposes of this post I define a data analyst as someone who uses tools like Excel and SQL to interrogate data to produce reports, plots, recommendations but crucially doesn’t deliver code. If you work in online retail and create an algorithm recommending tiaras for pets  I call you a data scientist. If you query a database and discover that chihuahua owners prefer pink tiaras and share this finding with the advertising team  you are a data analyst.
Let me get one thing out of the way first: this post is not bashing analysts. Of course data analyst’s work is useful and rewarding in its own right. And there is more demand for it (under various names) than there is for data science. But that is beside the point. The point is that a lot of people will tell you that taking a job as a data analyst is a good way to prepare for data science and that is a lie. In terms of transferable skills you may as well be working as a dentist.
A data analyst is not a larval stage of a data scientist. They are completely different species.
Data Analyst  Data Scientist 

Sits with the business  Sits with engineers (but talks to the business) 
Produces reports, presentations  Produces software 
Interestingly, the part about sitting in a different place (often a different floor or a different building!) is the bigger obstacle to moving into data science. Independent of having or not having the right skills, a data analyst can’t just up and start doing data science because they don’t have the physical means to do it! They don’t have:
While those things can be eventually gotten hold of with enough perseverance, there are other deficits that even harder to make up for:
This should be obvious to anyone who has ever worked in a big company. You don’t simply walk into an software team and start making changes. It sometimes takes months of training for a new developer on the team make first real contribution. For an outsider from a different business unit to do it remotely is unheard of.
As a data analyst:
You will on the other hand do:
So that doesn’t sound all bad, right? Wrong.
I think a case can be made that the little technical work a data analysts do actually does more harm than good to their data science education. A data scientist and an analyst may be using some of the same tools, but what they do with them is very, very different.
Data analyst’s code  Data Scientist's code 

Manually operated sequence of scripts, clicking through GUIs etc.  Fully automated pipelines 
Code that only you will ever see  Code that will be used and maintained by other people 
Oneoff, throwaway scripts  Code that is a part of an live app or a scheduled pipeline 
Code tweaked until it runs this one time  Code optimised for performance, maintainability and reusability 
Doing things a certain way may make sense from a data analyst’s perspective, but the needs of data science are different. When former analysts are thrown into data science projects and start applying the patterns they have developed through the years, the results are not pretty.
Let me illustrate with an example which I promise is not cherrypicked and a fairly typical in my experience.
I joined a project led by analyststurneddata scientists. We were building prototype of a pipeline doing some machine learning on the client’s data and displaying pretty plots. One of my first questions when I joined was: how are you getting your data from the client? (we needed a new batch of data at that time). The answer was:
Needless to say, this workflow made it impossible to get anything done whatsoever. To even run the pipeline again on fresher data would take weeks (when it should be seconds) and the results were junk anyway because the technologies they used forced them to only use 1% of available data.
On top of that, every single script in the pipeline was extremely hacky and brittle  and here’s why:
When faced with a task, an analyst would start writing code. If it doesn’t work at first, they add to it and tweak it until it does. As soon as a result is produced (a csv file usually), they move on to the next step. No effort is made to ensure reproducibility, reusability, maintainability, scalability. The piece of code gets you from A to B and that is that. A script made this way is full of hardcoded database passwords, paths to local directories, magic constants and untested assumptions about the input data. It resembles a lategame Jenga tower  weird and misshapen, with many blocks missing and others sticking out in weird directions. It is standing for now but you know that it will come crashing down if you as much as touch it.
The tragic part is that none of the people involved in this mess were dumb. No, they were smart and experienced, just not the right kind of experienced. This spaghetti of manual steps, hacky scripts and big data on old laptops is not the result of not enough cleverness. Way too much cleverness if anything. It’s the result of intelligent people with no experience in making software realising too late that they’re out of their depth.
If only my colleagues were completely nontechnical  never having written a SAS or SQL script in their lives  they would have had to hire an engineer to do the coding and they themselves would have focused on preparing the spec. This kind of arrangement is not ideal but I guarantee that the result would have been much better. This is why I believe that the data analyst’s experience is not just useless but actively harmful to data science.
Ultimately though the fault doesn’t lie with the analysts but with the management for mismatching people and tasks. It’s time managers understood that:
In case I wasn’t clear about this: I am emphatically not saying that analysts can’t learn proper software engineering and data science. If miners can do it, so can analysts. It’s just that an analyst’s experience makes it harder for them (and their managers!) to realise that they are missing something and easier to get by without learning a thing.
If you’re an analyst and want to switch to data science (And I’m not saying that you should! The world needs analysts too!) I recommend that you forget everything you have learned about coding and start over, like the miners.
If you’re a grad considering a data analyst role as training for data science I strongly recommend that you find a junior software developer job instead. If you’re lucky, you may get to do some machine learning and graduate into fullon data science. But even if not, practically everything you learn in an entrylevel engineering position will make you a better data scientist when you finally become one.
]]>A popular meme places data science at the intersection of hacking, statistics and domain knowledge. It isn’t exactly untrue but it may give an aspiring data scientist the mistaken impression that those three areas are equally important. They’re not.
I’m leaving domain knowledge out of this discussion because, while it’s absolutely necessary to have it to get anything done at all, it usually doesn’t have to be very deep and you’re almost always expected to pick it up on the job.
First of all, hacking is something that we do every day while we can go months or years without touching any statistics. Of course, statistics and probability are baked into much of the software we use but we no more need to think about them daily than a pilot needs to think about the equations of aerodynamics.
Secondly, on those rare occasions when you do come up with some brilliant probabilistic model or business insight, it will still have to be implemented as a piece of software before it creates any value. And make no mistake  it will be implemented by you or not at all. A theoretical data scientist who dictates equations to engineers for implementation is not  and will never be  a thing.
Data science is a subset of software engineering. You design and implement software. It’s a peculiar kind of software and the design process is unusual but ultimately this is what you do. It is imperative that you get good at it.
Your colleagues will cut you a lot of slack with respect to programming on account of you bringing other skillsets to the table. As a result it is entirely possible for someone to be doing data science for years without picking up good engineering practices and modern technologies. Don’t let this happen to you.
The purely technological part of data science  installing things, getting things in and out of databases, version control, db and cluster administration etc.  may seem like a boring chore to you (I know it did to me)  best left to vanilla engineers who are into this stuff. This type of thinking is a mistake. Becoming better at engineering will:
That doesn’t mean that you have to be an expert coder to start working as a data scientist. You don’t even have to be an expert coder to start working as a coder. But you do need to have the basics and be willing to learn.
A trained engineer with no knowledge of statistics is one online course away from being able to perform a majority of data science jobs. A trained statistician with no tech skills won’t be able to do any data science at all. They may still be a useful person to have around (as a data analyst maybe) but would be completely unable to do any data science on their own.
Why do we even have data scientists then? Why aren’t vanilla engineers taking all the data science jobs?
Data science may not require much in terms of hard maths/stats knowledge but it does require that you’re interested in data and models. And most engineers simply aren’t. The good ones are too busy and too successful as it is to put any serious effort into learning something else. And the mediocre simply lack the kind of curiosity that makes someone excited about reinforcement learning or tweaking a shoe reccomender.
Moreover, there is a breed of superstar software engineers doing driveby data science. I know a few engineers each of whom can run circles around your average data scientist. They can read all the latest papers on a given AI/ML topic, then implement, test and productionise a state of the art recommender/classifier/whatever  all without breaking a sweat  and then move on to nondata related projects where they can make more impact. One well known example of such a person is Erik Bernhardsson  the author of Annoy and Luigi.
These people don’t call themselves ‘data scientists’ because they don’t have to  they already work wherever they want, on whatever projects they want, making lots of money  they don’t need the pretense. No, ‘data scientist’ is a term invented so all the failed scientists  the bored particle physicists and disenchanted neurobiologists  can make themselves look useful to employers.
There is no denying, that
“I’m a data scientist with a strong academic background”
Does sound more employable than
“I’m have wasted 10 best years of my life on theoretical physics but I also took a Python course online, can I have jobs now plz”
I’m being facetious here but of course I do think a smart science grads can be productive data scientists. And they will become immensely more productive if they make sure to steer away from ‘academic with a python course’ and towards ‘software professional who can also do advanced maths’.
]]>Last year I wrote a post about using word embeddings like word2vec or GloVe for text classification. The embeddings in my benchmarks were used in a very crude way  by averaging word vectors for all words in a document and then plugging the result into a Random Forest. Unfortunately, the resulting classifier turned out to be strictly inferior to a good old SVM except in some special circumstances (very few training examples but lots of unlabeled data).
There are of course better ways of utilising word embeddings than averaging the vectors and last month I finally got around to try them. As far as I can tell from a brief survey of arxiv, most state of the art text classifiers use embeddings as inputs to a neural network. But what kind of neural network works best? LSTM? LSTM? CNN? BLSTM with CNN? There are doezens of tutorials on the internet showing how to implement this of that neural classfier and testing it on some dataset. The problem with them is that they usually give metrics without a context. Someone says that their achieved 0.85 accuracy on some dataset. Is that good? Should I be impressed? Is it better than Naive Bayes, SVM? Than other neural architectures? Was it a fluke? Does it work as well on other datasets?
To answer those questions, I implemented several network architectures in Keras and created a benchmark where those algorithms compete with classics like SVM and Naive Bayes. Here it is.
I intend to keep adding new algorithms and dataset to the benchmark as I learn about them. I will update this post when that happens.
All the models in the repository are wrapped in scikitlearn compatible classes with .fit(X, y)
, .predict(X)
, .get_params(recursive)
and with all the layer sizes, dropout rates, ngram ranges etc. parametrised. The snippets below are simplified for clarity.
Since this was supposed to be a benchmark of classifiers, not of preprocessing methods, all datasets come already tokenised and the classifier is given a list of token ids, not a string.
Naive Bayes comes in two varieties  Bernoulli and Multinomial. We can also use tfidf weighting or simple counts and we can include ngrams. Since sklearn’s vectorizer expects a string and will be giving it a list of integer token ids, we will have to override the default preprocessor and tokenizer.
1 2 3 4 5 6 7 8 9 10 11 

SVMs are a strong baseline for any text classification task. We can reuse the same vectorizer for this one.
1 2 3 

In other words  a vanilla feed forward neural network. This model doesn’t use word embeddings, the input to the model is a bag of words.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

Inputs to this model need to be onehot encoded, same goes for labels.
1 2 3 4 5 6 

This is where things start to get interesting. The input to this model is not a bag of words but instead a sequence word ids. First thing to do is construct an embedding layer that will translate this sequence into a matrix of ddimensional vectors.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 

Now for the model proper:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

This and all the other models using embeddings requires that labels are onehot encoded and word id sequences are padded to fixed length with zeros:
1 2 3 4 5 

This is the (slightly modified) architecture from Keras tutorial. It’s specifically designed for texts of length 1000, so I only used it for document classification, not for sentence classification.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 

This is the architecture from the Yoon Kim’s paper, my implementation is based on Alexander Rakhlin’s. This one doesn’t rely on text being exactly 1000 words long and is better suited for sentences.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 

Authors of the paper claim that combining BLSTM with CNN gives even better results than using either of them alone. Weirdly, unlike previous 2 models, this one uses 2D convolutions. This means that the receptive fields of neurons run not just across neighbouring words in the text but also across neighbouring coordinates in the embedding vector. This is suspicious because there is no relation between consecutive coordinates in e.g. GloVe embedding which they use. If one neuron learns a pattern involving coordinates 5 and 6, there is no reason to think that the same pattern will generalise to coordinates 22 and 23  which makes convolution pointless. But what do I know.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

In addition to all those base models, I implemented stacking classifier to combine predictions of all those very different models. I used 2 versions of stacking. One where base models return probabilities, and those are combined by a simple logistic regression. The other, where base models return labels, and XGBoost is used to combine those.
For the document classification benchmark I used all the datasets from here. This includes the 20 Newsgroups, Reuters21578 and WebKB datasets in all their different versions (stemmed, lemmatised, etc.).
For the sentence classification benchmark I used the movie review polarity dataset and the Stanford sentiment treebank dataset.
Some models were only included in document classification or only in sentence classification  because they either performed terribly on the other or took too long to train. Hyperparameters of the neural models were (somewhat) tuned on one of the datasets before including them in the benchmark. The ratio of training to test examples was 0.7 : 0.3. This split was done 10 times on every dataset and each model was tested 10 time. The tables below show average accuracies across the 10 splits.
Without further ado:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 

Well, this was underwhelming.
None of the fancy neural networks with embeddings managed to beat Naive Bayes and SVM, at least not consistently. A simple feed forward neural network with a single layer, did better than any other architecture.
I blame my hyperparameters. Didn’t tune them enough. In particular, the number of epochs to train. It was determined once for each model, but different datasets and different splits probably require different settings.
And yet, the neural models are clearly doing something right because adding them to the ensemble and stacking significantly improves accuracy.
When I find out what exactly is the secret sauce that makes the neural models achieve the state of the art accuracies that papers claim they do, I will update my implementations and this post.
]]>“Data science is statistics on a Mac”
Then there is the famous Venn diagram with data science on the intersection of statstics, hacking and substantive expertise.
What the hell?
Based on all those memes one would think that data scientists spend equal amounts of time writing code and writing integrals on whiteboards. Thinking about the right data structure and thinking about the right statistical test. Debugging pipelines and debugging equations.
And yet, I can’t remember a single time when I got to solve an integral on the job (and believe me, it’s not for lack of trying). I spent a total of maybe a week in the last 3 years explicitly thinking about statistical tests. Sure, means and medians and variances come up on a daily basis but it would be setting the bar extremely low to call that ‘doing statistics’.
Someone is bound to comment that I’m doing data science wrong or that I’m not a true data scientist. Maybe. But if true data scientist is someone who does statistics more than 10% of the time, then I’m yet to meet one.
But maybe this is the wrong way to think about it. Maybe my problem is that I was expecting mathematical statistics where I should have been expecting real world statistics.
Mathematical statistics is a branch of mathematics. Data scientists like to pretend they do it, but they don’t.
Real world statistics is an applied science. It’s what people actually do to make sense of datasets. It requires a good intuitive understanding of the basics of mathematical statistics, a lot of common sense and only infrequently any advanced knowledge of mathematical statistics. Data scientists genuinely do it, a lot of the time.
In my defense, it was an easy mistake to make. Mathematical statistics is what is taught in most courses and textbooks. If any statistics questions come up in a job interview for a data science role  it will be the mathematical variety.
To illustrate what I mean by ‘real world statistics’, to show that this discipline is not trivial and is interesting in its own right, I prepared a short quiz. There are 5 questions. None of them require any complicated math or any calculations. They do require a good mathematical intuition though.
I encourage you to try to solve all of them yourself before checking the answers. It’s easy to convince yourself that a problem is trivial after you’ve read the solution! If you get stuck though, every question has a hint below.
According to CDC data, US counties with the lowest incidence of kidney cancer happen to all be rural, sparsely populated and located in traditionally Republican states. Can you explain this fact? What does it tell us about the causes of cancer?
According to a series of interviews conducted with people who have been in a bar fight, 9 out of 10 times, when someone dies in a bar fight, he was the one who started it. How can you explain this remarkable finding?
After Google measured onthejob performance of their programmers, they found a negative correlation between being a former winner of a programming competition and being successful on the job. That’s right  champion coders did worse on average. That raises important questions for recruiters. Do programming competitions make you worse at 95 programming? Should employers start screening out champion coders?
It is well documented that students from underprivileged backgrounds underperform academically at all levels of education. Two students enter a university  one of them comes from an underprivileged group, the other from a privileged one. They both scored exactly the same on the university admission exam. Should you expect the underprivileged student to do better, the same or worse in the next university exam compared to the other student? Bear in mind that while their numerical scores from the admissions test were the same, it means that the underprivileged student heavily outperformed expectations based on his/her background while the other student did as well as expected from his/her background.
According to studies the average number of sex partners Britons have had in their lives is 9.3 for men and 4.7 for women. How can those numbers possibly be different? After all, each time a man and a woman become sex partners, they increase the total sex partners tally for both sexes by +1. Find at least 5 different factors that could (at least in theory) account for the difference and rate them for plausibility.
It tells us nothing about causes of cancer, it’s a purely statistical effect and it has to be this way. Sparsely populated counties have less people in them, so sampling error is higher. That’s it. Think about an extreme case  a county with a population of 1. If the only inhabitant of this county gets kidney cancer, the county will have 100% kidney cancer rate! If this person remains healthy instead, the county will have cancer incidence rate of 0%. It’s easier for a small group of people to have extremely high or extremely low rate of anything just by chance. Needless to say, republicanism has nothing to do with cancer (as far as we know)  it’s just that rural areas are both sparsely populated and tend to lean Republican.
This example comes from Daniel Kahneman’s awesome book Thinking Fast And Slow. This blog post has a really nice visualisation of the actual CDC data that illustrates this effect.
People lie. Of course the dead one will be blamed for everything!
This one is slightly more subtle. It is not inconceivable that being a Programming Competition Winner (PCW) makes one less likely to be a Good Engineer (GE). But this is not the only and IMO not the most plausible explanation of the data. It could very well be that in the general population there is no correlation between GE and PCW or a positive correlation and the observed negative correlation is purely due to Google’s hiring practices. Imagine a fictional hiring policy where Google only hires people who either won a competition (PCW) or are proven superstar engineers (GE)  based on their open source record. In that scenario any engineer working at Google who was not a PCW would automatically be GE  hence a negative correlation between GE and PCW among googlers. The correlation in the subpopulation of googlers may very well be the opposite of the correlation in the entire population. Treating PCW as a negative in hiring programmers would be premature.
Erik Bernhardsson has a post with nice visual illustration of this phenomenon (which is an of Berkson’s Paradox). The same principle also explains why all handsome men you date turn out to be such jerks.
The underprivileged student should be expected to do worse. The reason is simple  the admissions test is a measure of ability but it’s not a perfect measure. Sometimes students score lower or higher than their average just by chance. When a student scores higher/lower than expected (based on demographics and whatever other information you have) it is likely that the student was lucky/unlucky in this particular test. The best estimate of the student’s ability lies somewhere between the actual test score and our prior estimate (which here is based on the demographics).
To convince yourself that it must be so, consider an example from sports. If a third league football team like Luton Town plays a top club like Real Madrid and ties, you don’t conclude that Luton Town is as good as Real Madrid. You raise your opinion of Luton Town and lower your opinion of Real Madrid but not all the way to the same level. You still expect Real Madrid to win the rematch.
This effect is an example of regression to the mean and it is known as Kelley’s Paradox. This paper illustrates it with figures with actual data from SAT and MCAT exams. You will see that the effect is not subtle!
Average number of sex partners for males is the sum of the numbers of sex partners for all the males divided by the number of all the males:
similarly for females:
The reason we think $MSP$ and $FSP$ should be equal is that every time a man and a woman become sex partners, the numerators of both $MSP$ and $FSP$ increase by $+1$. And the denominators are approximately equal too. Let’s list all the ways this tautology breaks down in real life:
And there are other factors as well, although it’s not clear to me which way would they tend to bias the ratio:
My good friend Nadbor told me that he found on Reddit someone asking if data scientists end up doing boring tasks such as classifying shoes. As someone that has faced this problem in the past, I was committed to show that classifying shoes it is a challenging, entertaining task. Maybe the person who wrote that would find it more interesting if the objects to classify were space rockets, but whether rockets or shoes, the problem is of the same nature.
Imagine that you work at a fashion aggregator, and every day you receive hundreds of shoes in the daily feed. The retailers send you one identifier and multiple images (with different points of view) per shoe model. Sometimes, they send you additional information indicating whether one of the images is the default image to be displayed at the site, normally, the sideview of the shoe. However, this is not always the case. Of course, you want your website to look perfect, and you want to consistently show the same shoe perspective across the entire site. Therefore, here is the task: how do we find the side view of the shoes as they come through the feed?
Before I jump into the technical aspect of the solution, let me just add a few lines on teamwork. Through the years in both real science and data science, I have learned that cool things don’t happen in isolation. The solution that I am describing here was part of a team effort and the process was very entertaining and rewarding.
Let’s go into the details.
The solution implemented comprised two steps:
1. Using the shape context algorithm to parameterise shoeshapes
2. Cluster the shapes and find those clusters that are comprised mostly by sideview shoes
Details on the algorithm can be found here and additional information on our python implementation is here. The steps required are mainly two:
1. Find points along the silhouette of the shoe useful to define the shape.
2. Compute a Shape Context Matrix using radial and angular metrics that will effectively parameterise the shape of the shoe.
Finding the relevant points to be used later to compute the Shape Context Matrix is relatively easy. If the background of the image is white, simply “slice” the image and find the initial and final points that are not background per slice. Note that due to the “convoluted” shapes of some shoes, techniques relying on contours might not work here.
I have coded a series of functions to make our lives easier. Here I show the results of using some of those functions.
The figure shows 60 points of interest found as we move along the image horizontally.
Once we have the points of interest we can compute the radial and angular metrics that will eventually lead to the Shape Context Matrix. The idea is the following: for a given point, compute the number of points that fall within a radial bin and an angular bin relative to that point.
In a first instance, we computed 2 matrices, one containing radial information and one containing angular information, per point of interest. For example, if we select 120 points of interest around the silhouette of the shoe, these matrices will be of dim (120,120).
Once we have these matrices, the next step consists in building the shape context matrix per point of interest. Eventually, all shape context matrices are flattened and concatenated resulting in what is referred to as Bin Histogram.
Let’s have a look at one of these shape context matrices. For this particular example we used 6 radial bins and 12 angular bins. Code to generate this plot can be found here:
This figure has been generated for the first point within our pointsofinterestarray and is interpreted as follows: if we concentrate on the upperleft “bucket” we find that, relative to the first point in our array, there are 34 other points that fall within the largest radial bin (labelled 0 in the Figure) and within the first angular bin (labelled 0 in the Figure). More details on the interpretation can be found here
Once we have a matrix like the one in Figure 2 for every point of interest, we flatten and concatenate them resulting in an array of 12 $\times$ 6 $\times$ number of points (120 in this case), i.e. 8640 values. Overall, after all this process we will end up with a numpy
array of dimensions (number of images, 8640). Now we just need to cluster these arrays.
A detailed discussion on how to pick the number of clusters and the potential caveats can be found here. In this post I will simply show the results of using MiniBatchKMeans to cluster the arrays using 15 clusters. For example, clusters 2,3 and 10 look like this.
Interestingly cluster 1 is comprised of images with an nonwhite and/or structured background, images with a shape different than that of a shoe and some misclassifications. Some advise on how to deal with the images in that cluster can be found here
There are still a few aspects to cover to isolate the side views of the shoes with more certainty, but I will leave this for a future post (if I have the time!).
In addition, there are some other features and techniques one could try to improve the quality of the clusters, such as GIST indicators or Halarick Textural Features.
Of course, if you have the budget, you can always pay for someone to label the entire dataset, turn this into a supervised problem and use Deep Learning. A series of convolutional layers should capture shapes, colours and patterns. Nonetheless, if you think for a second about the nature of this problem, you will see that even deciding the labelling is not a trivial task.
Anyway, for now, I will leave it here!
The code for the process described in this post can be found here
]]>I’m interviewing data engineering contractors recently. All of the candidates are very senior people with 10+ years of experience. My go to question:
Me: What data structure would you use (in your favorite programming language) to store a large number (let’s say 100k) of strings  so they can be looked up efficiently? And by ‘looked up’ I mean  user will come up with a new string (‘banana’) and you have to quickly tell if this string is an element of your collection of 100k?
Candidate: I would load them in an RDD and then…
Me: No, no, I’m not asking about Spark. This is a regular singlethreaded, inmemory, computer science 101 problem. What is the simplest thing that you could do?
Candidate: Grep. I would use grep to look for the string.
Me: Terrific. Sorry, maybe I wasn’t clear, I’m NOT talking about finding a substring in a larger text… You know what, forget about the strings. There are no strings. You have 100k integers. What data structure would you put them in so you can quickly look up if a new integer (1743) belongs to the collection?
Candidate: For integers I would use an array.
Me: And how do you find out if the new integer belongs to this array?
Candidate: There is a method ‘contains’.
Me: Ok. And for an array of n integers, what is the expected running time of this method in terms of n?
Candidate: …
Me: …
Candidate: I think it would be under one minute.
Me: Indeed.
This one was particularly funny, but otherwise unexceptional. This week I interviewed 4 people and not a single one of them mentioned hash tables. I would have also accepted ‘HashMap’, ‘Map’, ‘Set’, ‘dictionary’, ‘python curly braces’  anything pointing in vaguely the right direction, even if they didn’t understand the implementation. Instead I only got ‘a vector, because they are thread safe’, ‘ArrayList because they are extensible’, ‘a List because lists in scala are something something’, ‘in my company we always use Sequences’. Again: these are very experienced people who are being paid a lot of money contracting for corporations in London and who can very convincingly bullshit about their Kafkas, Sparks, HBases and all the other Big Data nonsense.
Another bizarre conversation occurred when a candidate with 16 years of experience with Java (confirmed by the Sun certificate) immediately came up with the idea of putting the strings in buckets based on their hash and started explaining to me basically how to implement a hash table in Java, complete with the discussion of the merits of different string hashing functions. When I suggested that maybe Java already has a collection type that does all of this he reacted with indignation  he shouldn’t have to know this, you can find out on the internet. Fair enough, but one would think that after 16 years of programming in that language someone would have encountered HashMaps once or twice… This seemed odd enough that for my next question I went off script:
Me: Can you tell me what is the signature of the main method in Java?
Candidate: What?
Me: Signature of the main method. Like, if you’re writing the ‘hello world’ program in Java, what would you have to type?
Candidate: class HelloWorld
Me: Go on.
Candidate: int main() or void main() I think
Me: And the parameters?
Candidate: Yes, I remember, there are command line parameters.
Me: …
Candidate: Two parameters and the second is an integer.
Me: Thank you, I know all I wanted to know.
Moral of this story?
Come to London, be a data engineering contractor and make £500/day. You can read about Java on wikipedia, put 15 years of experience on your resume and no one will be the wiser.
]]>Which should you use and when? Which should you learn first? Is type safety more important than flexibility? Is Python fast enough for performanceheavy applications? Is Scala’s machine learning ecosystem mature enough for serious data science? Are indents better than braces?
This post won’t answer any of those questions.
I will show how to solve a related problem though. Given the following text, which was stitched together from bits of scikitlearn and scalaz code files, can you tell where does Python end and Scala begin?
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 

I will show how Keras LSTMs and bidirectional LSTMs can be used to neatly solve this problem. The post will contain a some snippets of code but the full thing is here.
I once interviewed with a cyber security company that was scraping the web looking for people’s phone numbers, emails, credit card numbers etc. They asked me how I would go about building a model that finds those things in text files and also categorizes the files into types like ‘email’, ‘server logs’, ‘code’, etc.
The boring answer is that with enough feature engineering you could classify files pretty well with any old ML algorithm. If all lines have a common prefix 
1 2 3 4 5 6 

 then we’re probably dealing with a log file. If we’re there’s a lot of camelCase()
 that means we’re seeing code. And so on.
Finding e.g. phone numbers in text is more involved but still doable this way. You would have to first generate potential potential matches using regular expressions and then classify each as a true or spurious based on the context it appears in.
Inevitably, for every new file type and every type of entity to be found in the file, one would have to come up with new features and maybe train a separate classifier.
Super tedious.
The fun and potentially superior solution uses charRNNs. Instead of all those handcrafted features and regular expressions and different models, we can train a single recurrent neural network to label each character in the text as either belonging to a phone number (credit card number, email …) or not. If we do it right and have enough training data, the network should be able to learn that phone numbers are more likely to occur in emails than in server logs and that Java code tends to use camel case while Python has indented blocks following a colon  and all kinds of other features that would otherwise have to be hardcoded.
Let’s do it!
As it turned out, the hardest part was getting and preparing the data. Since I don’t have access to a labeled dataset with phone numbers and emails, I decided to create an artificial one. I took all the Python files from scikitlearn repository and all the Scala files from scalaz and spliced them together into one giant sequence of characters. The sequence takes a few dozen consecutive characters from a Python file, then a few dozen from a Scala file, then Python again and so on. The result is the Frankenstein’s monster at the top of the post (except tens of megabytes more of it).
The sequence made up of all the Python and Scala files wouldn’t fit in my RAM (Big Data, as promised ;), so it is generated online during training, using a generator:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 

The other reason for using a generator is that the sequence can be randomized (both the order of files and the number of consecutive characters taken from one source). This way the network will never see the same sequence twice which will reduce overfitting.
Next step is encoding the characters as vectors (onehotencoding):
1 2 3 4 5 6 7 8 9 10 

To take advantage of the parallel processing powers of the GPU, the input vectors need to be shaped into batches. Keras requires that batches for LSTM be 3dimensional arrays, where first dimension corresponds to the number of samples in a batch, second  number of characters in a sequence and third  dimensionality of the input vector. The latter is in our case equal to the number of characters in our alphabet.
For example, if there were only two sequences to encode, both of length 4, and only 3 letters in the alphabet, this is how we would construct a batch:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

If the sequences are too long to fit in one batch  as they are in our case  they need to be split into multiple batches. This would ordinarily mean losing some context information for characters that are near the boundary of a sequence chunk. Fortunately Keras LSTM has a setting stateful=True
which tells the network that the sequences from one batch are continued in the next one. For this to work, the batches must be prepared in a specific way, with nth sequence in a batch being continued in the nth sequence of the next batch.
1 2 3 4 5 6 7 8 9 10 11 12 13 

In our case, each sequence is produced by a generator reading from files. We will have to start a number of generators equal to the desired batch size.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

Done. This generator produces batches accepted by Keras’ LSTM. batch_size
and sequence_len
settings influence GPU/CPU utilisation but otherwise shouldn’t make any difference (as long as stateful=True
!).
Now for the easy part. Construct the network:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

And train it:
1 2 3 4 5 6 

Making predictions is just as easy:
1


That’s it! The full code I used has a few more bells and whistles, but this is the core of it.
I have split the Python and Scala files into train and test sets (80:20) and trained the network on the training set for a few hours. This is what the network’s prediction on the test set (same text as on top of of this post) looks like:
package scalaz
package syntax
"""
Extended math utilities.
"""
# Authors: Gael Varoquaux
# Alex/** Wraps a value `selfandre Gramfort
# Alexandre T. Passos
# Olivier Grisel
# Lars Buitinck
# Stefan van der Walt
# Kyle Kastner
# Giorgio Patrini
# License:` and provides methods related to `MonadPlus` */
final class MonadPlusOps[F[_],A] private[syntax](val self: BSD 3 clause
from __future__ import division
from functools import partial
import warnings
import numpy as np
from scipy import linalg
from scipy.sparse import issparse, csr_matr F[A])(implicit val F: MonadPlus[F]) extends Ops[F[A]] {
////
impoix
from . import check_random_state
from .fixrt Leibniz.===
def filter(f: A => Boolean): F[A] =
F.filter(self)(f)
def withFilter(f: A => Boolean): F[A] =
filter(f)
final def uniteU[T](implicit T: Unapply[Foldable, Aes import np_version
from ._logistic_sigmoid import _log_logistic_sigmoid
from ..extern]): F[T.A] =
F.uniteU(self)(T)
def unite[T[_], B](implicit ev: A === T[B], T: Foldable[T]): F[B] = {
val ftb: F[T[B]] = ev.subst(seals.six.moves import xrange
from .sparsefuncs_fast import csr_row_norms
from .validation import check_array
from ..exceptions import NonBLASDotWarning
lf)
F.unite[T, B](ftb)
}
final def lefts[G[_, _], B, C](implicit ev: A === G[B, C], G: Bifoldable[G]): F[B] =
F.lefts(ev.subst(self))
final def rigdef norm(x):
"""Compute the Euclidean or Frobenius norm of x.
hts[G[_, _], B, C](implicit ev: A === G[B, C], G: Bifoldable[G]): F[C] =
F.rights(ev.subst(self))
final def separate[G[_, _], Returns the Euclidean norm when x is a vector, the Frobenius norm when x
is a matrix (2d array). More precise than sqrt(squared_norm(x)).
"""
x = np.asarray(x)
nrm2, = lin B, C](implicit ev: A === G[B, C], G: Bifoldable[G]): (F[B], F[C]) =
F.separate(ev.subst(self))
////
}
sealed trait ToMonadPlusOps0 {
implicit def Talg.get_blas_funcs(['nrm2'], [x])
return nrm2(x)
# Newer NumPy has a ravel that needs leoMonadPlusOpsUnapply[FA](v: FA)(implicit F0: Unapply[MonadPlus, FA]) =
new MonadPlusOps[F0.M,F0.A](F0(v))ss copying.
if np_version < (1, 7, 1):
_ravel = np.ravel
else:
_ravel = partial(np.ravel, order='K')
def squared_no(F0.TC)
}
trait ToMonadPlusOps extends ToMonadPlusOps0 with ToMonadOps with ToApplicatrm(x):
"""Squared Euclidean or Frobenius norm of x.
Returns the Euclidean norm when x is a vector, the Frobenius norm when x
is a matrix (2d array). Faster than norm(ivePlusOps {
implicit def ToMonadPlusOps[F[_],A](v: F[A])(implicit F0: MonadPlus[F]) =
new MonadPlusOps[F,A](v)
////
////
}
trait MonadPlusSyntax[F[_]] extends MonadSyntax[F] withx) ** 2.
"""
x = _ravel(x)
if np.issubdtype(x.dtype, np.integer):
ApplicativePlusSyntax[F] {
implicit def ToMonadPlusOps[A](v: F[A]): MonadPlusOps[F, A] = ne warnings.warn('Array type is integer, np.dot may overflow. '
'Data should be float type to avoid this issue',
UserWarning)
return np.dot(xw MonadPlusOps[F,A](v)(MonadPlusSyntax.this.F)
def F: MonadPlus[F]
////
////
}
package scalaz
package syntax
/** Wraps a value `self` and provides methods, x)
def row_norms(X, squared=False):
"""Rowwise (squared) Euclidean norm of X.
E related to `Traverse` */
final class Tquivalent to np.sqrt((X * X).sum(axis=1)), but also supporaverseOps[F[_],A] private[syntax](val self: F[A])(implicit val F: Traverse[F]) exterts sparse
matrices and does not create an X.shapesized temporary.
Performs no input valnds Ops[F[A]] {
////
import Leibniz.===
final def tmap[B](f: A => B): F[B] =
F.map(seidation.
"""
if issparse(X):
if not isinstance(X, csr_matrix):
Font size shows the true label (small  Python, big  Scala) and background color represents the network’s prediction (white  Python, dark red  Scala).
It’s pretty good overall, but network keeps making a few unforced errors. Consider this bit:
package scalaz
package syntax
"""
package scalaz
should be a dead giveaway, the prediction only becomes confident at about the character ‘g’""""
following a stretch of Scala code. Triple quotes should immediately be labeled as Python but only the third one is.These mistakes stem from the fact that the RNN doesn’t look ahead and can only interpret a character in the context of characters that came before. Triple quotes almost certainly come from a stretch of Python code, but you don’t know that you’re seeing triple quotes until you’ve seen all three. That’s why the prediction gradually changes from Scala to Python (red to white) as the RNN encounters the second and third consecutive quote.
This problem actually has a straightforward solution  bidirectional RNN. It’s a type of RNN where the sequence is fed to it from both ends at the same time. This way, the network will be aware of the second and third quotation marks already when it’s producing the label for the first one.
To make the LSTM bidirectional in Keras one needs simply to wrap it with the Bidirectional
wrapper:
1 2 3 4 5 6 

Everything else stays the same.
Here’s a sample of results from a bidirectional LSTM:
package scalaz
package std
import std.AllInstances._
import scalaz.scalacheck.ScalazProperties._
import scalaz.scalac"""
===============================heck.ScalazArbitrary._
import org.scalacheck.{Gen, Arbitrary}
import Lens.{lens => _, _}
import org.scalacheck.Prop.fo=========
Comparison of Calibration of ClassifrAll
object LensTest extends SpecLite {
{
implicit def lensArb = Arbitrary(Gen.const(Lens.lensId[Int]))
implicit def lensEqual = new Equal[Lens[Int, Iiers
========================================
Well calibrated classifiers are probabint]] {
def equal(a1: Lens[Int, Int], a2: Lens[Int, Int]): Boolean = a1.get(0) == a2.get(0)
}
checkAll("Lens", category.laws[Lens]) // not really testing much!
}
checkAll("id",listic classifiers for which the output
of the predict_proba method can be directly interpreted as a confidence level.
For instance a well calibrated (binary) classifier should classify the samp lens.laws(Lens.lensId[Int]))
checkAll("trivialles
such that among the samples to which it gave a predict_proba", lens.laws(Lens.trivialLens[Int]))
checkAll("codiagLens", lens.laws(Lens.codiagLens[Int]))
checkAll("Tuple2.first", lens.laws(Lens.firstLens[Int, Int]))
checkAll("Tuple2.second", le value close to
0.8, approx. 80% actually belong to the positive class.
Logisticns.laws(Lens.secondLens[Int, Int]))
checkAll("Set.containRegression returns well calibrated predictions as it directly
os", lens.laws(Lens.lensId[Set[Int]].contains(0)))
checkAll("Map.member", lens.laws(Lens.lensId[Map[Boolean, Int]].ptimizes logloss. In contrast, the othemember(true)))
checkAll("sum", lens.laws(Lens.firsr methods return biased probabilities,
with different biases per method:
* GaussianNaiveBayes tends to push probabilities to 0 otLens[Int, String].sum(Lens.firstLens[Int, String])))
"NumericLens" should {
"+=" ! forAll((i: Int) => (Lens.lensId[Int] += i).run(1) must_=== ((i + 1) > (i +
I think this looks better overall. The problem of updating prediction too slowly is mostly gone  package scalaz
is marked as Scala immediately, starting with the letter ‘p’. However, now the network started making weird mistakes in the middle of a word for no reason. Like this one:
Comparison of Calibration
Why is the middle of the ‘Calibration’ all of a sudden marked as Scala?
The culprit is statefulness. Remember that stateful=True
means that for each sequence in a batch, the state of the network at the beginning of a sequence is reused from the state at the end of the previous sequence*. This acts as if there were no batches, just one unending sequence. But in a bidirectional layer the sequence is fed to the network twice, from both directions. So half of the state should be borrowed from the previous sequence, and half from the next sequence that has not been seen yet! In reality all of the state is reused from previous sequence, so half of the network ends up in the wrong state. This is why those weird mispredictions appear and appear at regular intervals. At the beginning of a new batch, half of the network is in the wrong state and starts predicting the wrong label.
* or more precisely, the state at the end of the corresponding sequence in the previous batch
Let’s get rid of statefulness in the bidirectional version of the network:
1


Unfortunately this means that we will have to use longer sequences (in the previous experiments I used 128 characters, now 200) to give the network more context for labeling a character. And even with that, prediction for characters near the boundary between consecutive sequences is bound to be poorer  like in regular unidirectional LSTM. To make up for it I decided to give the network more layers (4) and more time to train (a day). Let’s see how it worked out:
package scalaz
import scalaz.syntax.equal._
import scalaz.syntax.show._
sealed abstract class Either3[+A, +B, +C] extends Pro"""Bayesian Gaussian Mixture Modduct with Serializable {
def fold[Z](left: A => Z, middle: B => Z, right: C => Z): Z = this match {
case Left3(a) => left(a)
caseel."""
# Author: Wei Xue <xuewei4d Middle3(b) => middle(b)
case Right3(c) => right(c)
}
def eitherLeft: (A \/ B) \/ C = this match {
case Left3(a) => \@gmail.com>
# Thierry Guillemot <thierry.guillemot.work@gmail.com>
# License: BSD 3 clause
import math
import numpy as np
from scipy.special import betaln, digamma, /(\/(a))
case Middle3(b) => \/(\/(b))
case Right3(c) => \/(c)
}
gammaln
from .base import BaseMixture, _check_shape
from .gaussian_mixture import _check_precision_matrix
from .gaussian_mixture import _check_precision_positivity
from .gaus def eitherRight: A \/ (B \/ C) = this match {
case Left3(a) => \/(a)
case Middle3(b) => \/(\/(b))
case Right3(c)sian_mixture import _compute_log_det_cholesky
from .gaussian_mixture import _compute_precision_cholesky
from .gaussian_mixture import _estimate_gaussian_p => \/(\/(c))
}
def leftOr[Z](z: => Z)(f: A => Z): Z = fold(f, _ => z, _ => z)
def middleOr[Z](zarameters
from .gaussian_mixture import _estimate_log_gaussian_prob
from ..utils import check_array
from ..utils.validation import check_is_fitted
def _log_dirichlet_norm(dirichlet_concentration: => Z)(f: B => Z): Z = fold(_ => z, f, _ => z)
def rightOr[Z](z: => Z)(f: C => Z): Z = fold(_ => z, _ => z, f)
}
final case class Left3[+A, +B, +C](a: A) extends Either3[A, B, C]
final case cla):
"""Compute the log of the Dirichlet distribution normalization term.
Parameters

dirichletss Middle3[+A, +B, +C](b: B) extend_concentration : arraylike, shape (n_samples,)
The s Either3[A, B, C]
final case class Right3[+A, +B, +C](c: parameters values of the Dirichlet distribution.
Returns

log_dirichlet_norm : float
The log normalization of the DirichleC) extends Either3[A, B, C]
object Either3 {
def left3[A, B, C](a: A): Either3[A, B, C] = Left3(a)
def middle3[A, B, C](b: B)t distribution.
"""
return (gammaln(np.sum(dirichle