Thou Shalt Use Open-Ended Questions (or thou shalt not hear your customer’s voice)

Dmitry Grenader, Luminoso’s Director of Product Management, writing to challenge the status-quo in market-research and voice-of-the-customer surveys where multiple-choice questions dominate the scene and dwarf open-ended questions.

Why do we love multiple-choice questions so much? There are a few reasons.

  • They are easy to create!  Let’s say I am surveying folks on their favorite ice-cream flavors – it soooo easy for me to just fill “strawberry, vanilla, chocolate, green tea, …”.
  • They are also easy to take – they are a perfect “don’t make me think” task – as a respondent I just pick the one I like and move on.
  • They are easy to analyze – you get your bar-graphs and pie-charts quickly, and the statistically inclined might even look at the mean and (ahem) standard deviation.
  • If distributed to enough people, the results are statistically significant.

What about open-ended questions? Why do we shun them like the Wizarding world of Harry Potter shuns squibs? The main reason is that the analysis is just too damn hard.

Imagine some poor schmuck – sorry, I mean a Product Manager (like me) – trying to analyze results and having to look through 10,000 responses. Impossible! I mean c’mon people – I only get a two-day extension from CEO to research and validate the product viability before we commit like two million dollars to it, and I had to promise my first-born for even that reprieve – I surely don’t have time to read what thousands of people said. So what do I do? Well, I read through a few of them, and pick the few juicy comments that support my view of the world plus a couple of negative ones to appear balanced, and copy-n-paste into my powerpoint deck – and call it done. Ok, I don’t really do that, I am sure you don’t either – but you know, those other people do.

Bottomline – multiple-choice questions are the main dish, whilst the open-ended ones are a side-dish at best, an after-thought forever destined to be a lowly “catch-all”. And everyone is ok with it.  Well, I am not!

There are huge problems with this tyranny of the multiple-choice:

  • Options are leading the witness! Let’s face it – when you ask about a favorite ice-cream flavor and you give a choice of five, and the user selects one you have no idea that the user actually would have selected a different one. They have a real favorite – but it was not listed.
  • By pre-creating the choices you are quite literally biasing the user to respond within the framework you have created – so ultimately you are only getting answers within the space of answers you yourself kind of pre-cooked (there is something from quantum theory in this – something about the eye of the beholder and how what you find is determined by how you look for it)
  • They are not allowing your customers to fully express their voice. It is not hard to imagine this internal dialog that happens in your customer’s head when they are asked to choose between strawberry and vanilla ice-cream flavors on the survey: “Well, I kind of like strawberry but I wish it were more aromatic and not as sweet. Vanilla flavor is ok, but I have recently tried this french vanilla by their competitor and it had a hint of pistachios, which I absolutely adored.  I wonder if they can tweak their vanilla to be bolder.” – Wow! Mic drop. No seriously – this is incredible feedback, a voice of the customer at its purest, glorious, unrefined, primordial and incredibly useful state. The kind of feedback that if falls on hearing ears can make the product and company great and create lifelong brand-champions. But NOOOOO, instead this comment gets lost and ends up on the cutting floor by default – ‘cause nobody had time to read it. Or worse nobody dared to even ask the question in the first place.

I hope you are sharing in my righteous indignation, and pounding your fist on the table in outrage and screaming “IS THERE A BETTER WAY?!” The answer is yes.

There is a way to understand the feedback without having to yourself read through the 10,000 responses. You can do that by relying on the text-analytics of a modern voice-of-the-customer platform. The technology has recently come of age where you can rely on machine-learning based approaches to do the feedback reading for you and provide you with insights and the tools to understand what your customers are saying and how they are feeling. Distilling feedback into concepts and topics, finding the connections between them and surfacing the emotional context enable you to quickly and effectively analyze responses. Think of it as “augmented reality” where you can wear a lens with which you can read thousands of responses and understand what is contained in them. And you know what else? This technology can turn qualitative into quantitative – i.e. you can get actual number that are statistically significant on how people feel about various aspects of your business.

This breakthrough allows you to rethink how you construct your surveys and hear more of your customer’s voice. Delve into nuances! Be provocative!  Ask really Open Ended Questions like

What can we do to serve you better, Ms./Mr. Customer?

Do not shy away from the uncertainty – embrace it, you now can with voice-of-customer technology in your corner.  You will be amazed at the results you get.

P.S. As for the multiple-choice questions, keep those too of course – but remember they are not panacea.

Can we do Arabic?

And now, a message from our Senior Linguistics Developer, and one of our original Luminosi, Dr. Lance Nathan…

What happens when the CEO of Luminoso comes to my office and asks, “Can we do Arabic?”.

In general, when people ask me whether Luminoso’s software can handle a language we don’t yet support–Estonian, Esperanto, Klingon, what have you–my answer is always “yes, of course”. Admittedly, I follow this up with “That is to say, you can put it into the system and see what happens”…which is my answer because “handling” a language involves a number of complicated factors. We’d like to have some background knowledge in the language, and we’d like a word frequency list (see Rob’s blog post from earlier this month for more on that topic).

But the thing we need most is software to parse the text: to break it up into words and to give us base forms we can use to represent those words. Without that, analysts are left looking at our software and thinking, “Well, here’s what e-book users say about ‘reading’, and here’s what they say about ‘read’, and here’s what they say about ‘reads’, and…why are these different concepts?”. Of course, they’re not different concepts, but if you did put Klingon into our system, it wouldn’t know that be’Hom and be’Hompu’ are the same concept. (Those mean “girl” and “girls”. I had to look them up.) You would still find insights–you’d probably learn that “battle” and “happiness” are closely related in Klingon–they just wouldn’t be quite as solid as they would be if we had a parser.

So when the CEO comes to my office and asks, “Can we do Arabic?”, I give this explanation, ending with something like “So all we would need is software that can convert plurals to singulars and so forth.” At which point she says to me, “Terrific! Get right on that”–and I am reminded that talking to your CEO is different than talking to most other people. (Of course, to be fair, she knew we already have software that would do most of the work; my real task would be evaluating it and working around any idiosyncracies I found.)

In truth, though, while the project looked daunting, it also looked exciting. Developing Russian for our product was an interesting journey, but in some ways a very familiar one. Russian has a different alphabet, but like English it forms plurals by putting a suffix on a noun, and forms tenses and other verb variations by putting a suffix on a verb, and so forth. All a parser has to do is recognize the word, take some letters off the end, and voilà: a root word that represents the base concept! Arabic doesn’t work that way at all.

How does Arabic work?

It turns out that there were two basic challenges to parsing Arabic, and its approach to suffixes was only the first one.

Take the Arabic root كتب, which is just the three consonants k, t, and b. It means “write”, and interspersing certain vowels will give you the words for “he wrote” (kataba), or “he writes” (yaktubu), or even “he dictates”, along with other vowels for the “I” form, the “you” form, and so forth. Add different vowels and you get a slew of related nouns: “book” (kitaab) or “library” (maktaba) or “office” (maktab)…to say nothing of the vowels you would change those to if you wanted a plural like “books” (kitub) or “offices” (makatib). All of which would be complicated enough, except that outside of the Qur’an, most of the vowels are almost never written, leaving a parser to reconstruct “yaktubu” from just “yktb”, and to know that “ytkb” is the same concept as the verb “write” but not the noun “book”. This bears so little relation to English or French or Russian that I hesitated to even believe anyone could write a parser to handle it.

Fortunately, I didn’t have to write the parser; once I had one that worked, I would merely need to offer some guidance, correct it when it went astray, and decide which of its many outputs I wanted (yaktubu? yktb? ktb? something in between?). Unfortunately, the language’s rules for word formation was only the first problem; my second problem was that no one speaks Arabic.

Now, obviously that can’t be true; with over 240 million speakers, Arabic is the fifth most spoken language in the world. It turns out, however, that what no one speaks is standard Arabic–that is, Modern Standard Arabic, or MSA. When speaking formally or in an international setting, as at the United Nations or on Al-Jazeera, speakers do indeed use this standard form. Outside of such settings, speakers use their local dialect: Moroccan, Sudanese, Egyptian, Levantine, and many others, and that extends to writing, especially in online forums like Twitter. Often the local written form matches the local spoken form–not unknown in online English, where someone might write “deez” instead of “these”, but much more common in written Arabic, and in this case rather than getting a nonsense word from a small variation in the spelling of “these”, you get a word meaning “delirious”. (Which actually happens.)

Early in the career of a computational linguist, you learn that most language-processing systems are designed to work on standard versions of languages: a French parser may not handle quirks of Québecois French, an English parser probably used news articles as training data and won’t know many of the words it sees on Twitter. Any Arabic parser would similarly be based on Modern Standard Arabic; could it be convinced to handle dialects?

Of course, there was also a third problem I haven’t even mentioned: I don’t speak Arabic. But here at Luminoso, we don’t let minor technicalities stop us, so we contracted a native speaker to help me, I downloaded a few apps to teach me the alphabet, and off we went.

What a parser can (and can’t) do

On the bright side, writing a program to parse Arabic wouldn’t really be my job; I only needed to evaluate the ones available and build on those. Some initial exploration suggested that pretty good parsers did indeed already exist. All the same, putting Arabic in our system wouldn’t be as simple as dropping one into our software and letting it roam free.

Many Arabic parsers are built on the grammatical structures seen in the Qur’an, which is written in language essentially the same as Modern Standard Arabic. Therefore, they may classify the prefix “l-” as ambiguous between the preposition “to” and an indicator of emphasis on the noun, but the latter is only used in literary Arabic (for instance, the Qur’an). We had to tell our software that if the parser categorized anything as “emphatic particle”, it should go back and find another option.

But there were other, subtler problems inherent to the nature of Arabic grammar. An “a-” prefix on a verb might indicate a causative form; it’s this form that turns “he writes” into “he dictates” (i.e., he causes someone to write), or “to know” into “to inform” (i.e., to cause someone to know something). On the other hand, an “a-” prefix can also indicate that “I” is the subject of the verb. A good Arabic parser may return both alternatives, but we found that we couldn’t necessarily rely on our parser to guess which right in a particular sentence. For this, I had to sit down with our native speaker and simply look at a lot of sentences and their parses, asking for each, “Did the parser return the right result here? What about here? If the result was wrong, was it at least a reasonable interpretation in context, or can we determine which result we wanted?”

In the end, we did have to accept some limitations of the parser. The Arabic word ما (“maa”) means “what”, but it is also used for negation in some circumstances, and deciding which as which proved too difficult for the computer. You see ambiguity in all languages, of course: in English, “can” might mean “is able to”, in which case it’s an ignorable common word, or it might mean “metal container”, in which case we wouldn’t want to ignore it. But most cases are easy to distinguish–you don’t even need the whole sentence to know which “can” is which in the phrases “the can” or “can see”. In this case, where both meanings are common function words, it became much harder to get reliable results.

The dialect problem never went away, but we did learn to minimize its effects. We included several common dialect spellings of function words on our “words to ignore” list, so that even if the parser thought they were nouns or verbs, we knew to skip them in our analysis. And we found that in an international data set like hotel reviews, there was enough Modern Standard Arabic for us to successfuly gain insights from it. I’d want to fine-tune the program before loading, say, thousands of sentences of a single dialect, especially if that dialect varies significantly from the standard (Tunisian Arabic, for example, has influences from several European and African languages), but after the development we’ve already done, I’d be confident in our ability to do that fine-tuning.

A final unexpected challenge came when we looked at the results in our visualizer: many things were backwards! Not the words, fortunately, but arrows would point in the wrong direction, text would align flush against the wrong edge, even quotation marks would appear at the wrong edge of the text. It turns out that many, many programs, including web browsers, simply despair when you mix text that reads left-to-right (like English) with text that reads right-to-left (like Arabic).


Pictured: on the left, left to right (wrong); on the right, right to left (right). It’s as confusing as it sounds.

That one turned out to be far easier to fix than we expected: style sheets for web pages allow you to specify that the direction of the text is right-to-left, at which point the browser everything flips to look the way it should.

What now?

In the end, I’m quite pleased at how well our system handles Arabic. Starting as a task that I knew would be hard and I feared would be simply impossible, this project has ended with the ability to find insights in Arabic text that I’d readily put up against our French or Russian capabilities. I can now tell people that I’ve taught a computer to understand Arabic, which may be an exaggeration, but it does still understand more Arabic than I do.

Adding Arabic also means that we can now find insights in the language of nearly 40% of the world’s population, including all six languages of the United Nations; and that we cover four of the five most spoken languages in the world–and who knows, perhaps Hindi will be next (unless Klingon turns out have higher demand than I anticipated, in which case, Heghlu’meH QaQ jajvam).

If Trump is elected, I’m moving!

We’ve been doing a lot of tracking and analysis recently during the preliminary months of the Presidential Election. From the Republican Circus to the Democratic (er, Hillary) PR foibles, there has been no short of conversation across the social media, especially the Twittersphere.

Most notably, it seems that the Donald has evoked the most emotional and reactionary commentary. How emotional and reactionary, you ask? Well, let’s just say that people are so turned off by his candidacy that they would consider moving to other countries if he is elected!

From August 6 to September 9, we looked at over 4.5 million Tweets related to Donald Trump. Take a look at our Eli Orkin infographic original, highlighting our findings.

Trump Tracker

Trump Tracker2

#tbt to #SXSW2015 & @havasi @IgniteSXSW

If you had five minutes, what would you say?

That’s what IgniteSXSW asks. “Enlighten us, but make it quick.”

Ignite is a series of 5 minute presentations about geeky subjects delivered in a format of 20 slides (15 seconds per slide, auto advancing). Luminoso’s CEO & co-founder, Catherine Havasi, participated in this event at this past SXSW.

Check out Catherine’s talk, “I love Big Data” here!

wordfreq: Open source and open data about word frequencies

Often, in NLP, you need to answer the simple question: “is this a common word?” It turns out that this leaves the computer to answer a more vexing question: “What’s a word?”

Let’s talk briefly about why word frequencies are important. In many cases, you want to assign more significance to uncommon words. For example, a product review might contain the word “use” and the word “defective”, and the word “defective” carries way more information. If you’re wondering what the deal is with John Kasich, a headline that mentions “Kasich” will be much more likely to be what you’re looking for than one that merely mentions “John”.

For purposes like these, it would be nice if we could just import a Python package that could tell us whether one word was more common than another, in general, based on a wide variety of text. We looked for a while and couldn’t find it. So we built it.

wordfreq provides estimates of the frequencies of words in many languages, loading its data from efficiently-compressed data structures so it can give you word frequencies down to 1 occurrence per million without having to access an external database. It aims to avoid being limited to a particular domain or style of text, getting its data from a variety of sources: Google Books, Wikipedia, OpenSubtitles, Twitter, and the Leeds Internet Corpus.

The 10 most common words that wordfreq knows in 15 languages.

The 10 most common words that wordfreq knows in 15 languages. Yes, it can handle multi-character words in Chinese and Japanese; those just aren’t in the top 10. A puzzle for Unicode geeks: guess where the start of the Arabic list is.

Continue reading

Luminoso Software Update 8/22/15

Work, work, work…that’s what we do to make our solutions work for you!

Take a look at a few of our most recent updates here, including:

  • Out with the term, “Correlation Scores,” and in with “Association Scores” (Analytics)
  • Association Scores are presented as numbers rather than as percentages (Analytics)
  • We’ve resized the Concept Cloud spacing to increase readability and support multiple character sets (Analytics)
  • Topic association charts in XLSX are now arranged from high to low (Analytics)

If you ever have any questions or comments, please don’t hesitate to reach out to us at

How I Stopped Being Afraid of AI

Using mechanical devices such as the wheel, the lever, the sail, the steam engine to reduce the burden of human labour is not a new idea. And, in most cases, these devices helped societies to vastly improve their quality of life and increase opportunities for its citizenry. However, using intelligent, independently thinking machines to help, enhance or substitute human labor and more importantly human thought, is a new phenomenon. The 2014 short documentary film by C.G.P. Grey called Humans Need Not Apply thoughtfully discusses this impact of automation on humans and paints a rather bleak future of workThere is an inherent unease about the kind of tasks intelligent machines are now performing while replacing human workers. This view is also shared by some rather influential figures in technology and science such as Ray Kurzweil and Elon Musk.

But, then, a glass half-empty glass is also half-full.

There is a different, more optimistic perspective on Artificial Intelligence – that there are vast, untapped, positive impact it can have on humans and on the nature of work. That AI is another tool, albeit more powerful and more impactful tool, but a tool nevertheless whose power is waiting to be harnessed by businesses. In fact, at Luminoso, we leverage AI to perform significant amount of tasks that help create a better product and a more robust product.

There are also other perspectives on AI. Geoff Colvin of Fortune, recently argued in his book, Humans are underrated, that people fearful that their jobs are at risk may be asking the wrong question as to what kind of work a computer will never be able to do.

Instead, Mr. Colvin proposes that we ask what are the activities that we humans, driven by our deepest nature or by the realities of daily life, will simply insist be performed by other humans, even if computers could do them?

We think the subject of AI and its application areas are so exciting that we have even proposed a panel at the upcoming 2016 SXSW Interactive titled – How I Stopped Being Afraid of AI. We think this will bring a fresh perspective to this raging debate. Please click here to know more and vote for us.

We are more likely to fear what we do not understand.  Get to know AI and what it can do for your business.