I used to work at a massive data analytics company that had access to billions of users’ online activity.
We were able to use this data to predict the behavior of millions of users.
It was a big deal.
The idea was that it allowed us to predict what users would do based on their browsing history and their friends’ activity.
But, over time, it became clear that this approach had its limits.
It wasn’t as good at predicting who was likely to engage in a particular behavior.
We could not predict the next person that we’d see in a chatroom.
The algorithms couldn’t work on millions of people at once.
We couldn’t predict who would visit our website.
And so, we ended up dropping the use of the big data model altogether.
We started to rely more on natural language processing to solve the problem.
We stopped using large data sets that had millions of data points and started using a handful of words.
It’s not an enormous difference, but it’s a big difference.
What about the next generation of data analytics tools?
We’re still trying to figure out the best way to use the data.
In the long term, there are two big questions we have to ask ourselves: Is it better to be able to predict who is likely to click on certain keywords or who will open a specific tab, or to do something like predict who will visit your website?
I think the next step is for companies to make that tradeoff.
If you’re using a huge data set, and you’re just getting a small fraction of that data, you’re going to have to make some hard choices.
And if you’re working with a small amount of data, it’s going to be easier to make a tradeoff between accuracy and power.
If we start to rely on the natural language model, it becomes easier to think about the tradeoffs.
If we use a few words and it makes sense, we can just drop that model and go with the language model.
If it’s just too noisy, it can be hard to make tradeoffs between power and accuracy.
I would love to see companies like Twitter do a more careful analysis of their data.
The bottom line is that we need to stop relying on the big-data models that rely on word counts, word counts alone.
We need to have better tools that can learn from the data and predict how people will behave based on how they’re interacting with the world.
