Nothing new under the Sun
What has been will be again, what has been done will be done again; there is nothing new under the sun.
Is it valid in the era of AI and the technical revolution? In the time when state of the art changes daily and unseen things emerge regularly?
I think so.
Form over substance
For example, what was called “search engine with recommendations” ten years ago today is called the “user preferences prediction algorithm.” What was back then called system administrator, developer and good communicator today is called “DevOps.”
Mathematical analysis (especially function optimization) and some methods known for decades or centuries thanks to automation and brute force processing power became “machine learning methods.”
It’s the clear impression that every technology out there tries to get under the umbrella of “Big Data” or “AI” because of the market benefits you receive when you make it there 1.
But is this just a cynical marketing or maybe something else?
You have no AI in your company and this is fine
Some time ago I was working with a company that aimed to produce some advanced AI solution. Sounded great, but as soon as I got there I realised that they were reinventing the wheel, starting with a custom knowledge database, a custom search engine with lexical features etc. although there were already existing technologies that do those things (for example Neo4j, Grakn, ElasticSearch). Also, they were aware that those things exist but instead decided to pursue their own custom solutions because doing things your own way was perceived as an innovative approach.
I kind of understand that, I was doing such things at the beginning of my career, when I wasn’t experienced enough to finally know that you should “stand on the shoulders of giants” (yes, I know it’s a cliché, but it’s actually true) to achieve best results (or any results at all in some cases). At this point, I thought my custom tailored solutions would do better. Later I learned some humility.
I perceived the situation as a lot of wasted time and potential. Soon I wanted to make a point. We were using PostgreSQL as a content database in the CMS, which was arguably the simplest element of the entire system. I just enabled PostgreSQL full text which took me a few hours and did a small demo that did show it gives better results than the custom search engine (when it comes to searches as it does not have much NLP related features) that was being developed for weeks.
This simple thing helped to open eyes, and soon we started replacing custom “AI” solutions with already existing technologies and quickly recreated everything really more rapidly with the same set of features, better quality, and stability, which also allowed us to move much faster and eventually do some meaningful and actually innovative AI solutions.
…but at the beginning of the process there was also a “problem”.
There was no “AI” and “technical innovation” in an “AI” startup! We were just using the same old tech that was out there for years.
There were clients, working solution existed, but other fear started to emerge at the management level: “what will we tell the investors now? How will we do advertising? We have no custom machine learning models or anything like it”.
Well, again I did not see the problem with that and just suggested the product itself is the actual innovation (with its set of features, UI and overall functionalities). Combining technologies in a way that nobody else did and creating a new idea “is” the innovation. Being innovative does not mean to reinvent the wheel, or “doing things my way
Soon we actually started to have some innovations in the field by allocating efforts not to “recreate” or “rethink” and just to add other bricks on top of the proven technical stack. The initial shock and realization that we solved problems without using machine learning or any kind of big data methodologies made a huge impact on how I started perceiving the current technical ecosystem.
Fear of missing out
Reality is that there are no revolutions in tech (and I suppose there never will be) and we still just continue working on the same problems we did 10 or 20 years ago (with a massive difference in the quality of course). There are clever ideas emerging here and there, some of them gain unbelievable hype, but still, we are moving incrementally, with “revolution” proclaimed by marketing departments at every tiniest step.
Well, those “revolution” proclamations are getting tiresome, to be honest.
I observed (actually I believe most of us did at some point) how good self-presentation skills and talking bullshit 2 were enough to sell or at least get a free trip around the world to speak as an ‘expert’ on conferences. If you promise your client to do something without Machine Learning or some other hype words because there is just no need for that - you might just simply be ignored - someone who promises bigger things for more money (then does it the same way anyway) - will get the job 3.
Is this just a giant bubble of FOMO (Fear Of Missing Out) on a grand scale? No, that’s impossible… right?
I came to think that it does not necessarily mean that companies are evil and cynical - they are just afraid of competition, investors, changes or only wholeheartedly believe that you can’t do innovation without Big Data.
Where to go from here?
Well, there is nothing new under the Sun. Technical and marketing realities change or just shift, but we - people, never change. Too ambitious, sometimes too afraid or too proud to admit the truth about what we are actually doing or working on.
We should just face it and jump off the hype train. If we want the reason to prevail we should focus on good old values - learn to tell the truth and stay transparent in front of clients and ourselves, start using adequate tools to do the job to not waste time and resources.
Also, do not be afraid to express your concerns and put some pressure on the management or marketing departments to remove “AI” or “Big Data” hype phrases from their materials, if those technologies are not actually used in the company, to not spoil the technological ecosystem.
Something that Big Data and Machine Learning will never do for us. In the end, as always, everything stays in our hands.