The acute shortage of Data Science talent across the globe is no longer a secret.There are multiple reports that estimate the shortage of this talent, ranging from a few hundred thousand to millions! Several universities, technical/managerial institutes,MOOCs (massively online courses) and full time training institutes across the world have started training and certification programs. Some universities/institutes are even offering a Bachelor’s degree in AI and Data Science.So despite the educational and training support, why is there still a shortage of quality data scientists? Is it just a short-term shortage that would be overcome soon or there is something more to it? What does it take to produce a high-quality data scientist or become one?

Science by definition is repeatable, trainable, and consistent. Art is the opposite of science and one can only gain it with one’s own experience. The term Data Science was coined by the Harvard University to mean various disciplines of AI, such as statistical modeling, machine learning, deep learning, data mining etc. We use AI and Data Science interchangeably for the purpose of this article, though some leaders in this field differentiate them in subtle manner. How justified is the term “Data Science” in the context of current status of the fields it represents?

Those who have been a part of IT industry from mid 90s to 2015, would have seen a sudden mushrooming of engineering colleges providing IT education. Despite this, all organizations had to spend years of training on these graduates to make them productive on the job. Now we can similar issues with Data Science talent as well.

Even where curriculum matches with what industry needs, the focus is more on an application of tools, rather than understanding of underlying data, algorithm, its functioning and limitations. Moreover, many of the engineering aspects of model development and deployment to production are not covered in these training programs.

Hence, when faced with real life problems, even trained data scientists are unsure of finding the right solution for a given problem.

As mentioned earlier, AI consists of different sets of algorithms. The concepts/origins behind these algorithms are different, problem formulations are different, and their ability to work with a variety of data (relational, text, image/video, audio) is different.

There are many different ways these algorithms are classified in the literature. All the recent successes we have seen are from an AI branch called Deep Learning, which in turn is based on neural networks concept. Bayesian, Genetic, and Symbolic algorithms are some of other branches in this type of classification.

Another way of classification is supervised, semi-supervised, unsupervised, reinforcement, relational and transfer learning algorithms. Each of these classifications and sub-categories within, have bunch of different algorithms, and they are growing as we speak. There are many statistical and mathematical tools and techniques that also compliment to these machine/deep-learning algorithms. Most of the training and certification programs focus only on some of these algorithms, mathematical and statistical tools.

Historically enterprise (structured) data has been used in applications such as forecasting, classification, anomaly detection, optimization, intelligent search etc. These applications have been more prevalent in large global enterprises for long, but now every organization is adopting, hence the demand for these skills has increased dramatically.

In the recent past, several large fields such as computer vision (image and video data), Natural Language Processing (text and documents), Speech Recognition (audio) have gained traction with wider adoption of deep learning algorithms, while massive quantities of unstructured data is being generated by internet/social media, and high computing power of GPUs and TPUs.

While many of these are emerging into specialized fields, a few applications such as autonomous vehicles, digital assistants, robotics are more complex and leverage variety of data as well as combination of many types of algorithms. These latest applications are prevalent mostly in new age technology companies at this stage, but they are expected to gain traction in all business enterprises soon. This will push the demand to even higher levels.

All these algorithms are industry domain agnostic, problem formulation and model output validation require deeper functional knowledge. For example, if we want to understand whether the country’s people are getting richer or poorer year after year, we typically use per capita income, which is the average of overall population’s income. But if a handful of billionaires increase their income by small percentage, the overall average may also increase, irrespective of many people’s income at the bottom of the pyramid going down drastically.

Hence, in such cases instead of an average (mean), one should use median metric that divides the population into two groups (those above 50% and below 50% of the population). If median income has increased, then it indicates that a greater number of people’s income is rising.

Similarly using absolute number versus percentage could also make huge difference. There has been a huge debate on some of richer families spending a huge amount of money on their kids’ weddings. Spending INR 500 crore may sound big, but when it is seen as a percentage of a family’s overall wealth, it may be a very small fraction such as 0.01%, but in case of middle-class families the spend on a wedding as percentage of their net worth could be as large as 1-5%. It essentially means that middle class people spend more than rich people. So the whole perspective can change based on domain understanding and right metric usage.

These are simple real-life examples indicating the importance of domain knowledge to choose right metrics. In case of enterprises, it gets more complex and difficult to define the problem accurately, find right data, identify right metrics of success, select right models, and finally interpret the results coming out of the model. Each of them needs deep understanding of the functional/business knowledge.

Even after overcoming all the business domain hurdles, sometimes the results/recommendations that emerge from the models may not be politically acceptable, hence may not get published or implemented. The ongoing debate on latest GDP methodology, employment data and other public statistics in India is the classical example of this issue.

Similarly, the recommendations produced by these models are dependent on the training data available. If this training data contains some bias, then the resulting outcomes also could be biased. Facebook and Microsoft both faced this issue when they released their initial versions of digital assistants, which made some racially sensitive comments. Hence, for data scientists it is critical to understand business and social impact of the outcomes from these models, as well as ethical and legal use of these results, and convey the results to all stakeholders in an appropriate way to avoid serious reputational damage to the enterprise. A data scientist should be a good storyteller to achieve this.

Model design and development involves many software engineering skills. Most of the training courses don’t focus on this aspect at all. Hence, this experience has to be gained on real life projects.

As mentioned earlier, there is a plethora of different algorithms available to solve a given problem. Knowing the business problem and the type of solution required, one can shortlist a set of algorithms to be tested on available training data. It again comes with the experience in what algorithms this subset contains. Based on the results from this subset, a suitable algorithm and model is chosen for production use, after business validation.

In model development to solve a specific problem, data pre-processing (missing, inconsistent, duplicate and outlier data handling, scaling features to similar range etc.) and feature engineering (which features to be used in the model, which new features to be derived, encoding and aggregating categorical variables, binning continuous variables etc.), are the two critical activities. While there are many theoretical foundations, guidelines and best practices on how to accomplish these activities, both of them are more of an art than science. Each of these disciplines is also quite large and complex.

While deep learning algorithms do reduce the effort of this pre-processing and feature engineering to some extent, machine learning algorithms are highly sensitive to these activities.

Traditional IT solutions are deterministic in the sense there is only one pre-defined outcome, irrespective of the technology used. AI algorithms are inherently stochastic, i.e. the specific outcome could vary based on the initial starting point of the solution for the same algorithm/model, and it could vary across different algorithms for the same data. More often, we don’t aim for zero error or 100% accuracy while training the model. Determining the level of acceptable error or accuracy is again more of an art than science, and mostly determined by business comfort.

During the model training there are several parameters that need to be fine-tuned to get desired results. These parameters also change depending on the algorithm(s) and optimizers being used. So, it again comes with the experience, rather than just training.

A model’s performance on training data is not good enough, as the model might have been over fit and may not generalize well to new data. Hence, it needs to be validated by a different set of data that was not used during the training. If several rounds of that validation had to be done by tuning hyper parameters, it needs to be further tested by yet another set of data that was neither used in training, nor validation. If a model does well in training, validation as well as testing, then it is qualified for business validation.

Business validation consists of checking whether the model’s results are interpretable and explainable in business terms. If they are, then it needs to be seen whether the results are in line with well-established beliefs/facts or there are contradictions. Only when the results are explainable and do not contradict any of existing beliefs/facts, do business leaders become comfortable in moving the model for production use.

Machine Learning model deployment into production environment poses its own unique challenges, especially in enterprises where a large number of models are deployed at high scale.

Development platform (Python, R based packages, platforms) is usually different from production environment (Java, .net, C/C++ etc.). Either the model needs to be written in these languages or be converted to web APIs so that they can be called from other web applications.

In development we use only sample data, but in production, the model has to run with complete data, which can lead to latency and many other performance issues. In big data environments and/or real time deployments, these challenges get compounded significantly. To overcome these challenges, a model may have to be deployed in a distributed environment, and leverage special purpose processors such as GPUs (Nvidia), TPUs (Google), IPUs (Graphcore) etc.

For faster and effective deployments, DevOps processes (continuous integration, deployment or delivery) and tools can also be used. Models can also be deployed in a cloud environment, which brings in the complexity of integration with multiple environments.

Hence, successful deployment requires technology, infrastructure, performance engineering and IT operations skills, and Data Scientists also need to work with other IT professionals in the organization.

Historically, Business Intelligence, Statistical Analysis, Machine Learning applications are run in isolation from other business applications due to performance issues, and resulting insights are used to take decisions and actions manually.

With current real time enterprise, the decisions and actions can’t wait for manual intervention, rather they need to be integrated with other business applications closing the feedback loop. Also, Machine Learning models often need to learn in realtime, as new data comes in. Recommendation engines on e-commerce sites are a classical example of this.

These requirements pose great deal of technology challenges compounding other model deployment challenges.

Traditional business applications need changes only when there is a change to underlying business process. However, analytical models may need fine tuning or a complete model change, if the distribution of data changes from the distribution of the data that model was trained on. For example, when the model was trained, an enterprise was selling two products and customers were concentrated in one country. Over a period of time they added more products and the customer base expanded across the globe. Then the model needs to be retrained, if it has to forecast right sales.

There are many open source libraries that provide implementation of these AI algorithms. Python is the most preferred language, but R and Java are other popular languages being used in the industry. Prebuilt libraries of algorithms are available in each of these languages.

Having seen the complexity of the AI domain and the associated challenges in producing quality talent, there have been several developments that are trying to ease out these challenges and democratize the use of AI.To understand these developments, we can look back on how traditional IT technologies and applications have evolved over a period of time.

IT has evolved to hardware/software disciplines, and within software services specializations such as infrastructure services, application services, digital technology services, testing services etc. Similarly, AI will also emerge into specializations based on the types of data they deal with, and the types of problems they solve within each of the industry domains.

AI service providers could create specializations around Natural Language Processing, Computer Vision, Speech Recognition, Predictive Analytics, Business Intelligence, and Cognitive Process Automation etc.

As software tools evolved from programming languages to business applications, the AI field is also evolving to APIs, frameworks and platforms that simplify model development process. Microsoft, Google, and Amazon are in the forefront of developing such platforms and APIs.

APIs are pre-built and trained models to solve generic problems such as vision, text, speech etc. These can be further leveraged to solve specific problems such as reading characters (typed or hand written), face, label, logo recognition etc. from the images, entity recognition, sentiment analysis, content classification etc. from documents/text, and speech, language recognition, real-time streaming of voice etc.

The platforms being developed by these vendors are trying to automate all aspects of engineering activities as explained above. This is similar to what DevOps did to traditional IT engineering activities. This will help in improving the productivity of data scientists exponentially, and ensure consistency of results.

Then education and training institutes can also focus on these platforms, APIs, and functional specializations thus effectively bridging the gap between academy and industry. Some online training institutes have already started thinking along these directions, when they introduced focused courses around cyber security, digital marketing, HR analytics, and Financial Analytics.

Data Science or AI is a diverse field and has been evolving for more than a century. As AI technology evolves further, applications will catch up, and their adoption will increase across the enterprises, irrespective of size, industry or region. The main reason for the current talent gap is that Data Science is more of an art than science today. As tools and technologies evolve to more developer friendly platforms, APIs, and business analytics applications, this art will move towards science.

The main advantage of these APIs and platforms is that they come with pre-trained models, so that they can overcome critical bottleneck that enterprises face today, as well as the lack of sufficient data to train these models. Some of these APIs/platforms are also open for “transfer learning” so that they can be further trained with small data sets specific to the problem being solved to improve the accuracy. Accordingly, talent also will evolve into various specializations, and improve quality and productivity to meet the needs of the industry.

However, unlike traditional IT services, where all engineering disciplines, even non-engineering graduates could participate, to succeed in Data Science and AI, one needs a formal education in mathematics, statistics, machine learning and deep learning algorithms. Currently, leading organizations are looking for doctorate holders in this field, which may not be a compulsory requirement in future, as technical tools become more business user friendly.

Finally, this competency will no longer be the secret sauce of industry/technology leaders and democratization of AI and Data Science will happen sooner than we expect.