The Metamorphosis of Knowledge Work

May 2, 2025

5 min read

Part one of a three-part series

Knowledge work. It was and still is the engine of the corporate world, but what really happened to the discipline? I had the pleasure of speaking with two titans of the Knowledge Work in the regulated industries, Lynn Epstein, COO at BlueberryLabs and Samuel Fryer, BlueberryLabs CEO, to get a handle on where knowledge work sits in modern organizations, and what lessons can be learned from the past when confronted with vast technological and regulatory shifts. Read this 3-part series to learn more.

A quick snapshot - the changing nature of Knowledge Work

Knowledge work has faced the brunt of technological changes over recent decades. Lynn describes how “earlier in my career, data storage was limited to hand-written files or emails, and internal teams often consisted of hundreds of people. Despite the advances in technology, I don’t believe we’ve reached systematic change in knowledge work”.

“Companies are built by people, and that hasn't changed… yet. What has changed is the people and skill sets of those who create and maintain the knowledge. For the past 50 years, organizations have been dependent on specialists to research, report, and analyze their data. Traditionally, these specialists would hold a degree in Library Science (or similar), but I believe we’re starting to see an evolution in this specialization. The number of people acquiring these degrees is estimated to have halved since its peak in the 1970s [1].”

Musings on the Decline in MLIS Degrees

So, what happened to the roles usually inhabited by talented knowledge workers? Lynn explains: “The truth is that today, the skill set of MLS and its sister qualification, the Master of Library and Information Science (MLIS), remains a highly valued and much-needed requirement. The degrees of years ago focused on how to conduct research; however, today they need to understand the digital skills and technology that are now available, in order to stay relevant. The analysis and knowledge management are the final steps, but you must possess all these other skills to perform the job effectively. I think that’s why we’ve seen the responsibilities formerly housed within these teams transfer and ripple out to other disciplines and departments within organizations. People are scared of change - that’s what’s holding us back. The truth is that the integration of AI enables humans to perform more in-depth analysis and engage in more meaningful work”.

As Lynn describes, AI brings a new horizon for knowledge work. Before AI, digitization had left its mark. Samuel says, “Lynn and I have been working together in this space for more than 10 years, and even in that relatively short period, we’ve seen a shift in the explosion of data. We’re creating 2.5 quintillion bytes of data each day [2]. It’s not just the quantity of data, it’s the acceleration of it - 90% of the world’s data is estimated to have been created in the last two years alone, with more than half of that comprising video, and approximately 12% comprising social media content. What’s harder to understand is how much of this data is being generated by organizations”.

“We began to notice that the size of the knowledge and research departments started to shrink in 2016. Departments had between 20 and 50 people. Very quickly, around 2018, the sizes of teams had shrunk drastically. That didn't mean to say that they didn't have a place or a purpose; it was that the requirement for lots of people to deal with manual records changed because of digitization”.

The interplay between knowledge work and AI

Digitization was a first step, enabling faster retrieval of information. More importantly, however, it laid the groundwork for what’s happening now. Sam explains: “If you think about it, most businesses today are comprised of digital files. Take your inbox and your emails, right? How many thousands of emails does the average professional have if they've been in the same job for 5 plus years? How many PDFs, PowerPoints, market research reports, research papers, or internal policies and procedures are sitting somewhere in your archive, with rich information but nothing to string them together?”

“As the archive continues to grow, AI enables people, when applied effectively, to perform data retrieval at scale. What doesn't go away is that you still need a human. If you're going to embed a large language model into your organization, the model must be applicable and appropriate for what you're trying to achieve.”

Quality data in, quality data out

So, we have all this data, but who decides whether it’s any good at solving very human problems? Sam answers: “Data science plays a fundamental role in knowledge management, and many of these teams now include individuals with these skills. Large and sophisticated organizations, like a leading law firm that Lynn and I met with recently, are investing millions with highly intelligent individuals trying to solve the problem of consistent data accuracy”.

“In addition to that investment, something that always rings true is this requirement and need for highly accurate data to not only train models, but also to improve the search, retrieval, production of documentation, or whatever that knowledge worker or information professional has designed to support their group’s decision-making.”

“The shift in knowledge management focuses on quality, structuring data inputs in a way that leverages AI solutions to their best potential. When you're running high-impact artificial intelligence models, if you train on something or if you're leveraging it in a way that is entirely irrelevant to the problem you're trying to solve, you're more likely to get hallucinations.”

Nothing but the Truth

The topic of data quality naturally led us to consider the role of trusted data and how it is perceived, both historically and currently. Lynn says that “sources that we used to consider as ‘trusted sources’ are now labelled premium data providers - most of which can be used via APIs, for a fee. Additionally, we now have a proliferation of open web sources, including government websites, for example. However, even the validity of a ‘trusted source’ is subject to debate…”

“Ground News, a publishing company, rates the ownership of the publishing company to allow you to see how they could have influenced individual articles. In practice, this means that when you're reading a ‘trusted source’, you have to know the political leaning of the ownership to understand it better.”

“That was unheard of back in the day: considering whether a particular publication had a Democratic liberal owner, for example. The need to indicate political bias has only become relevant with the advent of all this digital noise. I recall the time when we debated using anything besides premium content. Slowly, blogs and websites were included in knowledge databases, but even then, large financial and government organizations still have to put a warning or disclaimer against that information because it originated from a blog. That was an absurdity because a blog is just a new publishing format, but they were still scared that it wasn’t properly vetted and that a blog was simply someone's opinion. Now, we’ve lived through the day when our government organizations went from 100 trusted wire sources to sourcing large parts of the internet”.

Follow BlueberryLabs to follow the story of the changing nature of knowledge work. In Part 2, we explore the Wild West and its implications from a data governance perspective.

If you like what you’ve read so far, don’t hesitate to share this article.