Large Language Models (LLMs) are quietly weaving their way into your organization, empowering employees with capabilities like automation, content creation, and data analysis. Yet, for many leaders, their actual use remains unclear. What tasks are LLMs performing? What impact are they having? Tackling these questions can be challenging, given the vast amount of data generated by numerous users, prompts, and content. But worry not, there’s a method to the madness!

AI

Continue Reading

Generative AI models like ChatGPT, DALL-E 2, and Bard are poised to transform businesses across New Zealand and Australia with their unprecedented speed and efficiency. They offer the potential to automate tasks across diverse industries, from marketing and legal to IT and finance.

AI

Continue Reading

The ever-growing volume and complexity of enterprise data pose a significant challenge: how to unify and leverage it for strategic decision-making. Data lakes and fabrics emerge as powerful solutions, offering a central platform to store, manage, and analyze all your data assets.

Azure

Continue Reading

Tired of hitting timeout walls when moving data between Snowflake and Azure Data Factory (ADF)? Fear not, for the Snowflake Durable Connector for ADF emerges as your champion!

This improved connector, a fork of Jeremiah Hansen’s work, tackles the infamous 230-second HTTP trigger limit head-on. No longer will lengthy Snowflake operations hold your workflows hostage.

snowflake

Continue Reading

Have you ever wondered what hidden themes lurk beneath a vast collection of text documents? Whether it’s news articles, customer reviews, or social media posts, understanding underlying topics can unlock valuable insights. This is where topic modeling comes in, and with the power of Apache Spark and its MLlib LDA Model, you can tackle large-scale datasets with ease.

spark

Continue Reading

With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala, the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies of a few minutes. In order to achieve this, however, it requires efficient and low latency data ingestion and data preparation in the Hadoop Distributed File System (HDFS).

To address this at Uber, we built Hudi (pronounced as “hoodie”), an incremental processing framework to power all business critical data pipelines at low latency and high efficiency. In fact, we’ve recently open sourced it for others to use and build on. But before diving into Hudi, let’s take a step back and discuss why it’s a good idea to think about Hadoop as the unified serving layer.

hadoop

Continue Reading