OLAP stands for Online Analytical Processing and is a method in which data from multiple sources can be used together, allowing companies to group data from websites, applications, internal systems, and more together for analysis.

“For example, a retailer stores data about all the products it sells, such as color, size, cost, and location. The retailer also collects customer purchase data, such as the name of the items ordered and total sales value, in a different system. OLAP combines the datasets to answer questions such as which color products are more popular or how product placement impacts sales,” AWS wrote in a post explaining OLAP.

Key features of Apache Pinot include low-latency queries, the ability to handle hundreds of thousands of concurrent queries per second, batch and streaming ingestion, versatile joins, rich indexing options, and more.

It was first created at LinkedIn in 2013 because the company wanted to provide its users interactive analytics, but with the amount of data LinkedIn had already amassed at that time, it was struggling to find something that could scale at the level it needed.

“Pinot was born as an answer to our problems, a web-scale real-time analytics engine designed and built at LinkedIn. Pinot enables us to slice, dice and scan through massively large quantities of data in real-time across a wide variety of products,” said Praveen Neppalli Naga, engineering manager at LinkedIn at the time, wrote in a blog post when the project was first announced.

It powers 25 of LinkedIn’s user-facing features such as Who Viewed My Profile, Company Follow Analytics, Jobs Analytics, and more, as well as over 30 of the company’s internal tools, such as its A/B testing platform.

In 2018, Apache Pinot joined the Apache Software Foundation as an incubator project and became a top-level project in 2021.

Since its creation it has been adopted by a number of major companies, including Robinhood, Slack, Stripe, Target, Uber, and Walmart.

The most recent release is 1.1, which came out in March, adding features such as vector index support and multi-stage query engine improvements.

Looking forward, some of the things the project maintainers are working on in 2024 include making V2 on-by-default, enabling column null storing by default, full PostgreSQL compliance, pagination, and continuing ease-of-use updates such as improved documentation, more user friendly error messages, and more.

The post Apache Pinot – SD Times Open Source Project of the Week appeared first on SD Times.

The tool, called OpenHouse, has been in use at LinkedIn for the past year. The company has 3,500 OpenHouse tables in production currently. 

It was designed to offer self-service management of tables in open data lakehouses. According to LinkedIn, it was running into challenges internally because it didn’t have a good managed experience for running data lakehouses, which meant that end users were often dealing with low-level infrastructure concerns, which took time away from time they should have spent working on their products.

“Overall, since rolling out OpenHouse, we’ve seen drastic reduction in operational toil for data infra teams, improved developer experience for data infra customers, and enhanced governance for LinkedIn’s data,” Sumedh Sakdeo, senior staff software engineer at LinkedIn and creator of OpenHouse, wrote in a blog post. 

OpenHouse consists of a declarative catalog and a suite of data services. The catalog includes definitions of tables, their schemas, and associated metadata, and it integrates with Apache Spark. It supports standard syntax such as SHOW DATABASE, SHOW TABLES, CREATE TABLE, ALTER TABLE, SELECT FROM, INSERT INTO, and DROP TABLE. The catalog is also where users can specify retention, replication, and sharing policies for the table. 

Another key element of OpenHouse is that it reconciles a table’s observed state and its desired state, and this is where invoking data services comes in. Data services are responsible for orchestrating table maintenance jobs. 

According to LinkedIn, the goal was always to open source the project at some point, and therefore it was designed to provide pluggability with storage, authentication, authorization, database, and job submission services.  

“Now that we’ve reached the open sourcing milestone, we invite you to explore OpenHouse and provide us with your valuable feedback. We’re keen on collaborating with users to understand how OpenHouse performs within different environments, whether it’s integrated into cloud infrastructures or adapted to preferred table formats,” Sakdeo wrote.

The post SD Times Open-Source Project of the Week: OpenHouse appeared first on SD Times.

The DPH Framework describes “the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn,” Max Kanat-Alexander, principal staff software engineer at LinkedIn, and Grant Jenks, senior staff software engineer at LinkedIn, wrote in a blog post.  

The Framework can be adapted by organizations looking to implement systems to improve productivity and developer satisfaction. 

It describes the metrics LinkedIn follows, how it chose what to measure, and provides insights into why some metrics are better than others. For example, some of the metrics in place at LinkedIn include Developer Build Time, which is the time developers wait for their builds to finish; Net User Satisfaction, which measures how happy developers are with the internal tools they are using; and Code Reviewer Response Time, which measures how long it takes to a review to respond to code updates.  

The DPH Framework also recommends creating Developer Personas to better understand developers by categorizing them into groups based on their workflows. This enables leaders to think about the priorities separately for each group. 

There are also guidelines for teams who are creating feedback systems, and guidelines for quantitative metrics. 

Finally, the Framework ends with a set of example metrics that companies can base theirs on. 

“Now more than ever, developers are navigating so much change and new opportunity in this new era of Generative AI, so ensuring teams have the systems, processes, metrics and feedback systems to be successful is paramount. Our goal with this release was to offer an answer to one of the main questions we hear asked across the software industry, “How can I help my software development teams be more efficient, more effective, and happier?” We found that the best way to answer this question is through data, usually meaning metrics and feedback systems,” Kanat-Alexander and Jenks wrote.

The post SD Times Open-Source Project of the Week: Developer Productivity and Happiness Framework appeared first on SD Times.

According to LinkedIn, it was experiencing bottlenecks in its machine learning workloads that were caused by the need to read multiple terabytes of input data. AvroTensorDataset can speed up preprocessing of data by multiple orders of magnitude, according to the company.

The tool was built internally at LinkedIn, and it wanted to open-source the project so that others could experience the large performance boosts to training workloads. It has been in production for over a year already at LinkedIn. 

LinkedIn says that with this tool it has been able to improve processing speed by 162x compared to existing solutions and has decreased overall training time by 66%

“ATDSDataset is LinkedIn’s solution to efficiently read Avro data into TensorFlow. Through multiple performance enhancements, we were able to speed up I/O throughput by orders of magnitude over existing Avro reader solutions. Our team at LinkedIn worked closely with the TensorFlow I/O community to open-source this feature, and we hope that by open-sourcing it, the TensorFlow community can also benefit from these performance enhancements,” Jonathan Hung, staff software engineer at LinkedIn, wrote in a blog post. 

The post SD Times Open-Source Project of the Week: AvroTensorDataset appeared first on SD Times.

Feathr works to address the problem of overburdening teams with the increasing costs of maintaining their feature preparation pipelines. 

Feathr works as an abstraction layer that provides users with a common feature namespace for defining features and a common platform for computing, serving, and addressing them “by name” from within ML workflows. 

Feathr also brings advanced support for feature transformations which enables users to experiment with new features based on raw data sets. 

Additionally, Feathr’s abstraction creates producer and consumer personas for features. Both roles can be played by the same engineer, producing and consuming features for their own project.

Feathr users have reported a reduction in the time required to add new features to model training workflows as well as an improved runtime performance.

According to LinkedIn, replacing application specific feature preparation pipelines with Feathr has reduced engineering time by as much as 50% and also allows for feature sharing between similar applications. 

The team at LinkedIn is continuing to develop the ecosystem around Feathr with new infrastructure and tools, including the enabling of CI/CD for feature engineering. With this, customers will be able to create upgraded versions of widely-shared ML features that will then be tested against existing models that depend on that feature.

The post LinkedIn announces the open sourcing of Feathr appeared first on SD Times.

The project was recently open-sourced by LinkedIn and was used at the company to improve member experience in products such as People You May Know (PYMK), newsfeed ranking, search, and job recommendations, as well as customer-facing products within sales and marketing.

The FastTreeSHAP open-source package implements the algorithms FastTreeSHAP v1 and FastTreeSHAP v2 that make the packages 1.5x and 2.5x times faster than TreeSHAP, respectively. 

Parallel multi-core computing is fully enabled in the FastTreeSHAP package and it contains the same API as the TreeSHAP implementation in the SHAP package, with the exception of three additional arguments which are easy to tune in practice.

“SHAP calculates the average impact of adding a feature to the model by accounting for all possible subsets of the other features. In contrast to other approaches, SHAP has been justified as the only consistent feature attribution approach with several unique properties (local accuracy, missingness, and consistency), which agree with human intuition,” Jilei Yang, Humberto Gonzalez, Parvez Ahammad from LinkedIn wrote in a blog post. “Due to its solid theoretical guarantees, SHAP has become a top model interpretation approach in industry.”

After looking into many TreeSHAP use cases, LinkedIn found that despite its algorithmic complexity improvement, computing SHAP values for a large sample size or a large model size still remained a computational concern in practice. This resulted in the key improvements in the FastTreeSHAP versions. 

The post SD Times Open-Source Project of the Week: FastTreeSHAP appeared first on SD Times.

The company launched its global skills initiative last summer and has since helped over 30 million people learn new digital skills. 

“A new generation of 21st century infrastructure calls for new investments that will broaden access to the digital devices and broadband connectivity that have become the lifeblood of commerce, healthcare and education. And it similarly calls for a renewed commitment to the education and skills that a new generation of technology has made essential for people’s personal progress,” Brad Smith, the president of Microsoft said. 

Microsoft said that there are new challenges to meet this goal with a more diverse workforce that now confronts a wider array of educational needs and opportunities. Many industries require that people fill gaps in their current skill sets, which requires employers and employees to become better at identifying those necessary skills. 

People also want to learn new skills as 30.7 million people in 249 countries and territories took advantage of the free access on LinkedIn Learning to more than 500 courses that teach skills for in-demand roles. 

Also, people are good at identifying the right skills that are needed for most in-demand jobs including the extensive uptake in courses that address horizontal skills including the three most popular LinkedIn Learning pathways for critical soft skills: diversity, inclusion and belonging; and digital transformation. 

LinkedIn announced that it is working on a widely accepted skills taxonomy in the LinkedIn Skills Graph to create a common skills language for individuals, employers, educational institutions and government agencies.

LinkedIn will pull data from its Economic Graph to help people identify skills that map to in-demand jobs of potential interest. It also will extend to the end of 2021 the free course offerings from the skilling initiative on LinkedIn Learning. 

The new Skills Path will bring together LinkedIn Learning courses with Skill Assessments to help recruiters source candidates based on their proven skills. 

Microsoft is also encouraging young students to get involved with these digital skills by launching a Minecraft: Education Edition, a beginner skillmap in Microsoft MakeCode and a new  Advanced Placement Computer Science Principles with Microsoft MakeCode curriculum using MakeCode Arcade for high school students.

Microsoft said it is also strengthening its work in Microsoft Philanthropies to advance digital equity through nonprofit partnerships that serve those hit the hardest by the COVID-19 downturn, including Black and African American communities in the United States, according to a blog post that contains additional details about all of the new initiatives from Microsoft and LinkedIn.

The post Microsoft unveils next steps of its global skills initiative appeared first on SD Times.

“Passing the arguments through the CLI becomes a producer and consumer problem: on the workflow generation side, you need to produce a set of arguments which are passed to the CLI to launch the jobs; on the other side, the launched jobs would consume the arguments passed from the CLI,” Jun Jia and Alice Wu, senior staff software engineers at LinkedIn, wrote in a post.

The principles of the Smart Argument Suite are:

• It should be simple. The suite makes it as easy as defining an argument container object and passing it through a function call. 
• It should be safe. The tool has a verifiable and testable systematic process with certain safety guarantees.
• It should be human-friendly. According to the team, the tool should be easy for humans to inspect or debug on the serialized form. 
• It should be extensible. Users should be able to extend the support to the argument container classes if desired. 

“It’s a very common scenario that an AI solution involves composing different jobs, such as data processing and model training or evaluation, into workflows and then submitting them to an orchestration engine for execution. At large companies such as LinkedIn, there may be hundreds of thousands of such executions per day, submitted and executed by multiple teams and engineers,” Jia and Wu wrote. “Any improvements in the tools used by machine learning engineers lead to significant improvements in productivity, which highlights the need for robust productivity infrastructure to support machine learning engineers.”

The solution has been released to PyPl and tested with LinkedIn’s other open-source AI solutions including GDMIx and DeText. 

Going forward, the team plans to add escaping to make the serialization safer and expand beyond the language boundaries. 

The post SD Times Open-Source Project of the Week: Smart Argument Suites appeared first on SD Times.

LiFT is a Scala/Spark library that enables the measurement of fairness, according to a multitude of fairness definitions, in large-scale machine learning workflows. It has broad utility for organizations who wish to conduct regular analyses of the fairness of their own models and data, according to the company. 

“News headlines and academic research have emphasized that widespread societal injustice based on human biases can be reflected both in the data that is used to train AI models and the models themselves. Research has also shown that models affected by these societal biases can ultimately serve to reinforce those biases and perpetuate discrimination against certain groups,” AI and machine learning researchers at LinkedIn wrote in a blog post. “Although several open source libraries tackle such fairness-related problems, these either do not specifically address large-scale problems (and the inherent challenges that come with such scale) or they are tied to a specific cloud environment. To this end, we developed and are now open sourcing LiFT.”

The toolkit can be deployed in training and scoring workflows to measure biases in data, evaluate different fairness notions for ML models, and detect statistically significant differences in their performance across different subgroups, the researchers explained. 

The library provides a basic driver program powered by a simple configuration, allowing quick and easy deployment in production workflows. 

Users can access APIs at varying levels of granularity with the ability to extend key classes to enable custom computation. 

The currently supported metrics include different kinds of distances between observed and expected probability distributions; traditional fairness metrics (e.g., demographic parity, equalized odds); and fairness measures that capture a notion of skew like Generalized Entropy Index, Theil’s Indices, and Atkinson’s Index.

The solution also introduced a metric-agnostic permutation testing framework that detects statistically significant differences in model performance – a testing methodology that will appear in KDD 2020. 

Metrics available out-of-the box (like Precision, Recall, False Positive Rate (FPR), and Area Under the ROC Curve (AUC)) can be used with this test and with the CustomMetric class, users can define their own User Defined Functions to plug into this test. In order to accommodate the variety of metrics measured, LiFT makes use of a generic FairnessResult case class to capture results

“While a seemingly obvious choice for comparing groups of members, permutation tests can fail to provide accurate directional decisions regarding fairness. That is, when rejecting a test that two populations are identical, the practitioner cannot necessarily conclude that a model is performing better for one population compared with another,” the team wrote. “LiFT implements a modified version of permutation tests that is appropriate for assessing the fairness of a machine learning model across groups of users, allowing practitioners to draw meaningful conclusions.”

LinkedIn stated that the release of its toolkit is part of the company’s R&D efforts to avoid harmful bias in its platform, alongside Project Every Member and ‘diversity by design’ in LinkedIn Recruiter. 

The post The LinkedIn Fairness Toolkit launched to measure fairness in large-scale AI apps appeared first on SD Times.

The initiative will focus on the use of data to identify in-demand jobs and the skills needed to fill them. It will provide free access to learning paths and content, as well as low-cost certifications. 

The company also announced a new learning app in Microsoft Teams to help employers upskill new and existing employees. 

Additional details on the initiative are available here.

Compuware announces new features to automate shift-left testing
The new capabilities will further automate and integrate test data and test execution to help IT teams. 

The new integration tightly couples the Topaz for Enterprise Data solution with the Topaz for Total Test solution enabling test data setup to be directly embedded into automated testing.

With this integrated process, once the test data is loaded, it can also be privatized to protect personally identifiable information (PII). Automating the privatization as part of the continuous test reduces the risk of a security breach, according to Compuware. 

Additional details on the new features are available here.

Git client Tower 5 for Mac released 
The new version introduces the ability to show or hide whitespace changes with a single click, as well as the ability to see the diff of a new untracked file, and stage/unstage/discard parts of the changes. 

It also includes the ability to highlight inline changes to make it convenient to spot changes and customizable diff themes. 

“With version 5, we are excited to bring some of the most requested features to our diff viewer – making it much more powerful,” the developers behind the project wrote in a blog post.

Yellowbrick Data announces new features and a new cloud disaster recovery service
Yellowbrick data announced the general availability of its Cloud Disaster Recovery service, as well as new database replication and enhanced backup/restore features that remove cost and complexity from the business continuity strategies of Yellowbrick customers.

“We’re complementing the existing business continuity functionality inside a single Yellowbrick Data Warehouse–including support for high availability, erasure coding, and fault tolerance–with new features that provide continuity across databases and locations in a low-cost, low-effort way using the power and flexibility of hybrid cloud architecture,” said Nick Cox, the head of products at Yellowbrick. “That is essential for business-critical applications.”

Cloud Disaster Recovery supports backups at near-line speed, allows for incremental backups, and provides transactional consistency (ACID) of restored data.

Additional details are available here.

JFrog launches first security-focused, immutable chart repository for Helm

JFrog launched ChartCenter, a free, security-focused central repository of Helm charts to help developers access consistent versions of any publicly available Helm chart. 

“We are creating a true unified and open repository that allows developers to set up a single, trusted location to consume immutable charts from every chart creator, together with important security information and metadata attached to these charts,” said Yoav Landman, the CTO and co-founder of JFrog.

JFrog ChartCenter includes all major Helm charts currently available across the web today, along with important security information and metadata around dependencies and application versions. Additional details are available here.

The post SD Times news digest: Microsoft and LinkedIn launch new initiative to reskill workers, Compuware announces new features to automate shift-left testing, Git client Tower 5 for Mac appeared first on SD Times.