dotData Feature Factory

A Paradigm Shift for Enterprise Data

Feature Factory is a fundamental shift in how enterprise data science teams develop curated data and accumulate data know-how as reusable assets. Feature spaces and the ability to discover features through a data-centric, programmatic approach leads to enhanced collaboration, better efficiency, increased model quality, greater reusability, reproducibility, scalability, and transparency. Break down silos and capitalize on the wealth of information at your disposal.

“Feature engineering is powerful and scalable, even across tens of tables with billions of rows.”

Step 1

Prepare Tables as Dataframes

Connect to multiple data sources, data lakes, or data warehouses and ingest the data as Spark Dataframes in Python

Load data from modern cloud data marts (including Amazon Redshift, Google Big Query, Snowflake, MS Azure Synapse), traditional data warehouses (Oracle, Teradata, and MS SQL Server), and flat data sources (CSV files, Tableau Hyper files, etc.) via Spark Dataframe API
Automatic data type detection and data schema inference
Connect multiple data sources together by specifying Dataframe relationships
Define and configure temporal data relationships for automated temporal feature discovery

Step 2

Run dotData Feature Factory

Specify your target variable and the source tables as Dataframes you will use to build features. Define your search criteria and run dotData Feature Factory from your favorite Python IDE or notebook

Resolve data quality issues like illegal values, outliers, data canonicalization, missing values, target label mapping, and more.
Explore millions of feature hypotheses – including numeric, categorical, time-series, text, and even geospatial data.
Resolve feature over-fitting, feature collinearity, feature drifts, and feature redundancy based on dotData’s proprietary algorithms.
Custom feature primitives and search criteria to add your own domain features into the feature exploration space.

Step 3

Discover Features & Insights

Explore and evaluate discovered features interactively from Python

Feature leaderboard (feature list) that surfaces features that are the most relevant and correlated with your target variable
Understand each feature’s business value and construction via an easy-to-understand auto-generated explanation and feature blueprint diagram
Select your preferred features based on various feature metrics like correlation, feature-wise AUC, permutation importance, feature locality, popularity, and more.
Extract feature tables as Dataframe and visualize each feature using the built-in visualization tool or any Python visualization library you like.

Step 4

Extract & Iterate Feature Discovery Experiments

Iterate feature discovery experiments to derive better quality and higher-order features. insightsExplore, optimize, and tune features interactively. Choose which features to extract for further analysis, modeling, or reporting from within Python

Edit feature descriptors (definitions) to customize discovered features and leverage your domain expertise.
Natural interface to add new datasets and run new experiments. Combine features from multiple experiments with different granularity.
All steps and feature space details are reported without any black box
Modularized execution allows you to run your experiments from any intermediate steps and iterate them faster

Step 5

Deploy Your Features Into Production

Populate feature stores and continuously update features in production applications

Ingest features and metadata (feature explanation, feature statistics, feature schema) into any feature stores (Databricks, Snowflake, AWS SageMaker and more) and enhance your ML models
Automatic feature pipeline generation with fully specified query statements for reuse and eliminate error-prone manual feature query implementation.
One command deployment of feature pipelines into dotData Ops. Continuously recalculate features values with the newest data and monitor feature quality and drifts.

Databricks

Seamlessly integrate dotData Automated Feature Discovery into Databricks Python workflows and ML experiments.

Azure Synapse

dotData Feature Factory is available as a Python library ready for use in Synapse.

Amazon EMR

Install dotData Feature Factory in your AWS EMR instance to accelerate feature discovery for your data science team.

Pip Install

Quickly deploy dotData Feature Factory via pip-install – even on your own personal laptop.

Product Features

Multi-Source, Multi-Table Feature Engineering

Quickly connect to diverse data sources to enrich feature discovery and unlock iterative feature additions from new sources.

No SQL
Required

Leverage dotData Feature Factory to interact with data through Dataframes and generate entity relationships and features using familiar Python commands and syntax.

Automated
Data
Wrangling

Avoid time-consuming and error-prone data wrangling, dotData will automatically cleanse, align, and prepare data for feature discovery.

Time-Series Features, Without Leakage

Automatically generate and validate multidimensional time-based features, including holidays, lags or delays, periodicity, seasonality, and more.

Build Reusable Feature Discovery Assets

Record every data and feature transformation step into your Analytic Database and turn your “know-how” into reusable assets for your organization.

Automated Feature Discovery at Enterprise Scale

Generate and score millions of features from complex tables, relationships, and billions of rows.

Insights and Explainable AI

Produce actionable business insights and explainable features from natural language descriptions and blueprinting of discovered features from source variables.

Production-Ready Feature Pipeline Generation

Automatically generate feature pipelines with full SQL code from source tables, ready for production use.

How SMBC Discovered 2,000,000 new features

When SMBC, one of the world’s largest banks, wanted to get the maximum value from their feature engineering investment, they turned to dotData. Download the case study and read how the went from 2,000 features a year to over 2,000,000.

Are You Ready for Feature Factory?

Take our five-minute self-assessment to see if your data and organization could benefit from dotData’s Feature Factory revolution.

Take Our 5-Minute Assessment

Want to Learn More?

Book a 1:1 session

Book a 27-minute discovery session with a specialist to explore if dotData Feature Factory is right for your business

Connect with a Specialist

Are You Ready?

Take our 5-minute self-assessment to see if your company can benefit from dotData Feature Factory – and calculate your ROI

Take Our Readiness Assessment

Take it for a Spin

Take Feature Factory for a spin with a free thirty-day trial of dotData Feature Factory