Skip to content
forked from finos/datahub

DataHub - Synthetic data library

License

Notifications You must be signed in to change notification settings

mcleo-d/datahub

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DataHub

DataHub logo

Synthetic data generation

DataHub is a set of python libraries dedicated to the production of synthetic data to be used in tests, machine learning training, statistical analysis, and other use cases wiki. DataHub uses existing datasets to generate synthetic models. If no existing data is available it will use user-provided scripts and data rules to generate synthetic data using out-of-the-box helper datasets.

Synthetic datasets are simply artificiality manufactured sets, produced to a desired degree of accuracy. Real Data does play a part in synthetic generation, all depending on the realism you require. The product roadmaps details out the functionality planned in this respect.

DataHub's core is predominantly based around pandas data frames and object generation. A common question: Now that I have a data frame of synthetic-data, what do I do with it? The Pandas library comes with an array of options here - so for the time being sinking to databases is out of the scope of the core library, however see that examples in the test folder for some common patterns.

note As we build out a config based synthetic spec generator, we will bring this back into scope - please see our roadmap/issue list and get involved in the discussion.

Key documents

  1. For information on how to get started with DataHub see our Getting Started Guide
  2. For more technical information about DataHub and how to customize it, see the Developer Guide
  3. For a high-level road map see Road Map
  4. This project uses Gravizo for all diagrams and charts as highlighted in DataHub Issue 41.

Overview of Synthetic data

  • Synthetic data is information that's is artificially manufactured rather than generated by *real-world events.
  • Synthetic data is created algorithmically, and can be used as a stand-in for  test datasets of production data
  • Real data does play a part in synthetic data generation - depending on how realistic you want the output

License

Copyright 2020 Citigroup

Distributed under the Apache License, Version 2.0.

SPDX-License-Identifier: Apache-2.0

About

DataHub - Synthetic data library

DataHub - Synthetic data library

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 86.5%
  • JavaScript 10.6%
  • CSS 2.9%