A work breakdown structure (WBS) for big data analytics projects — Part 1

Ever wondered what activities are involved in a big data project? A work breakdown structure (WBS) helps in selecting technology/tools…

Ever wondered what activities are involved in a big data project? A work breakdown structure (WBS) helps in selecting technology/tools, defining project scope and estimating the effort.

Big data Analytics projects have the following unique features compared with traditional data warehouse projects

* Storage and Compute is decoupled
* Schema on write is not required.
* Supports polyglot storage ex: raw, object store, NoSQL, columnar.
* Supports polyglot consumption of data products

As such a big data analytics project scope, tools, tasks, and estimation is completely different from data warehouse.

Breaking down tasks for big data projects is hard and so is estimating.

The easiest way to start identifying tasks is to look at solution architectures of big data projects and in almost all of the cases, the design pattern is

Source --> Ingest --> Store --> Process --> Analyze > Consume

though each of these individual components can/will have multiple loosely coupled technologies/systems which makes it so hard in implementing and operationalizing.

The first part of blog post covers activities in the Ingest component of a big data analytics project.

Big Data Analytics Ingest — WBS

Link to the excel sheet

Disclaimer: All the opinions expressed are personal independent thoughts and not to be attributed to my current or previous employers.