Dataflow Streaming Data Generator This pipeline takes in a QPS parameter, a path to a schema file, and generates fake JSON messages (sample messages used for load testing and system integration testing) matching the schema to a Pub/Sub topic at the QPS rate. JSON Data Generator library used by the pipeline allows various faker…

If you’re new to BigQuery, I suggest starting here and reading the official docs first. Export all your audit and billing logs back to BigQuery for analysis. I don’t know how many times this pattern has saved my butt. Don’t be lazy with your SQL. Avoid SELECT * on big…

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python Installation or Setup Installing pandas with Anaconda Installing…

Cloud Data Fusion is a fully managed, cloud-native data integration service that helps users efficiently build and manage ETL/ELT data pipelines. With a graphical interface and a broad open-source library of preconfigured connectors and transformations, Cloud Data Fusion shifts an organization’s focus away from code and integration to insights and…