Best Practices Streaming Google Professional Data Engineer GCP

  1. Home
  2. Best Practices Streaming Google Professional Data Engineer GCP
  • Using composable transforms having business logic creates pipelines to easily maintain, troubleshoot and develop with.
  • Ensure they have a clear owner
  • Have a clear way to understand and document how changes to those transforms will affect all pipelines that use them.
  • Always use the name field to assign a useful, at-a-glance name to the transform.
  • Ensure that all transforms have versions and that all pipelines also have versions.
  • Ensure you have a consistent policy to deal with errors and issues across all of transforms.
  • End-to-end data pipeline testing should include lifecycle testing, particularly analyzing and testing all the different update/drain/cancel options.
  • In addition to the standard metrics, build custom metrics into pipeline as per
  • Create backups of pipelines
  • Create multiple replica pipelines
Menu