In this quick demo, we are going to use a Python Script to automate GCP BigQuery Table View Creation for all tables in a dataset in a different GCP project. Before running the script you need to create Application Default Credentials to let Python script use them to authenticate. The user account or service account…
moreIn this quick start demo we are going to cover Terraform Integration/Unit Testing with Tests Terraform has introduced testing framework in 1.6.0 version release. Terraform test files should have .tftest.hcl or .tftest.json file extensions. Terraform automatically discover code in these files and execute them. We can have multiple .tftest.hcl or .tftest.json files. Each test file…
moreThis post describes some of the frequently used Google cloud GCP auth types and troubleshooting tips while authenticating with gcloud CLI and ADC. Following are the two frequently used methods and they both are different: 1. Authenticate to run gcloud commands using user credentials/service account key file. We can use gcloud CLI tool for Google Cloud…
moreRead below Google Cloud interview and certification sample questions to cross check your knowledge. 1.By default, how long PubSub Subscription will retain unacknowledged messages. 2. There is a requirement to do a DML experiment on a table, which modify a large number of rows, but analysts are not sure about the results. What options will…
moreWe can use Azure DevOps API to trigger an ADO pipeline. In this quick start demo , we will go through how to trigger an Azure DevOps pipeline with a Python script. I have already created an ADO pipeline called demo-pipeline. we are going to trigger this pipeline by using below Python script. Following parameters…
moreIn this quick demo, we are going setup Monitoring Alerts for GCP IAM service account key creation, meaning we will get alerts when some one creates a service account key. To setup alerts for IAM service account key creation, first we need to create log based metric for logs which contains iam.serviceAccountKeys.create IAM permission. I’m…
moreIn this quick demo, we are going setup Monitoring Alerts for GCP IAM Policy changes, meaning we will get alerts for GCP IAM Role assignments and removals to users, service accounts and groups. To setup alerts for IAM changes first we need to create log based metric for logs which contains setIamPolicy method. I’m going…
moreIn this quick demo, we are going to setup Application Performance Monitoring on Kubernetes using Elastic APM server and Elasticsearch and Kibana. What is Application Performance Monitoring(APM)? APM focuses on tracking the performance of an application. APM collects errors, traces, requests rates, latency times from applications to identify the source of the performance issues/bottlenecks. How…
moreIn this quick start demo, we are going to do Prometheus Blackbox Exporter setup on Kubernetes cluster to probe HTTP or HTTPS endpoints. What is Blackbox Exporter? Blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC. We can use Prometheus to scrap blackbox exporter metrics and create dashboards in…
moreIn this quick start demo, we are going to do Alertmanager setup on Kubernetes cluster to handle Prometheus alerts. We will use slack as alert receiver. What is Alertmanager? The Alertmanager handles alerts sent by Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty,…
moreIn this quick demo, we will create a service account and key file and store the key into secret manager and rotate the key. We are going to use Terraform code to do all these tasks. What is Service Account? GCP IAM offers two types of accounts for authentication and authorization, they are user accounts,…
moreGCP BigQuery offers dynamic data masking at column level. By using data masking we obscure columns that holds sensitive data for users, groups and service accounts, while still allowing access to data in those columns. What is Data Masking? Data masking is a method of modifying sensitive data while presenting it to an end user…
moreWhy Terraform Code Testing is important? We prefer to modularise Terraform code to avoid repetition and to combine multiple resources that are used together. These modules then get consumed in other terraform configuration blocks. Testing Terraform code is very important especially when we have Terraform modules, that are being used in many places to deploy…
moreWhat is Fluentd? Fluentd is a cross platform data collector, which is very useful for log collection, transformation and shipping to backends like Elasticsearch. It decouples data sources from log storage systems by providing a unified logging layer in between. In this quick start demo, we’ll use Fluentd to collect, transform, and ship logs from…
moreWhat is Elasticsearch? Elasticsearch is a distributed search engine based on Apache Lucene library. It is multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is the central component of the ELK/EFK(Elasticsearch, Logstash/Fluentd, Kibana) Stack. It is very useful for managing logs of IT systems and applications. What is Kibana?…
moreWhat is Grafana? Grafana open source software enable users to query, visualize, alert on, and explore metrics, logs, and traces wherever they are stored. Grafana provides tools to turn time-series database (TSDB) data collected by tools like Prometheus, into insightful graphs and visualizations. In this quick start demo, we are going to deploy Grafana application…
moreWhat is Node Exporter? A Prometheus Exporter is a small application that can fetch monitoring metrics from a target system and expose those metrics through a web URL to allow a Prometheus server to scrape those metrics. Node exporter exports hardware and OS metrics for *NIX kernels. By default, Kubernetes does not expose node level…
moreGoogle Kubernetes Engine (GKE) GKE service allows users to provision Kubernetes clusters to host containerized applications on Google Cloud infrastructure. GKE cluster consists of one or more Compute Engine Instances. With GKE, users can gain benefit of advanced cluster management features like load-balancing, node pools, automatic scaling, automatic upgrades, auto-repair, logging and monitoring. GKE clusters…
moreWhat is Grype? Grype is a vulnerability scanner for container images and filesystems. It’s processor is Anchore engine. Grype can scan a directory, container image or SBOM file. Grype can be used in a CI/CD workflow to find out security issues in git PRs or to check main/master branches with scheduled workflow runs. When using…
moreWhat is Trivy? Trivy is an open-source security vulnerability and misconfiguration scanning tool. It can scan Container images, Filesystems, Git repos and Kubernetes Cluster/Resources. Trivy can be used in a CI/CD workflow to find out security issues in git PRs or to check main/master branch with scheduled workflow runs. It supports table, json, sarif output…
moreWhat is CodeQL? CodeQL is a code analysis engine and query tool for running security vulnerability checks to find out vulnerabilities across a repository. CodeQL treats code like data. When we run CodeQL, it extracts a single relational representation of each file in the codebase to create CodeQL database. Then it runs queries against the…
moreIn this quick demo of how to handle BgiQuery resources using Python SDK, we are going to create Datasets/Tables/Snapshots and load data into tables. Import Python BigQuery Module & Initializing a client Create a BigQuery Dataset create a BigQuery dataset called “demo_dataset” in EU location with 2 days default table expiry. Note: if you don’t…
moreWhat is Profiling and GCP Cloud Profiler? Performance profiling allows to discover which unit of application, is misbehaving in terms of execution time and system resource usage. For example, if a function in an application takes 80% of time, it’s worth investigating and fixing it to improve overall app performance. Analysing performance of production application…
moreWhat is Cloud Development Kit for Terraform (CDKTF)? CDKTF allows users to produce Terraform configuration using programming languages such as C#, Python, TypeScript, Java, or Go. Users can use all Terraform providers and modules with CDKTF. CDKTF uses the Cloud Development Kit from AWS, which provides a set of language-native frameworks for defining infrastructure, and…
moreSquash Git Commits Git has a feature called squash, which allows users to combine multiple commits into a single commit in a branch. By squashing commits, we can keep repository’s commit history clean and neat. Users generally squash commits when merging branches into “master” or “main” branches Why Squashing? When a team is working on a…
moreGCP Virtual Private Cloud(VPC) network overview and deployment using Terraform. Creating primary and secondary subnets for Kubernetes clusters deployments and basic firewall rules.
moreTerraform count and for_each meta-arguments to create several similar resources. if condition usage with for_each and count meta-arguments.
moreWhen our feature git branch is behind main branch, we need to bring feature branch up to date with main branch before raising a PR to merge feature into main branch. Updating Feature Git branch with main branch Step-1: first commit your changes to your development branch and checkout to local main branch. Step-2: fetch…
moreSearch Indexes in BigQuery lets you easily find unique data elements without having to know the table schemas in advance.By using this feature we can create indexes on log tables and search for specific error codes or identifying the rows of tables that contain a specific user’s PII for GDPR reporting. How to create search…
moreDisplay List of Contexts Display the current-context Set the default context to cluster-1 Get all namespaces on the cluster Create a namespace(below command creates nginx namespace) Create resource(s) from yaml files(app.yaml file has resource definition) List Kubernetes pods on a cluster Get a pod’s YAML List Deployments on a cluster List Services on a cluster
moreFrom Linux Command line you can run below command to get the Public IP Address Run curl against ifconfig website. see the command output below:
moreA BigQuery table snapshot preserves the contents of a table at a particular time. You can create a snapshot of a current table, or create a snapshot of a table as it was at any time in the past seven days. A table snapshot can have an expiration. When the configured amount of time has…
moreWhat is Pseudonymization? Pseudonymization is a data de-identification procedure to replace personally identifiable information with artificial information.Pseudonymization makes data record less identifiable while allowing data analysis. Pseudonymization allows re-identification of data with additional information, unlike Anonymization procedure. With Pseudonymization, we can mask BigQuery Table data while sharing it with other users by creating BigQuery table…
moreWhat is Prometheus? Prometheus is an open-source system monitoring and alerting framework. Prometheus collects and stores monitoring metrics with the timestamp at which it was recorded. Prometheus Architecture Diagram Main Components Prometheus Server – scrapes and stores time series dataAlert Manager – handle alertsExporter – special-purpose exporters for services like MySQL, ngnix etc. Prerequisites: A…
more