# Hands-On Python for DevOps Ankur Roy [*Chapter 1*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_01.xhtml#_idTextAnchor015), *Introducing DevOps Principles*, will help you understand the concepts behind DevOps and how they are important in improving the productivity of your workload. [*Chapter 2*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_02.xhtml#_idTextAnchor038), *Talking about Python*, covers the core philosophical principles behind DevOps and how these principles define the approach that you take toward creating a solution. [*Chapter 3*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_03.xhtml#_idTextAnchor061), *The Simplest Ways to Start Using DevOps in Python Immediately*, provides a quick look at Python and the principles behind it, along with how these principles align with the principles of DevOps. [*Chapter 4*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_04.xhtml#_idTextAnchor077), *Provisioning Resources*, explores the easiest ways to use Python such that it could enhance your DevOps workload. [*Chapter 5*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_05.xhtml#_idTextAnchor099), *Manipulating Resources*, covers using Python as a means to provision resources in a sustainable and accurate way for your DevOps workload. [*Chapter 6*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_06.xhtml#_idTextAnchor129), *Security and DevSecOps with Python*, looks at modifying resources that already exist using Python in order to automate updates and mass modify replicated resources. [*Chapter 7*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_07.xhtml#_idTextAnchor142), *Automating Tasks*, explores using Python to automate common DevOps tasks and increase productivity for users by saving time on repetitive tasks. [*Chapter 8*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_08.xhtml#_idTextAnchor155), *Understanding Event-Driven Architecture*, covers using Python as a way to connect different systems to system architectures using event-driven concepts. [*Chapter 9*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_09.xhtml#_idTextAnchor165), *Using Python for CI/CD Pipelines*, looks at using Python for the most common DevOps task of Continuous Integration/Continuous Deployment (CI/CD) and enhancing these CI/CD pipelines. [*Chapter 10*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_10.xhtml#_idTextAnchor189), *Common DevOps Use Cases in Some of the Biggest Companies in the World*, looks at Python in DevOps use cases in the context of some of the biggest companies and workloads provided by the major cloud platforms. [*Chapter 11*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_11.xhtml#_idTextAnchor204), *MLOps and DataOps*, provides a look at the machine learning and big data niches of DevOps and how Python can help enhance these workloads. [*Chapter 12*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_12.xhtml#_idTextAnchor218), *How Python Integrates with IaC Concepts*, explores how Python libraries and frameworks are used to provision resources using infrastructure as code to build and modify DevOps workloads in a standardized way. [*Chapter 13*](https://learning.oreilly.com/library/view/hands-on-python-for/9781835081167/B21320_13.xhtml#_idTextAnchor226), *The Tools to Take Your DevOps to the Next Level*, looks at advanced DevOps concepts and tools and how they can be integrated into your workload ## Chapter -1 Let's explore this topics in this chapter, * Exploring automation * Understanding logging and monitoring * Incident and event response * Understanding high availability * Delving into infrastructure as a code ``` DevOps is a series of practices and principles that aims to set a culture that supports the automation of repetitive work and continuos delivery of a product while integrating the software development and IT operation aspects of product delivery ``` * Automation is for lazy but many do not realize that how hard you must work and how much you must study to truly be lazy. * To achieve automation it requires a mindset, attitude, a frustation with present circumstances ``` Automation is the primary basis that frees up our time to do other things we want. ``` ``` Anything that can go wrong will go wrong at the worst possible time. ``` what's difference between logging and monitoring ?🤔 1. **Monitoring** looks at specific metric (Usually generated by logs) and whether or not that metric has passed a certain threshold. However, **logging** is simply collecting the data without generating any insight or information from it 2. Monitoring is active and focuses on the current state of an instance or object that is being monitored, whereas logging is passive and focusses more on the collection of largely historical data ``` History and lessons that we learnt from it are important. They give us perspective and appropriate lessons that we need to respond to future events. ``` A **logged metric** is monitored by a monitoring service. This service looks at the data produced from the logs and measures it against a threshold that is set for that metric. If the threshold is crossed for a sustained, defined period of time, an **alert** or alarm is raised. Most of the time, these alerts or alarms are either connected to a notification system that can inform the necessary personnel regarding the heightened alarm state, or a response system that can automatically trigger a response to the event. Dealing with **incident and event response** involves either a lot of work or zero work. It depends on how prepared you are and how unique the incident or event is. Incident and event response covers a lot of ground from automation and cost control, to cybersecurity. **Service Level Objective** (**SLO**) is used when a response is necessary. However, this is largely on production environments and requires the definition of a **Service Level Indicator** (**SLI**). It also involves the creation of an error budget to determine the right time to add new features and what the right time is to work on the maintenance of a system. * The incident response team is responsible for creation of post-mortems documents \[Another part of incident response is the understanding of what caused the incident, how long it took to recover, and what could have been done better in the future ] **SMART** principles * Specific: Know exactly what is happening * Measurable: Measure its impact * Achievable: Think of what your goal is for mitigation * Realistic: Be realistic with your expectations and what you can do * Time-bound: Time is of the essence, so don’t waste it Some of the common incidents devops engineers may have to deal with : ``` - The production website or application goes down - There is a mass spike in traffic suggesting a distributed denial of service attack - There is a mass spike in traffic suggesting an influx of new users that will require an upscale in resources - There is an error in building the latest code in the code pipeline - Someone deleted the production database ``` Incident response team has the following members : ``` Incident commander (IC): An incident commander leads the response to the incident and is responsible for post-incident response plan Communications Leader (CL) : A communications leader is the public facing member of the team who is responsible for communicating the incident and progress made to mitigate the incident to the stakeholders Operations Leader (OL) : Sometimes synonymous with the incident commander, the OL leads the technical resolution of the incident by looking at logs, errors and metrics and figures out a way to bring their site or application back online Team members (TM): Team members under the CL and OL who are coordinated by their respective leaders for whatever the purpose they may require ``` * Any good DevOps team will perform a post-mortem after an incident has occured. This **post-mortem** will be led by the incident response team that handled the situation * **Post-mortem** let the DevOps team understand the incident that occurred and how it happened, and they dissect the response made by the response team. * If an incident has occurred, it is the process that must be modified, not the person. This approach creates an environment of openness and makes sure that the results of the post-mortem are factual, objective, and unbiased. > Everything fails, all the time High availability metrics > SLIs (are measured for) --> SLOs (are defined in) --> SLAs **Service Level Indicator (SLI) :** These are the metrics that can be used to numerically define the level of service that being provided by the product. For instance, if you were to run a website, you could use the uptime (The amount of time the website is available for service ) as an SLI **Service Level Object (SLO) :** These provide a specific number to the aforementioned SLIs. That number is an objective that the DevOps team must meet for their client. Going back to the previous example in the SLI definition: if uptime is the SLI, then having an uptime of 99% a month is the SLO. Typically, a month has 30 days, which is 720 hours, so the website should have a minimum uptime of 712.8 hours in that month with a tolerable downtime of 7.2 hours **Service Level Agreements (SLA) :** These are contracts that enforce an SLO. In an SLA, there is a defined SLO (hope you’re keeping up now) for an SLI which must be achieved by the DevOps team. If this SLA is not fulfilled, the party that contracted the DevOps team is entitled to some compensation. Concluding that example, if there is an SLA for that website with an SLO of 99% uptime, then that is defined in the agreement and that is the metric that needs to be fulfilled by the DevOps team. However, most SLAs have more than one SLO. **Recovery Time Objective \[RTOs] and Recovery Point Objective \[RPOs]** are largely concerned with recovering operations after a disaster ``` An RTO is placed on a service when there is a need for a service to constantly be up and the time used in RTO is the amount of time that a service can afford to be offline before it recovers and comes online again. The fulfillment of an RTO is defined in the SLA as the maximum time that a system will be down before it is available again. To be compliant with the SLA that the DevOps has, they must recover the system within that time frame. ``` ``` The RPO is the maximum tolerable difference of time between the present and the date of the backup or recovery point. For example, a database of users on a smaller internal application can have an RPO of one day. But a business-critical application may have an RPO of only a few minutes (if that). ``` Based on the objectives and agreements, we can come up with metrics that can affect team behaviour, **Error budget :** In a team following DevOps principles, error budgets become a very important part of the direction that the team takes in the future. An error budget is calculated with this formula: *Error budget = 1-SLA (in decimal)* What this basically means is that *an error budget is the percentage left over from the SLA*. So, if there is an SLA of 99%, then the error budget would be 1%. It is the downtime to our uptime. In this case, the error budget per month would be around 7.2 hours. To recover success from failure is one of the most important skills to learn in life, not just in DevOps. ## chapter 2- Talking about Python * Python programming was built on a set of principles that was meant to simplify coding in it. The simplification came at a cost of a lot of speed and performance when compared to other programming languages * A popular programming language that was accessible and easy to build in, with a massive library of built-in functions. All of the above reasons made python versatile and capable of being used in a myraid of situations. `Now is better than never` {% code title="Declare\&ManipulatingVariables.py" overflow="wrap" fullWidth="true" %} ```python a=13 print(a, type(a)) b="if you can't figure this out, what are you doing here " print(b, b.upper()) ``` {% endcode %} ``` Sometimes, the best way to do something is to do it and then explain it later. ``` Some of the philosophies to define principles of python ``` Beautiful is better than ugly Explicit is better than implicit Simple is better than complex Complex is better than complicated Flat is better than nested Sparse is better than dense Readability counts Special cases aren't special enough to break the rules Although practicality beats purity Errors should never pass silently Unless explicitly silenced In the face of ambiguity, refuse the temptation to guess There should be one --and preferably only one --obvious way to do it Although that way may not be obvious at first unless you're dutch Now is better than never Although never is better than right now If the implementation is hard to explain, it's a bad idea If the implementation is easy to explain, it may be a good idea Namespaces are one honking great idea -- let's do more of those ``` > **`Iterative methodology helps turn even the vaguest goal into a bold mission statement and can provide unified direction`** **what python offers devops 🤔 ?** * They both share emphasis on automation, flexibility and conciseness * The reason of python only for devops is that it handles data that resides between curly brackets ({}) better than almost any other language 1. Operating systems 2. Containerisation 3. Microservices 4. Automation 5. Operating system : Some of the tasks can be done with python in os : * Set environment variables in os * Get information about files and folders * Manipulate, create or delete files and directories * Kill or spawn processes and threads * Create temporary files and file locations * Run Bash scripts 2. Containerisation : Some of the tasks can be done with python in containers : * Interaction with Docker API for commands, such as getting a list of Docker containers or images present in the OS * Automatically generating **Docker Compose files** from a list of Docker images * Building Docker images * Orchestrating containers using the **kubernetes library** * Testing and verifying Docker images 3. Microservices : Some of the tasks can be done with python in microservices : * Strong native library support inside of a python container - libraries such as json, asyncio and subprocess * Excellent native code modules that simplify certain iterative and manipulative operations on data such as collection module * Ability to properly natively handle semi-structured and varied Json data that is usually used in microservices 4. Automation : Some of the tasks can be done with python in automation : * Various software development kits (SDKs) for cloud based deployments in AWS, Azure, Google Cloud and other providers * Support for automated building and testing of applications * Support for monitoring applications and sending notifications * Support for parsing and scraping necessary data from web pages, databases and various other sources of data Now, it's time to see a couple of devops tasks in python 1. Automated shutdown of Ec2 instaces in aws 2. Pulling Docker images 3. Automated Shutdown of EC2 Instances in aws we can achieve our goal of shutting down of our ec2 instances in aws by following below steps * creating one lambda function. Name it as per your choice (ex: stopper) and select latest python version as runtime and architecture and create function * Now let's list down the instances before stopping them with below code {% code title="list-instances.py" %} ```python import boto3 import json def lambda_handler(event, context): # initialize ec2 client ec2_client = boto3.client('ec2') # Return Json return (json.loads(json.dumps(ec2_client.describe_instances(), default=str))) ``` {% endcode %} * we will obviously get authorisation error and to fix it, Lambda will have IAM role. we need to configure for that click on \`configuration\` | permissions there u will find role for permissions * on the page for the role, go to Add permissions and then attach policies. * Let’s give the Lambda function full access to the EC2 services since we will need it to stop the instance as well. If you prefer or if you feel that’s too much access, you can make a custom role: Now, let's re-run our code . u should be able to see list of servers and their current status. * Finally, to shut down the running instances. Add code to filter among the instances for ones that are running and get a list of their IDs, which we will use to reference the instances we want to stop: {% code title="stop\_instance.py" overflow="wrap" %} ```python import boto3 import json def lambda_handler(event, context): # Initialize Ec2 client ec2_client = boto3.client('ec2') # Get running instances instances_json = json.loads(json.dumps(ec2_client.describe_instances(Filters = [ { 'Name' : 'instance-state-name', 'Values' : [ 'running' ] } ]), default = str)) # Filter to only instance_ids instance_ids = [instance["Instances"][0]["InstanceId"] for instance in instance_json["Reservations"]] # shutdown selected instance ids response = ec2_client.stop_instancs( InstanceIds = instance_ids ) return response ``` {% endcode %} * Now that we have done it once, we can automate it further by using a service called `Amazon Event Bridge` and make an event bridge schedule and select our lambda function as a event target. 2. Auto Pull Docker images * we will see how we can use python to pull docker images automatically. ``` 1. First we need to install, docker python library in a virtual environment using `pip install docker` command ``` 2. write a script file named `docker_pull.py` to loop through a list of image names and pull them by leveraging the docker library : {% code title="docker\_pull.py" %} ```python import docker def pull_docker_images(): client = docker.from_env() image_list = ["redis:latest", "nginx:latest"] for image in image_list: print("pulling image {}".format(image)) client.images.pull(image) if __name__ = "__main__": pull_docker_images() ``` {% endcode %} 3. After this run the file using ``` python docker_pull.py ``` You can then run the **docker images** command to check out the Docker containers that you may have locally: ## 3. The simplest ways to start using DevOps in Python Immediately > Things don't just happen. They are made to happen #### Introducing API Calls * Before learning about API calls, let's what exactly API is API stands for application programming interface and it is a software interface that offers your application access to functions and processes from other applications. Think of it like UI of your software. if a user want some information from your application, he will get by UI. API is simmilar function for software, so you can call API the UI of software. Now, API calls can be made for a number of reasons : * you don't want to write the underlying logic for a big feature yourself (trust me a lot of times, you don't) * The API gives access to resources that you ordinarily would not have (i.e; creating a virtual machine using API of a cloud provider) * You just want to give some information into your application (Public APIs are very good for this) 1. Calling a Hugging Face Transformer API 2. Creating and Releasing an API for Consumption 3. Networking - Using Scapy to sniff packets and visualize packet size over time 4. Networking - Generating a routing table for your device **Calling a Hugging Face Transformer API** * We can either use our local dev environment i.e; vscode or google colab notebook () * Install required libraries 
pip install huggingface_hub transformers[agents]
* Get API key . Visit huggingface.co signup page and click on profile settings access tokens . * Enter in your terminal like below ```python from huggingface_hub import login login("") ``` We are going to use the Hugging Face Transformer API to take a line of text and turn it into an image. But first, we must import a Hugging Face agent using the **HfAgent** API (see the pattern?): ```python from transformers import HfAgentagent = HfAgent("https://api-inference.huggingface.co/models/ bigcode/starcoderbase") ``` * we are using **starcoderbase** model for this. Once you run this and get the agent, you can simply type in prompt to generate an image ``` agent.run("Draw me a picture of `prompt`", prompt="rainbow butterflies") ``` ``` API key is like a login but for your software. Most companies will allow full access to their APIs through the purchase of an API Key ``` API KEY is like a login but for your software. Most companies only allow full access to their APIs through the purchase of an API KEY. A lot of opensource projects such as HUGGING FACE have API Keys to promote and track user interaction and sometimes upgrade their users to a premium version if they want. **Creating and Releasing an API for Consumption** ```python from flask import Flask app = Flask(__name__) @app.route('/') def index(): return 'hello from Flask.' app.run(host='0.0.0.0', port=81) ``` ```python from flask import Flask app = Flask(__name__) @app.route('/') def index(): returning_json_value = {"charan":"checking"} return returning_json_value app.run(host='0.0.0.0', port=81) ``` The above is having only static values. To take user input, add request parameter in the URL Itself. {% code title="Flask api code to add two numbers" %} ```python from flask import Flask, request app = Flask(__name__) @app_route('/') def index(): num1 = request.args.get("num1") num2 = request.args.get("num2") returning_json_value = {"Sum of parameters":int(num1)+int(num2)} return returning_json_value ``` {% endcode %} ### Exercise 1 – using Scapy to sniff packets and visualize packet size over time scapy is a python library that can be used to replicate, simulate and manipulate data packets that are sent over a computer network In this exercise, we are going to use scapy to collect a list of packets and get their timestamps and packet sizes. we are then going to map these onto a chart that we make using the matplotlib library. ``` pip install matplotlib scapy ``` Now, let's write a code to use scapy's sniff module to get a list of packet sizes over timestamps : ```python from scapy.all import sniff # Lists to store packet sizes and timestamps packet_sizes = [] timestamps = [] #Handle packets and get the necessary data def packet_handler(packet): print(packet) packet_sizes.append(len(packet)) timestamps.append(packet.time) # start packet sniffing on the default network interface sniff(prn=packet_handler, count=100) ``` You will get a list of the length of the last **100** packets that went through your network along with the timestamp and the type of traffic. If you refer to the following diagram, the packet sizes are stored in the **packet\_sizes** array and the timestamps of the packet are stored in the **timestamps** variable: Now let's write the code to plot the packet size over time using **matplotlib:** ```python # create a plot plt.figure(figsize=(16,8)) plt.plot(timestamps, packet_sizes, marker='0') plt.xlabel("Time") plt.ylabel("Packet Size") plt.title("Packet Size Over Time") plt.grid(True) plt.show() ``` So, we have now tracked our network activity and generated data insights from it using Python. Let’s look at one more network implementation, this time focusing on the routing rules that your device (or the device you are running your workload on) has ### Exercise 2 – generating a routing table for your device Routing tables define the routes that certain web traffic takes within your devices. These tables exist in practically every device, and they define the routes by which those devices access computer networks. You can use the **netifaces Python library** to generate a routing table showing all the available routes and destinations that your device contains. The netifaces library in this case is used to collect the network interfaces (hence the name *netifaces*) of your operating system. You will then parse this information and display it in a tabular form. ``` pip install netifaces ``` ```python import library import netifaces # begin function def generate_routing_table(): routing_table = [] # loop through network interface for interface in netifaces.interfaces(): # initialize current address of interface Interface_addresses = netifaces.ifaddresses(interface) # check for, then loop through the addresses if netifaces.AF_INET in addresses: for entry in interface_addresses[netifaces.AF_INET]: # creating routing entry wherefound if 'netmask' in entry and 'addr' in entry: routing_entry = { 'interface': interface, 'destination':entry['addr'], 'netmask':entry['netmask'] } # Append route to routing table routing_table.append(routing_entry) return routing_table # call function routing_table = generate_routing_table() # Display routing table for entry in routing_table: print(f"Interface: {entry['interface']}") print(f"Destination: {entry['destination']}") print(f"Netmask : {entry['netmask']}") print("-" * 30) ``` ### **Chapter 4: Provisioning Resources** > life is chaotic, dangerous and surprising. Buildings should reflect that * Rightsizing is the art of finding the optimal resource sizes for your application or workload. A lot of this is just a trial and error (often yours, but someone else’s if you can get it) * **Scaling** your resources up and down is one of the important aspects of DevOps and removing resources is as key an aspect of provisioning them as adding resources is. **Python SDK's ( and why everyone uses them)** * SDK's or Software development kits, are official programming libraries and CLIs released by a platform that allows developers to develop tools and applications that leveraget that platform. These SDKs are written in most popular programming languages to cover the large number of developers possible python is a exact balance between loose and structured that is necessary to pull off a lot of DevOps principles ### Creating an AWS EC2 instance with Python’s boto3 library * BOTO3 - It's the name you've probably heard very often if you've worked with AWS and Python. It is the SDK that contains nearly every major AWS service that is currently available with python * sagemaker notebook is a jupyter notebook service run on AWS servers : {% code title="code to invoke boto3 API" %} ```python import boto3 ec2_client = boto3.resource("ec2", region_name="us-west-2") ec2_client.create_instances(ImageId="ami-002829755fa238bfa", InstanceType="t2.micro", MaxCount=1, MinCount=1) ``` {% endcode %} Note: Had to give the role some permissions or use the AWS CLI to attach a role profile to the instance **Scaling and Autoscaling** **Scaling** is the act of increasing or decreasing the size of workload or resource depending on the demand for it. **Autoscaling** is doing this automatically based on some sort of trigger. * **Scaling** can be done vertically(Adding the computing power to a device) or horizontally (adding more computers). When performing one powerful act, vertical scaling is ideal and when processing a greater number of requests , you'll need horizontal scaling. Most DevOps workloads require the latter over former. **Manual Scaling with Python** {% code title="code to create an EC2 instance" %} ```python import boto3 def create_ec2(): ec2_client = boto3.resource("ec2") print(ec2_client.create_instances (ImageId="ami-051f7e7fb6c2f40dc1", InstanceType="t2.nano", MaxCount=1 MinCount=1)) if __name__ == "__main__": create_ec2() ``` {% endcode %} {% code title="Function to stop ec2 instance" %} ```python import boto3 def stop_ec2(): ec2_client = boto3.client("ec2") print(ec2_client.stop_instances(InstanceIds=["i-02d42ec52027baa08"])) if __name__ = "__main__": stop_ec2() ``` {% endcode %} {% code title="code to modify instances" %} ```python import boto3 def update_ec2(): ec2_client = boto3.client("ec2") print(ec2_client.modify_instance_attribute{ InstanceId = "i-02d42ec52027baa05", InstanceType = { 'value' : 't2.micro', } )) if __name__ = "__main__": update_ec2() ``` {% endcode %} ### Autoscaling with Python based on a trigger * Autoscaling requires automating the process of increasing the available compute resource according to some sort of metric or statistic. In order to autoscale, we need to design a mechanism that will trigger our SDK call once a certain metric or threshold has been reached. Let’s write a basic script to make an autoscaling group and put a threshold on it using a policy for CPU utilization. We’ll go step-by-step from the launch configuration to the autoscaling group to the rule by which the instances will autoscale: {% code title="creating launch configuration via python" %} ```python autoscaling.create_launch_configuration( LaunchConfigurationName = "book_configuration", ImageId = "ami-051fe7e7f6c2f40dc1" InstanceType = "t2.micro" ) ``` {% endcode %} {% code title="creating autoscaling group to use launch configuration" %} ```python autoscaling.create_auto_scaling_group( AutoScalingGroupName = "book_autoscaler", LaunchConfigurationName ="book_configuration", MinSize = 2, MaxSize = 5, DesiredCapacity=2, AvailabilityZones=["us-east-1a", "us-east-1b"], ) ``` {% endcode %} {% code title="policy to scale upward if cpu utilization is greater than 70%" %} ```python autoscaling.put_scaling_policy( AutoScalingGroupName="book_autoscaler", PolicyName="book_scale", PolicyType='TargetTrackingScaling', TargetTrackingConfiguration={ 'PredefinedMetricSpecification': { 'PredefinedMetricType': 'ASGAverageCPUUtilization' }, 'TargetValue': 70 } ``` {% endcode %} ## Containers and where Python fits in with containers **Containers** are small packages of software that serve as a unique runtime containing all of the necessary resources to run a small facet of an application, or sometimes the application itself. We will write a script to get specific docker image and create a container for it. This script can be returned to to the same thing over and over again. You may also use the **restart** command if container malfunctions. Now let's look at the code to pull docker images and start a container {% code title="pull docker images and start a container" %} ```python import docker def docker_sample(): client = docker.from_env() # define the image name and tag image_name = 'python.latest' # pull the image client.images.pull(image_name) # create and run a container based on the pulled image container = client.containers.create(image_name) container.start() print(client.containers.list()) if __name__ = "__main__": docker_sample() ``` {% endcode %} ### Managing Kubernetes with Python **Namespaces** are abstractions within a Kubernetes cluster that are used to divide computer resources based on certain criteria. A criterion can be environment (dev, production, etc.), network restrictions, or based on resource quotas available to a namespace Let’s look at the steps to initialize a Kubernetes cluster and manage it using Python: ``` pip install kubernetes ``` let's write a script to create a few namespaces in our cluster. We can add resources to these namespaces later {% code title="code to create kubernetes namespaces from the list" %} ```python from kubernetes import client, config # load kubernetes configuration config.load_kube_config() # create a kubernetes client api = cliet.CoreV1Api() # define namespace namespaces=['ns1', 'ns2'] # Create namespace for namespace in namespaces: namespace_client = client.V1Namespace(metadata=client.V1ObjectMeta(name=namespace)) api.create_namespace(namespace_client) ``` {% endcode %} Now let's write a policy for these namespaces and implement them into kubernetes : {% code title="" overflow="wrap" %} ```python for namespace in namespaces: # define the network policy manifest network_policy_manifest = { "apiversion": "networking.k8s.io/v1", "kind": "NetworkPolicy", "metadata":{ "name":"block-external-traffic", "namespace": namespace }, "spec":{ "podSelector": {}, "policyTypes": ["Ingress"] } } # create network policy networking_api.create_namespaced_network_policy(namespace=namespace, body=network_policy_manifest) ``` {% endcode %} This will create policies for both namespaces, which will block external traffic from outside the namespace ### **Chapter 5 - Manipulating resources** ## Event-based resource adjustment ### Edge location-based resource sharing * One of the most difficult challenges that we have with a global application is load time for resources. If a user is too far from the closest data center, it can result in a significant amount of lag and latency when they use their application. For this reason, most companies with applications on a major scale have data centers and edge caches in high-population areas/areas with high web traffic * An **Edge or Edge Location** is just a datacenter that is closer to the target user, making their load time faster. * **Edge Caches** are user and website data (such as cookies) stored in edge locations for faster access for that user. This helps users get to datacenters that can serve them the fastest. However, the question now becomes, how do we direct these users to the appropriate data centers or caches that will give them the lowest latency? To do that, we must find a way to acquire the location data for the user’s device and reference that data to redirect that user to the closest data center. To do this, we can write a function that takes a web traffic request and extracts certain headers from it. Now that we have learnt how to redirect users to custom locations, let's look at how to redirect features toward a custom set of users. ### Testing features on a subset of users * When an application needs to implement a new feature and the application team wants that feature to be tested in a live environment, the need arises for some sort of mechanism that sorts a small subset of users into a group that will receive that new feature. The analytical data acquired from this group will then be used to judge the feature and its effectiveness based on certain criteria * This kind of testing is called as **A/B Testing** ## Analyzing data you must live for the person you are right now, and that person is defined by your past and yet is not the same person as in the past ### Analysis of live data * Live or streaming data is data that is constantly being processed by a system at present. It is data that’s being received or returned by a system. A system lives off of the data that it absorbs and generates (*input* -> *process* -> *output*). The system is shaped by this data and sometimes, in the case of critical systems, it needs to be molded by this data. To make tactical decisions based on recent data, collecting live data and immediate insights on that data is necessary * Even though most clouds and monitoring systems provide default ways to store and analyze live data. Some times, it needs to be customized and can be done with python not because of speed but because of convenience and pre-built library for analysis and conversion on practically any kind of data that a system gives out. {% code title="Example where we use python's built in marshal library to decode a byte string" %} ```python import marshal # function to decode bytes def decode_bytes(data): # load binary data into readable data with marshal result = marshal.loads(data) return data ``` {% endcode %} Byte strings are often used in **network communication** and **cryptography** (both of which usually involve live data), and converting them into other languages may require adding libraries and possibly creating custom data types. There’s no such need with Python ### Analysis of historical data * Live data can help to make adjustments; they are tactical. But to be truly strategic - to think about the big picture and to think longterm . You need historical data and a way to analyze it. The past contains a lot of data; this data contains patterns and it is in these patterns that the key is to optimizing your workload lies. * Python’s **multiprocessing library** can be used to batch-process a large amount of data in parallel by leveraging multiple CPU cores for this purpose ```python import multiprocessing ''' Get list and return sum ''' def get_sum(numerical_array): ''' return sum of array ''' return sum(numerical_array) '''method that calls sum method to get total sum for 1 million records ''' def get_total_sum(full_array): ''' Initialize 4 processors ''' pool = multiprocessing.Pool(processes=4) ''' List of sums in 10,000 number blocks ''' sum_list = [] ''' Get 10,000 values at a time for sum out of 1,000,000'''' for i in range(0, count(full_array)/10000): sum_array = full_array[i*10000:(i+1)*10000] '''Make an array of 10,000 length arrays ''' sum_list.append(sum_array) ''' Get final array of sums of each 10,000 value array ''' final_array_list = pool.map(get_sum, sum_list) ''' Get one final sum of all these elements ''' return sum(final_array_list) ``` in this code, an array of 1 million values is divided into 100 arrays of 10,000 values, each of which is summed up to calculate the total for their array. This is done on one of four different processors at the same time using the multiprocessing library. Breaking down this large task into smaller tasks can optimally use resources for data processing. All of this is nice, Where does this data comes from ? your applications. Well, for the data to be good, the application has to be good right ? * To get out of this technical debt, you have three options: **optimise, refactor or restart.** what does this means is you have three choices (not just in devops, but in life too): ``` live with your mistakes, improve upon them, or start fresh ``` what you want to do (and what you can do) depends on both you and your circumstances. **optimise** * In the case of optimizing legacy applications, we are very limited in what we can do. But that doesn’t mean we can’t do anything. One of the most important DevOps concepts is **Desired State Configuration** (**DSC**) and the reason for that is maintaining systems such as these. In a DSC, the virtual machine is given certain configurations that it must maintain. The state of these configurations is checked from time to time. If the state has changed significantly, then it is reset and brought back to the original desired state. * Probably, this is one of the best way to handle an application that cannot be refactored or containerized into docker containers **Refactor** * Refactoring is a strategic approach to improving code and project infrastructure without complete replacement. It involves making incremental improvements to an existing codebase, which can range from simple tasks like: * Upgrading dependencies * Writing new components * Removing unnecessary components More complex refactoring might include significant architectural changes, such as separating a website's frontend and backend. The key goal is to enhance the code's efficiency, maintainability, and functionality while preserving its core structure and avoiding a total rewrite. * One of the prominent method of refactoring a code is **strangler fig \[**It is like root coming from tree and eventually killing tree and root becoming tree] Here are the steps to perform strangler fig refactoring : ``` 1. Separate the database 2. Turn third-party API calls into functions/microservices 3. Separate the backend and frontend 4. Separate non-critical backend functions into microservices 5. Create a database connection mechanism and put the rest of the backend into microservices ``` **Restart** * If you are making a new application, here's some advice : ``` 1. Make decesions that are longterm and not just based on what you're thinking about now. 2. Make decesions that put quality first, but also understand that mistakes will always be there 3. Don't react to everything, being reactive will 100% result in a worse application. If you can help it, don't react to anything. Be proactive 4. Put quality, readability and maintainability first. Don't think that you'll comeback to something later. That kind of thinking adds many hours of work on your shoulders (even worse somebody else's later) ``` Don’t restart too much, though, because you’ll never get anything done. There are only so many times you can restart a project before you realize that maybe your approach is the problem. ### chapter 6 - Security and DevSecOps with Python > It’s a jungle out there. Disorder and confusion everywhere In DevSecOps (a branch of DevOps dedicated to security), the goal is to have security measures present and used before, during, and after any breach in security. ## Securing API keys and passwords * API keys and passwords are valuable for the reason most things are valuable: they cost money and the data protected by them are valuable * We can create environment variables and secrets that store variables that are sensitive in a separate folder or within the **operating system** (**OS**) configuration itself. We can also use Python (by itself or with some other tool or API) to extract and obfuscate **personally identifiable information** (**PII**) ### Store environment variables * Environmental variables are stored as both a means of separating credentials from application code to prevent hardcoding and to make sure that an application can run with different credentials on different systems just by configuring those credentials into the OS or a file. * That brings us to the two ways in which we can store and retrieve environment variables: we can store them as files or as variables defined in the OS. of course, also do this in a cloud secret manager or using hardware device, but those are simmilar concepts to the ones we are discussing. 1 st way : From files we can create and read from .env file in the following way : * These files are the standard for storing these environment variables and are usually ignored in practically every .gitignore file. To read these files, we must first install python-dotenv library : ``` pip install python-dotenv ``` * After this, we can create **.env** file and store variables and secrets within it. We separate these secrets based on the line that they are on. ``` API_KEY = API_SECRET_KEY = ``` all caps for constants. This approach helps consolidate these constants in one place so that you don't have to worry about changing them in all the locations you need them in your code, and you don't have to worry about missing one change somewhere. Now, let's write the code by which our program can access the environment variables : ```python from dotenv import load_dotenv import os # load .env file load_dotenv() # load API keys into variable api_key = os.getenv("API_KEY") api_secret_key = os.getenv("API_SECRET_KEY") ``` Running this code will give you the API Key and the SECRET KEY in their respective variables. * This method is a little bit simpler and perhaps a little more secure than the **.env** file for a fewer number of environment variables. But when using a large number of environment variables, a **.env** file is better at aggregation and easier to use and handle. * A person’s sensitive personal information is one of the most valuable things they have: financially, socially, and intimately. Compromising the security of this information can result in great harm to the person in all three of these areas. The privacy of people is of utmost importance, and we must take all possible measures in order to preserve it, especially when they entrust some bit of that information to the services that we provide for them * Well, there are ton of pre-built services such as Amazon Macie or Google Cloud's DLP (Data loss prevention) and these services are handy, but you may not understand the inner workings of them . ``` We are going to use a very simple regex or regular expression pattern for phone numbers to find them within text and replace them with some form of redaction. ``` Now, let's try the samething but with environmental variables exported and used directly from our own OS : 1. in linux ``` export API_KEY = export API_SECRET_KEY = ``` * These will set environmental variables having the names API\_KEY and API\_SECRET\_KEY in your OS Now let's see how to access these values : ``` import os # Get the API Key and Secret access Key from the environment api_key = os.environ.get("API_KEY") api_secret_key = os.environ.get("API_SECRET_KEY") ``` This method is little bit simpler and perhaps a little more secure than the .env file for a fewer number of environment variables. But when using a large number of environment variables, a **.env** file is better at aggregation and easier to use and handle. * This methods have thought you how to deal with sensitive information that you are responsible for and yourself add to the code base. ### Extract and obfuscate PII * A person's sensitive personal information is one of the most valuable things they have: financially, socially, and intimately. Compromising the security of this information can result in great harm to the person in all three of these areas. The privacy of people is of utmost importance, and we must take all possible measures in order to preserve it, especially when they entrust some bit of that information to the services that we provide for them. well, there are some pre-built services such as Amazon Now, let's implement the regex on a small passage that contains phonenumbers : ```python # initial text import retext = " The first number is 901-895-7906. The second number is: 081-548-3262" # pattern for search search_pattern = r'\d{3}-\d{4}-\d{4}' #replacement for pattern replacement_text = "" # text replacement new_text = re.sub(search_pattern, replacement_text, text) # output given: "The first number is .The second number is: " print(new_text) ``` This code finds phone numbers using a regex pattern and subsequently obfuscates it by replacing the phone numbers. ## Validating and verifying container images with Binary Authorization * Some sort of assurance or regulation that defines that the correct image has been used. This mechanic is known as **Binary Authorization** * **Binary Authorization** allows the docker containers that are deployed in a kubernetes cluster to be checked according to certain criteria. Authorization can be done either by using a compilance policy or an attestor. * A compilance policy creates rules that allow only images that meet a certain criterion. Adding an attestor means adding an individual user in your project who can testify that the image that is going to be used is correct, * In Binary Authorization, you can use one or both of these in order to authorize your images for your kubernetes cluster. let's see how we can do this. 1. we are now going to write a script to create an attestor and assign that attestor to the kubernetes engine. First, we must install the client libraries for containers and Binary Authorization ``` pip install google-cloud-binary-authorization google-cloud-container ``` 2. Now, let's write the script for Binary Authorization of the cluster {% code title="This will create an attestor that can attest your cluster requests" %} ```python from google.cloud import binaryauthorization_v1 def sample_create_attestor(): client = binaryauthorization_v1.BinauthzManagementServiceV1Client() attestor = binaryauthorization_v1.Attestor() attestor.name = request = binaryauthorization_v1.CreateAtttestorRequest( parent = , attestor_id = , attestor=attestor, ) client.create_attestor(request=request) ``` {% endcode %} 3. Now we need to add the Binary Authorization to a cluster that has already been created or is about to created : ``` from google.cloud import container_v1 def sample_update_cluster(): client=container_V1.ClusterManagerClient() request=container_V1.UpdateClusterRequest( "desired_node_pool_id": , "update" : { "desirec_binary_authorization": { "enabled": True, "evaluation_mode": 2 } } ) clent.update_cluster(request=request) ``` This script will update the kubernetes cluster to start using our Binary Authorization method. ## Incident monitoring and response * In order to respond to a security threat that potentially targets a fleet of virtual machine, python can help run what is called **runbook.** which is a series of commands that can be deployed to reset a system or to have it respond to some sort of threat. ### Running runbooks * For this exercise, we are using the **AWS EventBridge** to trigger a step function that will subsequently run a command on the desired EC2 instance that has triggered the event, restarting the instance from within. Again, we're keeping the commands simple here, but if you want you can get more advanced with it. let's get started 1. Let's have one running EC2 instance in AWS 2. let's make the structure for the command now in **AWS LAMBDA**. But, first we need the command that is to be sent, which is found in **systems manager**. Open systems Manager and go to the **Documents** tab at the bottom of the sidebar : 3. Within the documents, search for the **AWS-RestartEC2Instance** document. This is the default document, and you can base a lot of other VM manipulation documents on it. 4. Now. let's write an Eventbridge event that will trigger a lambda function in the event of a decrease in network traffic over the course of five minutes. For that, let's first go to Amazon Cloudwatch and create an alarm that triggers under our condition. Go to cloudwatch | Alarms | create alarm 5. Next, let's set the condition for the trigger for the alarm to be under 20,000 bytes of network input over a period of five minutes 6. Go to Amazon EventBridge in the AWS section, and then to the Rules tab, where you will create a new Event, on the first page, select Rule with an Event Pattern and then move on to the next page. Here, you will paste the EventBridge rule that you acquired from the alarm 7. Now that you have the pattern, press Next, then in another tab, open AWS lambda. Here we will write our python code to execute the restart of the EC2 instance. In AWS Lambda, choose the python execution time and maintain all of the default settings otherwise, then, add the following code : ```python import json import boto3 client = boto3.client('ssm') def lambda_handler(event, context): instance_id = event["instanceids"] client.send_command(InstanceIds = [instance_id], DocumentName = "AWS-RestartEC2Instance") return "command to restart has been sent" ``` This code will send the command of the playbook document to restart the EC2 instance. Now, let's select Lambda function as the trigger for EventBridge : * Keep hitting Next until you create the EventBridge rule. When the alarm is triggered, the rule will trigger and run the lambda function. This will restart the instance and keep doing so until the network input has been restored to acceptable levels. ### Pattern analysis of monitored logs * Distributed deniel of attacks (DDOS) attacks usually create a pattern of iniexplicable high CPU usage in an application. It essentially causes the application to suffer from fake loads, which affect the actual loads in the application. * public datasets of server logs are difficult to find, but they can be recreated fairly easily. For this example, I will use the public dataset mocking service **Mockaroo** . It let's you create a dataset of 1000 rows for free. I will only create a dataset with CPU utilization and the timestamp. 1. First, Install the pandas and matplotlib libraries . ``` pip install pandas matplotlib ``` 2. Next, upload the CSV that Mockaroo created to colab. 3. Now, next step is to read the data using pandas to create a dataframe and order it by time, because the mock data in the CSV was not pre-arranged in a sequence. ```python import pandas as pd df = pd.read_csv("MOCK_DATA.csv") df.sort_values(by="timestamp", inplace=True) df ``` * Now to find the pattern in the data, we can begin by visualizing the data. let's vistualize this data as a linear plot with the timestamp as the x-axis and CPU utilization as the y-axis: ```python import matplotlib.pyplot as plot plt.figure(figsize=(20, 20)) plt.plot(df['timestamp'], df['cpu_utilization'], label='Data') plt.title('Plot for CPU utilization') plt.xlabel('Time') plt.ylabel('CPU utilization') plt.grid(True) plt.legend() plt.show() ``` ### Chapter 7 Automating Tasks > The only way you control your destiny, the way you live, and what you do is by controlling your own time, by choosing what you do. And to do that, you need to choose what it is that you do with your time. * The concept of automation is Investing ways to reduce time investment in the boring stuff so that you can move on and do more exciting or engaging stuff. ## Automating server maintenance and patching ### Sample 1: Running fleet maintenance on multiple instance fleets at once * Let's first write the code for AWS instance to find the instances that are running : ```python import boto3 ec2_client = boto3.client('ec2') response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]) aws_instances = response['Reservations'] ``` for GCP, ```python from google.cloud import compute_v1 instance_client = compute_v1.InstancesClient() request = compute_v1.AggregatedListInstancesRequest() request.project = "ID of GCP project that you are using" gcp_instances = instance_client.aggregated_list(request=request, filter="status:RUNNING") ``` Now, let's find a command to run through these instances. ```python command = "sudo reboot" # for aws ssm.send_command(InstanceIds=aws_instances, DocumentName="", Comment = 'Run a command on an EC2 instance', Parameters={ 'commands': [command] } # for Google Cloud import os import subprcess from google.oauth2 import service_account from googleapiclient import discovery # load the service account credentials service_account_file = '.json' credentials = service_account.Credentials.from_service_account_file( service_account_file, scopes=['https://www.googleapis.com/auth/cloud-platform'] ) # Create a compute engine API Client compute = discovery.build('compute', 'v1', credentials=credentials) # get the public IP address of the VM instance request = compute.instances().get(project="", instance="your_instance_name") response=request.execute() public_ip = response['networkInterfaces'][0]['accessConfigs'][0]['natIP'] # SSH into the VM instance and run the command ssh_command = f'gcloud compute ssh {instance_name} --zone {zone} --command "{command}"' try: subprocess.run(ssh_command, shell=True, check=True) except subprocess.CalledProcessError: print("Error executing SSH Command.") ``` ### Sample 2: Centralizing OS patching for critical updates * process of patching a single OS ```python import subprocess update_command = "sudo apt update && sudo apt upgrade -y" subprocess.run(update_command, shell=True) ``` ```python import subprocess import platform def update_os(): system=platform.system().lower() if system == 'linux' or system == 'linux2': if 'debian' in platform.linux_distribution()[0].lower() or 'ubuntu' in platform.linux_distribution()[0].lower(): update_command = "sudo apt update && sudo apt upgrade -y" else: update_command = "sudo dnf update -y" subprocess.run(update_command, shell=True) elif system == 'windows': update_command = 'powershell -Command "Start-Service -Name wuauserv; Get-WindowsUpdate; Install-WindowsUpdate;"' subprocess.run(update_command, shell=True) if __name__=="__main__": update_os() ``` Automating Container Creation ### Sample 1: Creating containers based on a list of requirements 1. a simple code to start a container based on an image : ```python import docker client = docker.from_env() container = client.containers.run('ubuntu:latest', detach=True, command='/bin/bash') container_id = container.id print("Container ID:" + container_id) ``` Now, we have id of the container and a running container on latest version of ubuntu 2. ```python # you can put in any command you want as long as it works new_command = "ls" new_image = client.containers.get(container_id).commit() new_image_tag = ":latest" new_container = client.container.run(new_image_tag, detach=True, command=new_command) ``` * now we have a new container that has new command added on top of everything else in ubuntu. This container is different from original one but built upon the original 3. next, we need to export this image for later use : ``` image = client.images.get(":latest") image.save("") ``` 
import docker
#Step 1: Intialize and run a container
client = docker.from_env()
container = client.containers.run('ubuntu:latest', detach=True, command='/bin/bash')
container_id = container.id
print("Container ID:" + container_id)
#Step 2: Add a layer
#you can put in any command you want as long as it works
new_command = "ls"
new_image = client.containers.get(container_id).commit()
new_image_tag = "<whatever_you_want>:latest"
new_container = client.containers.run(new_image_tag, detach=True, command=new_command)
#Step 3: Export layered container as an image
image = client.images.get("<whatever_you_want>:latest")
image.save("<insert_file_path_here>")
### Sample 2: Spinning up Kubernetes clusters let's create a kubernetes cluster in azure ``` from azure.mgmt.containerservice.models import ManagedCluster,ManagedClusterAgentPoolProfile resource_group = '' cluster_name = ' * In any application, everything can be divided into events. Events are triggered either by some interaction with an external actor (either another application or user) or by other events. * An application is basically the triggering of multiple sequences of events to perform some sort of function. ### Chapter 8 Understanding Event-Driven Architecture ### Chapter 9 Using Python for CI/CD Pipelines ### Chapter 10- *Common DevOps Use Cases in Some of the Biggest Companies in the World* ### Chapter 11 - MlOps and DataOps ### Chapter 12 - How python integrates with IAC Concepts ### Chapter 13 - The tools to take your DevOps to the next level --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://charan-techjourney.gitbook.io/charan-techjournal/books/hands-on-python-for-devops-ankur-roy.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.