Welcome back! In What is Prompt Engineering? A Complete Introduction, you learned what prompt engineering is and why it matters. Now we're going deeper: understanding the engine under the hood. You don't need to become a machine learning engineer, but understanding how LLMs work will transform how you prompt them.

Why Understanding LLMs Makes You a Better Prompt Engineer

Here's a question: Would you be a better driver if you understood how an engine works? Maybe, maybe not—but you'd definitely be better at diagnosing problems and maximizing performance.

The same applies to prompt engineering.

When you understand:

• Why temperature settings change output personality

• How tokens affect your costs and results

• Why context windows are your biggest constraint

• What makes different models excel at different tasks

...you stop guessing and start engineering.

This post will give you that X-ray vision. By the end, when an AI gives you an unexpected response, you'll know exactly why—and how to fix it.

Part 1: How Large Language Models Actually Work (The Simplified Truth)

The Core Concept: Statistical Prediction at Scale

Let's start with a truth that changes everything:

Large Language Models don't "understand" language. They predict it.

Imagine you're playing a game where I say: "The cat sat on the..."

Your brain instantly suggests: mat, chair, windowsill, table—all reasonable completions. That's essentially what an LLM does, except:

1. It analyzes trillions of word patterns from its training data

2. It calculates the probability of each possible next word

3. It selects based on those probabilities (modified by parameters we'll discuss)

4. It repeats this process word by word until the response is complete

Key insight: Every word you see from an AI is a prediction based on:

• The patterns it learned during training

• Your prompt (the context you provided)

• The parameters you've set (temperature, etc.)

The Transformer Architecture: The Breakthrough That Changed Everything

In 2017, Google researchers published a paper called "Attention is All You Need" that introduced the Transformer architecture. This was the earthquake that created the AI revolution we're experiencing.

What made it revolutionary?

Before Transformers: Sequential Processing

Older models (RNNs, LSTMs) had to process text sequentially—one word at a time, left to right. This meant:

• ❌ Slow processing

• ❌ Limited context understanding

• ❌ Difficulty with long-range dependencies

• ❌ Couldn't parallelize (use multiple processors simultaneously)

Example problem:
In the sentence "The chef who trained in France and Italy and worked at multiple Michelin-starred restaurants opened his new restaurant," the word "his" refers back to "chef"—but there are 15 words in between. Old models struggled with these connections.

The Transformer Revolution: Attention Mechanism

Transformers introduced self-attention, which allows the model to:

• ✅ Look at all words simultaneously

• ✅ Understand relationships between any words in the input

• ✅ Process in parallel (much faster)

• ✅ Handle long-range dependencies effectively

The "Attention" mechanism answers: "When processing this word, which other words in the sequence should I pay attention to?"

Visual analogy:

• Old way: Reading a book by looking at one word at a time through a tiny hole

• Transformer way: Seeing the entire page at once and understanding how every word relates to every other word

The Three Components of a Transformer

1. Input Embeddings

• Your text is converted into numerical vectors (we'll cover this in tokenization)

• These vectors capture semantic meaning: "king" - "man" + "woman" ≈ "queen"

2. Encoder-Decoder Architecture (or Decoder-Only)

• Encoder: Processes the input and creates a rich representation

• Decoder: Generates the output based on that representation

• Modern LLMs (GPT series, LLaMA) use decoder-only architecture for generation tasks

3. Attention Layers (The Magic)

• Multiple layers of attention mechanisms

• Each layer learns different patterns: grammar, facts, reasoning, style, etc.

• GPT-4 has 120+ layers; Claude 3.5 has similar complexity

Training: How Models Learn

Phase 1: Pre-training (The Foundation)

The model reads massive amounts of text from the internet and learns to predict the next word.

Scale we're talking about:

• GPT-3: Trained on ~45TB of text (300 billion tokens)

• GPT-4: Estimated 10-20 trillion tokens

• LLaMA 2: 2 trillion tokens

• Claude 3: Estimated similar or greater scale

What it learns:

• Language patterns and grammar

• Factual knowledge (though imperfectly)

• Reasoning patterns

• Common sense associations

• Cultural and domain knowledge

• Unfortunately, also biases present in training data

Training objective: Given this sequence of words, predict the next one. Repeat billions of times across trillions of words.

Phase 2: Fine-Tuning (Specialization)

After pre-training, models undergo additional training:

a) Supervised Fine-Tuning (SFT)

• Human-written examples of good responses

• Teaches the model to follow instructions

• "When someone asks X, respond like Y"

b) Reinforcement Learning from Human Feedback (RLHF)

• Humans rank multiple model outputs

• Model learns which responses humans prefer

• This is why ChatGPT and Claude are so much better at conversation than raw GPT-3

The result: A model that can follow complex instructions, maintain context, and generate human-like text.

Parameters: The Model's "Memory"

You've probably heard models described by their parameter count: "GPT-4 has 1.7 trillion parameters!"

What are parameters?
Think of them as the model's "knowledge weights"—the numerical values that determine how the model processes and generates text.

Model sizes:

• GPT-3: 175 billion parameters

• GPT-4: ~1.7 trillion parameters (estimated, 8 expert mixture)

• Claude 3 Opus: Estimated ~500B-1T parameters

• LLaMA 2 70B: 70 billion parameters

• Gemini Ultra: Estimated 1.5T+ parameters

Bigger = better? Usually, but...

• ✅ More parameters = more knowledge capacity

• ✅ Better reasoning and nuance

• ❌ Much more expensive to run

• ❌ Slower response times

• ❌ Not always necessary for simpler tasks

Practical implication: A 70B open-source model might be perfect for your use case, saving you 90% in costs compared to GPT-4.

Why This Matters for Prompting

Understanding this architecture explains:

1. Why context order matters
The model processes your prompt sequentially (even though attention is parallel). Information later in your prompt gets more "attention."

Practical tip: Put the most important instructions near the end of your prompt.

2. Why the model can be confident yet wrong
It's predicting based on patterns, not retrieving facts from a database. High probability ≠ factually correct.

Practical tip: Request citations, use chain-of-thought prompting, verify critical information.

3. Why examples are so powerful
Examples directly influence the probability distribution for the next tokens.

Practical tip: Few-shot prompting works because you're literally showing the model the pattern you want.

4. Why some tasks are harder than others
Complex reasoning requires the model to maintain and manipulate abstract representations across many steps.

Practical tip: Break complex tasks into smaller steps (prompt chaining).

Part 2: Tokenization—The Hidden Language of AI

Here's something that will change how you write prompts forever: AI doesn't see words. It sees tokens.

What Are Tokens?

Tokens are the basic units of text that models process. They're not exactly words, not exactly characters—they're subword units.

The rough rule: 1 token ≈ 4 characters or ≈ 0.75 words in English

Examples:

Unknown"Hello world!" = 3 tokens ["Hello", " world", "!"]
"artificial intelligence" = 4 tokens ["art", "ificial", " intelligence"]
"ChatGPT" = 2 tokens ["Chat", "GPT"]
"antidisestablishmentarianism" = 6 tokens ["ant", "idis", "establish", "ment", "arian", "ism"]

Why not just use words?

• Many languages don't have clear word boundaries (Chinese, Japanese)

• Tokenization handles new/rare words better

• More efficient processing

• Handles numbers, punctuation, code, etc.

How Tokenization Works

Step 1: Byte-Pair Encoding (BPE)
The most common approach. The algorithm:

1. Starts with individual characters

2. Finds the most frequently occurring pairs

3. Merges them into single tokens

4. Repeats until reaching the desired vocabulary size

Result: Common words = 1 token, rare words = multiple tokens

Vocabulary sizes:

• GPT-3/4: ~50,257 tokens

• Claude: ~100,000 tokens

• LLaMA: ~32,000 tokens

Why Tokenization Matters for You

1. Cost Calculation

API pricing is per token, not per word:

• OpenAI GPT-4: $0.03/1K input tokens, $0.06/1K output tokens

• Claude Opus: $0.015/1K input tokens, $0.075/1K output tokens

Your 100-word prompt isn't 100 tokens—it's more like 130-150 tokens.

Pro tip: Use tokenization tools to check:

• OpenAI Tokenizer

• TikToken (Python library)

2. Context Window Limits

Models have token limits, not word limits:

• GPT-4: 8K, 32K, or 128K tokens

• Claude 3.5: 200K tokens

• Gemini 1.5 Pro: 1M tokens

Common mistake: "I can fit 100,000 words in Claude!"
Reality: 100,000 words = ~130,000-150,000 tokens—you'd exceed the limit.

3. Token Efficiency

Some ways of writing use fewer tokens:

Example:

Unknown❌ Inefficient (many tokens): 
"The individual who is responsible for the management and oversight of..."

✅ Efficient (fewer tokens):
"The manager of..."

But don't obsess over this. Clarity trumps token optimization except at massive scale.

4. Why Some Words Are "Harder" Than Others

Words that tokenize into more pieces are harder for the model:

Easy (1 token): "the", "and", "computer", "science"
Harder (multiple tokens): "antidisestablishmentarianism", rare names, specialized jargon

This is why:

• Models sometimes misspell unusual names

• They struggle with very rare technical terms

• They're better with common vocabulary

5. Numbers and Code

Numbers tokenize unpredictably:

Unknown"1234" might be ["123", "4"] or ["12", "34"] or ["1", "2", "3", "4"]

This is why LLMs are bad at arithmetic—they're predicting token sequences, not calculating.

Code tokenizes differently than prose:

Pythondef calculate_total(items):
    return sum(items)

This might be 15-20 tokens, depending on how the tokenizer handles code syntax.

Practical Tokenization Tips for Prompting

1. Check your token count before submission
Especially for long prompts approaching context limits.

2. Be aware of token-heavy formats

• JSON (lots of special characters)

• Tables (formatting characters)

• Code (syntax elements)

3. Don't pad unnecessarily
This doesn't help and wastes tokens: "Please, if you could, maybe, possibly..."

4. Use the model's native token count in API calls
Most APIs return usage.total_tokens—monitor this for cost tracking.

Part 3: Context Windows and Their Limitations

The context window is the total amount of text (in tokens) a model can consider at once—including both your prompt and its response.

Think of it as the model's "working memory."

Current Context Window Sizes

The race is on: Companies are competing to offer larger context windows.

Why Context Windows Matter

1. They Define What You Can Input

Want to analyze an entire book? You need:

• "The Great Gatsby" = ~47,000 words = ~63,000 tokens

• Required: 128K+ context window

2. They Include the Response

If you have 8K tokens total:

• Your prompt uses 6K tokens

• Only 2K tokens remain for the response

• That's ~1,500 words maximum output

Common issue: Long prompts with insufficient room for responses.

3. They Affect Attention Quality

Lost in the Middle Problem: Research shows models pay more attention to:

• The beginning of the context

• The end of the context

• Less attention to information in the middle

Study findings (Liu et al., 2023):

• Information at the start: ~70% retrieval accuracy

• Information in the middle: ~40% retrieval accuracy

• Information at the end: ~65% retrieval accuracy

Practical implication: Put critical information at the beginning or end of your prompt.

Context Window Limitations

1. Processing Time

Larger contexts = longer processing:

• 8K tokens: ~2-5 seconds

• 128K tokens: ~10-30 seconds

• 1M tokens: Several minutes

2. Cost

You pay for every token in the context:

UnknownExample: Analyzing a 100K token document
Input cost: 100K tokens × $0.015/1K = $1.50 per query
If you query 1,000 times: $1,500

3. Quality Degradation

Models perform worse as context windows fill:

• Needle-in-haystack tests show accuracy drops significantly beyond 50% capacity

• Complex reasoning degrades faster than simple retrieval

4. The Recency Bias

Models weight more recent information higher. In long contexts:

• Earlier information may be "forgotten"

• Later information dominates

Strategies for Working Within Context Limits

1. Summarization

UnknownStep 1: Summarize document in chunks
Step 2: Combine summaries
Step 3: Analyze final summary

2. Sliding Window
Process text in overlapping chunks:

UnknownChunk 1: Tokens 0-8000
Chunk 2: Tokens 6000-14000
Chunk 3: Tokens 12000-20000

3. Retrieval-Augmented Generation (RAG)
Don't put everything in context—retrieve only relevant sections:

Unknown1. Index all documents
2. User asks question
3. Retrieve top 5 relevant passages
4. Feed only those to the model

We'll cover this in detail in Post #18.

4. Prompt Compression
Remove unnecessary verbosity:

Unknown❌ "I would really appreciate it if you could possibly help me understand..."
✅ "Explain..."

5. Stateful Conversations
For chatbots, summarize history periodically:

UnknownEvery 10 messages:
- Summarize conversation so far
- Replace old messages with summary
- Continue with compressed context

The Future of Context Windows

Trends to watch:

• Infinite context: Research into models without fixed context limits

• Hierarchical attention: Models that process different context levels differently

• External memory: Models that can read/write to external storage

• Selective attention: Models that automatically focus on relevant parts

But for now: Work within the constraints. They're getting better, but they're not going away soon.

Part 4: Parameters That Control Output (Temperature, Top-p, and More)

These are the knobs and dials that change how the AI generates text. Understanding them is like understanding shutter speed and aperture in photography—technical knowledge that unlocks creative control.

Temperature: The Creativity Dial

Definition: Controls randomness in token selection. Range: 0.0 to 2.0 (practical range: 0.0 to 1.5)

How it works:

The model calculates probabilities for all possible next tokens:

Unknown"The sky is ____"
- blue: 40% probability
- clear: 25% probability
- cloudy: 20% probability
- falling: 0.1% probability

Temperature modifies these probabilities:

Temperature = 0 (Deterministic)

• Always picks the highest probability token

• Same prompt = same response every time

• Maximum consistency, zero creativity

Temperature = 0.7 (Default/Balanced)

• Mostly picks high-probability tokens

• Some variation allowed

• Balance of consistency and creativity

Temperature = 1.5 (High Creativity)

• Flattens probability distribution

• Low-probability tokens get more chances

• More creative, more unpredictable

• Higher risk of nonsense

Visual representation:

UnknownLow temp (0.2):   ████████ blue
                  ██ clear
                  █ cloudy

High temp (1.5):  ████ blue
                  ███ clear
                  ██ cloudy
                  █ falling (now more likely!)

When to Use Different Temperatures

Temperature 0 - 0.3: Maximum Precision

• ✅ Factual Q&A

• ✅ Code generation

• ✅ Data extraction

• ✅ Classification tasks

• ✅ Legal/medical applications

• ✅ Anything requiring consistency

Example prompt:

UnknownTemperature: 0
Extract the following from this email: sender, date, main request, urgency level.

Temperature 0.7 - 1.0: Balanced (Default)

• ✅ General conversation

• ✅ Explanations

• ✅ Content writing

• ✅ Problem-solving

• ✅ Most use cases

Example prompt:

UnknownTemperature: 0.7
Write a professional email declining a meeting request.

Temperature 1.0 - 1.5: Maximum Creativity

• ✅ Creative writing

• ✅ Brainstorming

• ✅ Unique content generation

• ✅ Exploring unconventional ideas

• ❌ Not for factual accuracy

Example prompt:

UnknownTemperature: 1.3
Generate 20 unusual marketing campaign ideas for a artisanal pickle company.

Temperature 1.5 - 2.0: Experimental

• Often produces nonsensical or incoherent results

• Rarely useful in practice

• Fun for experimentation

Top-p (Nucleus Sampling): The Alternative Control

Definition: Instead of temperature, controls which tokens are considered by cumulative probability. Range: 0.0 to 1.0

How it works:

Top-p = 0.9 means "consider tokens until their cumulative probability reaches 90%"

UnknownToken probabilities:
- blue: 40% (cumulative: 40%)
- clear: 25% (cumulative: 65%)
- cloudy: 20% (cumulative: 85%)
- bright: 10% (cumulative: 95%) ← stops here at top-p=0.9
- falling: 5% (excluded)

The difference from temperature:

• Temperature: Adjusts ALL probabilities

• Top-p: Excludes low-probability tokens entirely

Typical values:

• Top-p = 0.1: Very conservative (top 10% of probability mass)

• Top-p = 0.5: Moderately conservative

• Top-p = 0.9: Balanced (default for many models)

• Top-p = 1.0: Consider all tokens (no filtering)

Pro tip: Use EITHER temperature OR top-p, not both. Combining them can produce unexpected results.

Other Important Parameters

Max Tokens (Max Length)

Definition: Maximum number of tokens in the response.

Use cases:

• Controlling costs (shorter = cheaper)

• Enforcing brevity

• Ensuring responses fit in UI elements

Common mistake:

Unknown❌ Setting max_tokens = 50 and asking for a detailed essay
Result: Response will be cut off mid-sentence

Best practice:

Unknown✅ Set max_tokens to slightly more than expected output
For 500-word response: max_tokens = 700-800

Frequency Penalty

Definition: Reduces likelihood of repeating the same tokens. Range: -2.0 to 2.0

How it works:

• 0: No penalty (default)

• Positive values (0.5 - 1.0): Discourages repetition

• Negative values: Encourages repetition (rarely useful)

Use when:

• Model is being repetitive

• You want more diverse vocabulary

• Generating lists or variations

Example:

UnknownWithout frequency penalty:
"The product is great. The product is amazing. The product is fantastic."

With frequency penalty (0.7):
"The product is great. It's amazing. This is fantastic."

Presence Penalty

Definition: Encourages the model to introduce new topics. Range: -2.0 to 2.0

How it works:

• 0: No penalty (default)

• Positive values: Encourages talking about new concepts

• Negative values: Encourages staying on topic (rarely used)

Use when:

• Model keeps circling back to same points

• You want broader coverage

• Brainstorming diverse ideas

The difference:

• Frequency penalty: "Don't use the same WORDS repeatedly"

• Presence penalty: "Don't talk about the same TOPICS repeatedly"

Stop Sequences

Definition: Tokens that signal the model to stop generating.

Common use:

Pythonstop_sequences = ["\n\n", "###", "END"]

Use cases:

• Generating until specific delimiter

• Stopping at natural breakpoints

• Controlling output format

Example:

UnknownPrompt: "List 5 ideas. Use ### to separate each idea."
Stop sequence: "###"

Output: "Idea 1: Product launch ###"
(stops, preventing premature continuation)

Parameter Combinations for Common Tasks

1. Factual Q&A

JSON{
  "temperature": 0.1,
  "max_tokens": 500,
  "top_p": 0.9,
  "frequency_penalty": 0,
  "presence_penalty": 0
}

2. Creative Writing

JSON{
  "temperature": 1.0,
  "max_tokens": 2000,
  "top_p": 0.95,
  "frequency_penalty": 0.5,
  "presence_penalty": 0.3
}

3. Code Generation

JSON{
  "temperature": 0.2,
  "max_tokens": 1500,
  "top_p": 0.9,
  "frequency_penalty": 0.2,
  "presence_penalty": 0
}

4. Brainstorming

JSON{
  "temperature": 1.2,
  "max_tokens": 1000,
  "top_p": 0.95,
  "frequency_penalty": 0.8,
  "presence_penalty": 0.6
}

5. Conversation

JSON{
  "temperature": 0.7,
  "max_tokens": 800,
  "top_p": 0.9,
  "frequency_penalty": 0.3,
  "presence_penalty": 0.1
}

Experimentation Framework

To find optimal parameters:

1. Start with defaults (temp: 0.7, top_p: 0.9)

2. Test one variable at a time

3. Run multiple times (at temp > 0, outputs vary)

4. Measure results against your criteria

5. Document what works

Pro tip: Create a spreadsheet tracking:

Part 5: Different Model Architectures (GPT, Claude, Gemini, LLaMA, and More)

Not all LLMs are created equal. Understanding the landscape helps you choose the right tool for the job.

The Major Players: A Comparative Overview

OpenAI GPT Series

GPT-3.5 Turbo

• Size: 175B parameters

• Context: 16K tokens

• Strengths: Fast, cheap, good for most tasks

• Weaknesses: Less capable than GPT-4, more hallucinations

• Best for: High-volume, cost-sensitive applications

• Cost: $0.0015/1K input, $0.002/1K output

GPT-4

• Size: ~1.7T parameters (mixture of experts)

• Context: 8K, 32K, or 128K tokens

• Strengths: Best reasoning, complex tasks, instruction following

• Weaknesses: Expensive, slower

• Best for: Complex analysis, high-stakes content, reasoning

• Cost: $0.03/1K input, $0.06/1K output

GPT-4 Turbo

• Updated GPT-4 with better performance and longer context

• Context: 128K tokens

• Cost: Slightly cheaper than GPT-4

GPT-4V (Vision)

• Multimodal: text + images

• Can analyze screenshots, diagrams, photos

• Same core capabilities as GPT-4

Anthropic Claude Series

Claude 3 Haiku

• Size: Smallest in Claude 3 family

• Context: 200K tokens

• Strengths: Fastest, cheapest Claude, massive context

• Best for: Simple tasks needing large context

• Cost: $0.00025/1K input, $0.00125/1K output

Claude 3 Sonnet

• Size: Mid-tier

• Context: 200K tokens

• Strengths: Balance of capability and cost

• Best for: Most production use cases

• Cost: $0.003/1K input, $0.015/1K output

Claude 3.5 Sonnet

• Updated Sonnet with improved performance

• Better at: Coding, reasoning, nuanced tasks

• Context: 200K tokens

Claude 3 Opus

• Size: Largest Claude model

• Context: 200K tokens

• Strengths: Highest quality, excellent at complex reasoning

• Best for: Challenging tasks, long-document analysis

• Cost: $0.015/1K input, $0.075/1K output

Claude's Unique Characteristics:

• ✅ Constitutional AI (safety-focused training)

• ✅ Better at declining inappropriate requests

• ✅ Excellent at long-form analysis

• ✅ Strong performance on nuanced tasks

• ✅ Less verbose than GPT-4 (more concise)

Google Gemini Series

Gemini 1.0 Pro

• Context: 32K tokens

• Strengths: Multimodal (text, images, audio, video)

• Best for: Applications needing multiple input types

Gemini 1.5 Pro

• Context: 1M tokens (largest available)

• Strengths: Analyzing entire codebases, books, video transcripts

• Weaknesses: Long context = slower, expensive

• Best for: Tasks requiring massive context

Gemini Ultra

• Largest Gemini model

• Competitive with GPT-4 and Claude 3 Opus

• Multimodal capabilities

Gemini's Unique Characteristics:

• ✅ Native multimodal design (not bolted-on vision)

• ✅ Longest context window (1M tokens)

• ✅ Strong at technical/scientific tasks

• ✅ Deep Google integration

Meta LLaMA Series

LLaMA 2

• Sizes: 7B, 13B, 70B parameters

• License: Open source (with usage restrictions)

• Context: 4K-32K tokens depending on version

• Strengths: Can self-host, customize, fine-tune

• Best for: Privacy-sensitive applications, customization needs

LLaMA 3

• Improved over LLaMA 2

• Better multilingual support

• Enhanced reasoning

Open Source Advantages:

• ✅ Self-host (data stays internal)

• ✅ Fine-tune for specific domains

• ✅ No per-token costs (just compute)

• ✅ Full control over model behavior

Open Source Disadvantages:

• ❌ Requires infrastructure

• ❌ Need ML expertise for deployment

• ❌ Generally less capable than frontier models

• ❌ You handle all safety/moderation

Other Notable Models

Mistral AI

• Mistral 7B: Efficient, competitive with 13B models

• Mixtral 8x7B: Mixture of experts, strong performance

• Open source: Similar benefits to LLaMA

Cohere

• Command: Optimized for business applications

• Strong at: Classification, embeddings, search

AI21 Jurassic

• Jurassic-2: Various sizes

• Focus: Multi-language, long-form content

Specialized Models Worth Knowing

Code-Specific Models

CodeLlama (Meta)

• Based on LLaMA, trained on code

• Better at programming than base LLaMA

StarCoder

• Open source, 15B parameters

• Trained on 80+ programming languages

Phind-CodeLlama

• Fine-tuned CodeLlama for development

Embedding Models

OpenAI text-embedding-ada-002

• For semantic search, clustering

• 1536 dimensions

Cohere Embed

• Multilingual embeddings

• Various size options

Sentence Transformers

• Open source embedding models

• Self-hostable

Model Comparison Matrix

Architectural Differences That Matter

1. Mixture of Experts (MoE)

Used by: GPT-4, Mixtral

How it works:

• Multiple smaller "expert" models

• Router network decides which experts to activate

• Only activate relevant experts for each token

Advantages:

• More efficient than single large model

• Specialization (different experts for different domains)

Disadvantages:

• More complex to train and deploy

• Can be inconsistent

2. Constitutional AI

Used by: Claude series

How it works:

• Model is trained with explicit "constitution" of principles

• Self-critiques and revises responses

• Trained to explain refusals

Result: Claude tends to be more careful, explicit about limitations

3. Retrieval-Enhanced Generation

Used by: Some specialized models, Perplexity AI

How it works:

• Model can search external sources

• Grounds responses in retrieved information

• Provides citations

Advantage: Reduced hallucinations, up-to-date information

4. Multimodal Architecture

Native multimodal (Gemini):

• Trained on text, images, audio, video simultaneously

• Better cross-modal understanding

Adapter multimodal (GPT-4V):

• Base model + vision adapter

• Still very capable but different architecture

Choosing the Right Model: Decision Framework

Model Performance on Common Tasks

Part 6: Putting It All Together

Your Model Selection Worksheet

Answer these questions to choose the right model:

1. What's your primary task?

• Simple Q&A → GPT-3.5 Turbo, Claude Haiku

• Complex reasoning → GPT-4, Claude 3 Opus

• Coding → Claude 3.5 Sonnet, GPT-4

• Creative writing → Claude 3 Opus, GPT-4

• Document analysis → Claude 3 Opus, Gemini 1.5 Pro

2. What's your context requirement?

• < 8K tokens → Any model

• 8K-32K tokens → GPT-4, Claude, Gemini Pro

• 32K-200K tokens → Claude 3, GPT-4 Turbo

• 200K+ tokens → Gemini 1.5 Pro

3. What's your volume?

• Low (< 1M tokens/month) → Use best quality

• Medium (1M-10M) → Balance cost/quality

• High (> 10M) → Cost-optimize or self-host

4. What's your budget per 1K tokens?

• < $0.01 → GPT-3.5, Claude Haiku, self-host

• $0.01-$0.05 → Claude 3.5 Sonnet, GPT-4 Turbo

• $0.05 → GPT-4, Claude 3 Opus (when needed)

5. What's your latency requirement?

• Real-time (<2s) → GPT-3.5 Turbo, Claude Haiku

• Interactive (<5s) → Most models

• Batch processing → Any model, optimize for cost

6. Do you need special capabilities?

• Vision → GPT-4V, Gemini

• Massive context → Gemini 1.5 Pro

• Self-hosting → LLaMA, Mistral

• Maximum safety → Claude 3

Practical Exercises

Exercise 1: Token Counting

Test these prompts in a tokenizer:

1. "Hello world"

2. "The quick brown fox jumps over the lazy dog"

3. Your name (if it's unusual, see how it tokenizes)

4. A sentence in another language you speak

5. A code snippet

What did you learn about tokenization?

Exercise 2: Temperature Experiments

Use the same prompt with different temperatures:

Prompt: "Write a product description for wireless headphones"

Try:

• Temperature 0

• Temperature 0.7

• Temperature 1.2

Compare: Consistency, creativity, quality

Exercise 3: Context Window Testing

Take a long document (5,000+ words). Try:

1. Summarizing the whole thing

2. Asking about information at the beginning

3. Asking about information in the middle

4. Asking about information at the end

Notice: Where accuracy is highest/lowest

Exercise 4: Model Comparison

Same prompt, three different models:

Prompt: "Explain quantum entanglement to a high school student"

Test with:

• GPT-3.5 Turbo

• Claude 3.5 Sonnet

• GPT-4 (if available)

Compare: Clarity, accuracy, style, length

Common Mistakes to Avoid

❌ Mistake 1: Ignoring tokenization
"I'll just write naturally and not worry about it"
Problem: Hitting context limits unexpectedly, wasting tokens

❌ Mistake 2: Using maximum context always
"I'll always use the longest context available"
Problem: Slower, more expensive, worse quality at high capacity

❌ Mistake 3: Default parameters for everything
"I'll just use temperature 0.7 for all tasks"
Problem: Suboptimal results—code generation needs lower, creative needs higher

❌ Mistake 4: Not testing different models
"GPT-4 is best, so I'll use it for everything"
Problem: Overpaying for simple tasks, missing specialized strengths

❌ Mistake 5: Assuming determinism at temp > 0
"I ran it once, so that's what it always does"
Problem: Inconsistent results surprise you in production

❌ Mistake 6: Exceeding context without checking
"I'll just paste this whole document"
Problem: Truncated results, missed information, wasted tokens

❌ Mistake 7: Treating all models the same
"They're all LLMs, so prompts should work identically"
Problem: Each model has quirks, optimal prompting differs

Key Takeaways: What You Must Remember

🔹 LLMs predict text, they don't understand it
This explains why they can be confidently wrong.

🔹 Tokens ≠ words
Budget and plan in tokens, not words.

🔹 Context windows are your constraint
Design prompts that fit. Put critical info at the start/end.

🔹 Temperature controls creativity
Low for facts, high for creativity.

🔹 Different models have different strengths
Choose based on task, not just "best overall."

🔹 Parameters matter as much as the prompt
The same prompt with different parameters produces different results.

🔹 Bigger isn't always better
A well-prompted smaller model beats a poorly-prompted larger one.

🔹 Context quality > context quantity
200K tokens of irrelevant information < 2K tokens of perfect context.

What's Next

In Post #3: "Anatomy of an Effective Prompt", we'll take everything you've learned about how models work and translate it into practical prompt construction:

• The essential components every prompt needs

• Clear vs. vague instructions (with examples)

• How to provide context effectively

• Formatting outputs exactly as you want

• Common beginner mistakes and how to avoid them

• Your first prompt templates

Now you understand the engine. Next, you'll learn to drive it masterfully.

Resource Links

Tokenizers:

• OpenAI Tokenizer

• TikToken GitHub

Model Documentation:

• OpenAI Models

• Anthropic Claude

• Google Gemini

• Meta LLaMA

Research Papers:

• "Attention is All You Need" (Transformers)

• "Lost in the Middle" (Context window study)

• "Constitutional AI" (Anthropic's approach)

Your Assignment Before Post #3

1. Experiment with at least two different models on the same task

2. Try the same prompt with different temperatures (0, 0.7, 1.2)

3. Check the token count of your typical prompts

4. Test a long document against context limits

5. Document your findings—what surprised you?

Share your discoveries in the comments! What worked? What didn't? What confused you?

Next up: Post #3 - "Anatomy of an Effective Prompt"

You now understand the machine. Time to master the interface.

Questions? Confused about anything? Drop a comment—I read and respond to all of them. This series only gets better with your input.

Welcome to Level 2 of prompt engineering mastery. You're building something powerful.

Think of a Prompt Like a Recipe

Imagine you’re texting a friend to make dinner. You could say:

• “Make me dinner.”

• OR “Make me spaghetti with tomato sauce, extra garlic, and no cheese.”

The second version gives your friend all the details to make it just right. That’s what a great prompt does for AI: it hands over the full recipe, not just a wish list.

The Four Essential Ingredients of a Great Prompt

Let’s slice this recipe even finer. Whether you want a poem, a summary, or business insight from GenAI, these four ingredients are key:

1. Clarity

Be crystal-clear about what you want. Avoid fuzzy, open-ended requests.

Examples:

• Not clear: “Write something about birthdays.”

• Clear: “Write a funny birthday poem for a 12-year-old who loves dinosaurs.”

Clarity gets you results that actually match your needs.

2. Context

Provide background, purpose, or audience to help the AI dial in its answers.

Examples:

• Weak: “Suggest a gift.”

• Strong: “Suggest a birthday gift for my grandmother who enjoys gardening and lives in Florida.”

Context works like a north star, pointing the AI in the right direction.

3. Instructions

Tell AI what form you want: a list, bullet points, a story, or even a tone (formal, playful, technical, etc.).

Examples:

• Weak: “Explain rain.”

• Strong: “Explain how rain works to a 7-year-old, using simple words and a short story.”

The more you guide, the closer the results land.

4. Constraints

Set boundaries for length, amount, or format so you don’t get too much or too little.

Examples:

• Broad: “List movies.”

• Precise: “List three adventure movies suitable for family movie night.”

Constraints help AI respect your time and attention.

Bad vs. Good Prompts: See the Difference

Notice how the upgraded prompts include clarity, context, instructions, and constraints. The AI now knows exactly where you want to go and how to help you get there.

Real-Life Analogy: Giving Great Directions

Picture a friend trying to find your house:

• “Go that way until you see a tree.”
  (Confusion ahead!)

• “Drive straight for two blocks, turn left at the bakery, and my blue door’s on the right.”
  (Success!)

Great prompts are like great directions—they’re friendly, focused, and easy to follow.

Upgrade Challenge: Turn Weak Prompts Into Strong Ones

Try transforming these:

1. “Write an email.”
   → “Write a short, polite email to my professor requesting a deadline extension because I was sick.”

2. “What’s a good dinner?”
   → “What’s a quick vegetarian dinner recipe with just five ingredients?”

3. “Explain photosynthesis.”
   → “Explain photosynthesis to someone with no science background using a simple, step-by-step analogy.”

Practicing this skill makes prompting feel like second nature.

Beyond Basics: The Prompts that Power Professionals

You can use these principles anywhere:

• Business: “Create a SWOT analysis for a small online toy store looking to expand internationally. Limit each section to two bullet points.”

• Education: “Summarize Einstein’s theory of relativity in under 150 words for a high school science newsletter.”

• Personal: “List three creative, affordable date night ideas for couples in New York City.”

These detailed prompts set clear expectations, saving you time and frustration.

Key Takeaways

• The clearer, more contextual, and better-structured your prompt, the better and more relevant the result.

• Great prompts save you time, frustration, and guesswork, making GenAI more useful and delightful—no matter your goal or industry.

• Under the hood, your instructions are taken apart, analyzed, and rebuilt by a transformer model trained to spot patterns, follow rules, and respect every detail you provide.

Clear prompts aren’t just about saving time—they’re about unlocking the true magic of AI: getting answers that feel written just for you.

If you’ve ever wondered how people interact with AI tools, or why some folks seem to get exactly what they want from tools like ChatGPT while others receive confusing or generic responses, this article is for you. We’ll unveil the basics of Generative AI (GenAI), teach you about prompts, and set you up for success in the art of communicating with AI—no matter your background.

What is GenAI? Breaking Down the Buzzword

At its core, GenAI stands for Generative Artificial Intelligence. It’s not just another tech industry fad—it signifies a quantum leap in how computers can support us, work with us, and even create for us.

GenAI in Plain English

Think of GenAI as a highly intelligent assistant, one who doesn’t just fetch information, but who actually creates content. Whether you want it to write a story, summarize a report, sketch a logo, or compose a melody, GenAI listens to (or rather, “reads”) your instructions and produces something new. It doesn’t simply retrieve facts; it uses patterns learned from mountains of information to generate original outputs.

Real-Life Example: The Coffee Shop Scenario

Let’s put GenAI into a real-world situation you already know. Picture yourself at your favorite coffee shop:

• You say: “One cappuccino, please.”

  • The barista crafts your drink exactly to your taste.

But what if you’re less specific?

• You say: “Give me something hot.”

  • The barista could hand you tea, cocoa, an espresso… it’s anyone’s guess!

This is exactly how GenAI works: the clarity and detail of your request (or prompt) directly influences the success and accuracy of what you get back.

What is a Prompt? Your “Magic Wand” for GenAI

A prompt is what you say or write to “instruct” the AI. It’s your command, your wish, your ask—wrapped up in a short phrase, a question, or even a detailed scenario.

• Prompts can be simple:

  • “Summarize this article.”

• Or complex:

  • “Write a friendly email to my team, summarizing our project progress, highlighting Sarah’s achievement, and suggesting a virtual celebration.”

Analogy: The Genie in the Lamp

Recall fairy tales about genies who grant wishes:

• Vague wish: “I want to be happy.”

• Specific wish: “I want to have a picnic in Central Park this Saturday with my friends, sunny weather, and no bugs.”

The second wish is much more likely to lead to the outcome you desire. Similarly, your prompt is how you “wish” something out of GenAI—the clearer, the better.

Why Should You Care About Prompts?

Learning to use the right prompt is like finding the magic key that unlocks GenAI’s full potential. Good prompts can help you:

• Produce emails, reports, social posts, or creative writing.

• Analyze data, summarize complex information, or brainstorm fresh ideas.

• Automate repetitive tasks, create images or code snippets, and more.

Everyday Example: Email to Your Boss

Let’s make this practical:

• Poor prompt:

  • “Help me.”

    • (GenAI: “How can I help you?”)

• Better prompt:

  • “Please write me an email to my boss explaining I’m sick and can’t come to work.”

    • (GenAI: Produces a clear, professional message ready to send!)

Takeaway: The results improve dramatically when you put in a little thought, just like how your experience at a restaurant improves when you’re clear with your order.

More Real-World Examples Across Fields

Let’s broaden this with practical examples:

• Marketing:

  • “Create three catchy headlines for a spring sale on winter clothing.”

• Education:

  • “Explain the theory of relativity in simple terms for a 10-year-old.”

• Healthcare:

  • “Summarize this patient’s medical history in three sentences.”

• Design:

  • “Generate a logo concept for a modern vegan café, using green and white.”

• Programming:

  • “Write a Python script that sorts a list of names alphabetically.”

In each case, the clarity, context, and specificity of your prompt drives the quality of GenAI's response.

Behind the Scenes: How Does GenAI Actually Work?

While Generative AI can feel magical—responding instantly and intelligently to your prompts—what’s happening under the hood is a sophisticated blend of computer science, mathematics, and large-scale data engineering. Below, I’ll break down each key stage and component, using clear explanations and practical analogies to help you visualize what’s really going on during a typical GenAI interaction.

1. Pre-Training: Building the AI Brain

Massive Data Ingestion:

• GenAI models like GPT-4 are “pre-trained” on enormous datasets scraped from the internet, books, articles, code, images, and more.

• Data is tokenized (broken into small chunks), and each token is mapped to a numerical value that the model can manipulate.

Self-Supervised Learning:

• During training, the model is shown billions of sequences (sentences, code, etc.) and learns to predict the next token in a sequence.

• No human labels are required here—the model learns by trying to guess missing words or next words, seeing where it went wrong, and updating itself.

Emergence of Patterns:

• Over time, the neural network captures intricate relationships in language: grammar rules, contextual clues, cultural references, and even some logic.

• Weights and biases in the model are gradually tuned so that it gets better and better at predicting what comes next in any given context.

2. Prompt Processing: Interpreting Your Input

Tokenization:

• When you type a prompt, it’s instantly split into tokens (words, subwords, or even characters, depending on the model’s design).

• Example: “Write a summary of Hamlet” → [‘Write’, ‘a’, ‘summary’, ‘of’, ‘Ham’, ‘let’]

Embeddings:

• Each token is converted into a high-dimensional vector (embedding) that captures its semantic meaning.

• These vectors provide the foundation for all subsequent computations.

Neural Network Layers:

• The sequence of embeddings flows through dozens or even hundreds of transformer layers.

• Each layer adjusts its understanding by considering the position and context of each token within the entire prompt (“attention mechanisms”).

• These attention maps allow the AI to weigh meaning and relationships, just as a human would focus on key words in a sentence for overall meaning.

3. Token Prediction: Generating the Output

Stepwise Generation:

• The model doesn’t generate your answer all at once. Instead, it predicts one token at a time.

• After each token is predicted, the new token is appended to the prompt, and the model recalculates the probabilities for what should come next.

Probability Distribution:

• For every step, the AI constructs a probability distribution over its entire vocabulary, assigning a likelihood to each possible next token.

• The final output is selected via “sampling”—potentially influenced by parameters like temperature.

Temperature (Creativity Control):

• Lower temperature = safer, more repetitive/accurate answers

• Higher temperature = more creative and surprising outputs, but sometimes less precise.

Example:
If you prompt, “Once upon a time in a faraway kingdom,” the model scores all likely next words: ‘there’, ‘a’, ‘lived’, etc.—and picks the most probable, or sometimes a slightly less probable one if sampling is used.

4. Advanced Features: Context Window, Memory, and Roles

Context Window:

• Each model has a “context window” (e.g., 8,000 or even 128,000 tokens), which is the maximum amount of input and output it can consider at once.

• Input outside this window is forgotten, so precise or recent information is prioritized.

Conversation Memory:

• Some advanced models can hold onto information across turns in a conversation using session memory.

• This allows for coherent responses even when context is spread over several exchanges.

System and Role Prompts:

• You can set context or roles (e.g., “Act as a financial analyst”) to steer the AI’s tone, expertise, or style throughout the conversation.

• These system prompts act as invisible guide rails, subtly influencing the AI’s output.

5. End-to-End Example

Let’s tie it all together with a simple walk-through:

1. Prompt Submitted: “Explain quantum computing to a 10-year-old.”

2. Tokenization: Prompt broken into manageable tokens.

3. Embedding & Encoding: Each token mapped into a vector; processed by the transformer network.

4. Attention & Context: Model pays special attention to key concepts (“quantum computing”, “10-year-old”).

5. Stepwise Prediction: AI predicts, generates, and appends each new token until the sentence is finished.

6. Output Displayed: The generated text appears on your screen—reading like an expert’s explanation, but tailored to a child’s understanding.

Technical Example

Suppose you prompt:
“Write a SQL query to select the top 5 products by sales from orders table.”

Behind the curtain:

• The AI parses the request, recognizes technical keywords (SQL, select, top, sales, orders table), and assembles a likely correct statement from millions of similar examples in its training data.

• It will output:

    SELECT product_id, SUM(sales) AS total_sales
    FROM orders
    GROUP BY product_id
    ORDER BY total_sales DESC
    LIMIT 5;
  

• You didn’t have to code from scratch—the prompt did the heavy lifting!

Tips for Writing Great Prompts (and Avoiding Frustration)

• Be Clear and Direct: State exactly what you want, with as much relevant detail as necessary.

• Define Role or Tone: If you want a friendly email, say so. If you want a technical explanation, specify the audience.

• Use Examples: If appropriate, share a template or sample within your prompt.

• Iterate: If you don’t get what you want the first time, tweak your prompt and try again—like clarifying your order at a café.

Why Prompt Engineering is the “New Literacy”

In many ways, prompt engineering is becoming a vital skill—like reading or basic computer literacy. The more you practice, the more powerful and precise your interactions with AI will become.

• Students can leverage AI for studying and homework help.

• Professionals speed up research, drafting, analytics, and more.

• Creatives expand their toolkit, generating fresh ideas at the push of a button.

Prompt engineering isn’t about “talking to robots”—it’s about shaping technology so it truly works for you.

Tracking your daily routine.

🧠 Why Track as a Developer?

Here’s what happens when you start logging your day as a coder:

• ✅ Clarity: You get a clear view of how you spend your hours—coding, meetings, learning, procrastinating.

• 🚀 Progress: You see real growth, even in small wins like solving a bug or learning a new tool.

• 🧘 Balance: You catch when you’re burning out or skipping breaks.

• 🛠️ Reflection: You spot what works and what doesn’t, like that late-night debugging session that ruined your sleep.

But let’s face it, digital tools often feel too rigid or too distracting.

The “404: Motivation Not Found” Journal comes in here.

📝 What’s the 404 Journal?

“404: Motivation Not Found” isn’t just a clever name. It’s a physical (or printable) developer-friendly journal crafted to keep your coding life in check. It’s made by a developer, for developers.

🔍 Features Include:

• ✅ Daily Tracker – Log your tasks, code time, mood, bugs fixed, and breakthroughs.

• 📆 Weekly Recap – Review what worked and what didn’t.

• 📈 Productivity Graphs – Visual sections to plot your effort vs. results.

• ☕ Coffee/Water/Mental Break Tracker – Because we know caffeine powers Git commits.

• 💬 Coding Memes + Quotes – A sprinkle of humour to get you through that 500th error.

• 🧠 Brain Dump Area – Jot ideas, bugs, or random thoughts so you don’t forget them mid-debug.

• 💻 Cheat Sheets – Quick tips for Git, VS Code, Linux, Python, Regex, and more.

👨‍💻 Who’s It For?

Whether you’re:

• A student learning to code,

• A junior dev juggling bugs and tutorials,

• A senior engineer who wants more structure,

• Or a freelancer managing your time and projects,

The 404 Journal is your non-digital accountability partner.

📦 Where to Get It

You can grab your copy of the “404: Motivation Not Found” Journal here:

🔗 Buy Now on Pothi.com (404-motivation-no-found)

🔗 Buy Now on Amazon.com (404-motivation-no-found)

Available in:

• Paperback (6×9) – Perfect to carry around

🎯 Final Thoughts

You don’t need another app.

You need a system that sticks.

Writing things down—on real paper—keeps you focused, grounded, and aware of how awesome (or unmotivated) your day was. So the next time you hit a mental block or feel stuck in a loop of debugging and burnout…

Grab your 404 journal, flip to a fresh page, and take back control of your dev routine—one log at a time.

Stay productive. Stay sane. Keep coding.

JSON (JavaScript Object Notation) is a widely used data interchange format, especially in web development and configuration management. When dealing with JSON data, it's common to encounter scenarios where you need to compare two JSON objects to find differences. Whether you're tracking changes in configurations, validating data updates, or debugging code, having a reliable JSON diff checker at your disposal can be a lifesaver.

In this comprehensive guide, we'll dive into the world of JSON diff checking and explore how to build an advanced JSON diff checker in Python. We'll leverage the power of the deepdiff library to perform precise comparisons and the termcolor library to provide a visually appealing and user-friendly output.

Prerequisites:

Before we get started, there are a few prerequisites you should have in place:

• Basic knowledge of Python programming.

• Python installed on your system.

• Familiarity with installing Python libraries using pip.

• Access to a code editor or integrated development environment (IDE).

Section 2: Choosing the Right Tools

In our journey to build an advanced JSON diff checker in Python, the first step is selecting the right tools for the job. Fortunately, the Python ecosystem offers two essential libraries that will make our task significantly easier: deepdiff and termcolor.

deepdiff: Precise JSON Comparison

The deepdiff library is a powerful tool for comparing complex data structures in Python, including JSON objects. It provides a comprehensive set of functionalities to identify differences between two data structures, making it perfect for our JSON diff checker.

To install deepdiff, open your terminal or command prompt and run the following command:

pip install deepdiff

We'll explore how to use deepdiff in the upcoming sections to perform precise JSON comparisons.

termcolor: Enhancing Output

While the raw differences between JSON objects are valuable, presenting these differences in a human-readable and visually appealing format enhances the user experience. This is where the termcolor library comes into play.

termcolor allows us to add colors and styling to our console output. By color-coding the differences between JSON objects, we can quickly identify added, removed, and modified keys and values, making it easier to understand and act upon the differences.

To install termcolor, use the following command:

pip install termcolor

Section 3: Preparing Sample JSON Data

To effectively test and demonstrate our advanced JSON diff checker, we need sample JSON data with various complexities. In this section, we'll create two JSON objects with nested structures, representing before-and-after states of data.

Let's start with our first JSON object, which we'll call json_obj1:

json_obj1 = {
    "name": "John",
    "age": 30,
    "city": "New York",
    "hobbies": ["reading", "hiking"]
}

In json_obj1, we have a person's details, including their name, age, city, and a list of hobbies. This JSON structure is straightforward and serves as our initial data state.

Now, let's create the second JSON object, json_obj2, which represents a modified version of the data:

json_obj2 = {
    "name": "Jane",
    "city": "San Francisco",
    "country": "USA",
    "hobbies": ["hiking", "painting"]
}

In json_obj2, we see several changes compared to json_obj1. Jane's name has replaced John's, the city is different, a new "country" key has been added, and her hobbies have been updated.

These two JSON objects will serve as our testing data to demonstrate how our JSON diff checker detects differences. In the upcoming sections, we'll use deepdiff and termcolor to perform the comparison and present the results in a user-friendly format.

Feel free to use these sample JSON objects for testing or replace them with your own data as needed.

Section 4: Comparing JSON Objects

Now that we have our sample JSON data prepared (json_obj1 and json_obj2), it's time to dive into the core of our advanced JSON diff checker: comparing these JSON objects and identifying their differences.

We'll leverage the deepdiff library, which provides powerful tools for comparing complex data structures, including JSON objects. Here's how we can perform the comparison:

Step 1: Installing deepdiff (if not already installed)

If you haven't installed the deepdiff library yet, you can do so by running:

pip install deepdiff

Step 2: Importing deepdiff

In your Python script, make sure to import deepdiff:

from deepdiff import DeepDiff

Step 3: Comparing JSON Objects

To compare our JSON objects (json_obj1 and json_obj2), we can use the following code:

# Find differences between the two JSON objects
diff = DeepDiff(json_obj1, json_obj2, ignore_order=True)

In this code:

• DeepDiff is used to perform the comparison between json_obj1 and json_obj2.

• ignore_order=True ensures that the order of items in lists or dictionaries won't affect the comparison result, making it more suitable for JSON comparisons.

Step 4: Categorizing Differences

The diff variable now contains the differences between the two JSON objects. These differences are categorized into various types, such as "dictionary_item_added," "dictionary_item_removed," and "values_changed." We can use these categories to identify the nature of the differences.

Section 5: Colorizing the Output

In the previous section, we successfully compared our JSON objects using the deepdiff library and categorized the differences. However, a plain textual representation of these differences can be challenging to read and understand, especially when dealing with complex JSON data.

To enhance the user experience and make our JSON diff checker more user-friendly, we can add color-coded formatting to the output. This will make it easier to identify added, removed, and modified keys and values at a glance.

For this purpose, we'll use the termcolor library, which allows us to add colors and styling to console text.

Step 1: Installing termcolor (if not already installed)

Ensure you have the termcolor library installed by running:

pip install termcolor

Step 2: Importing termcolor

In your Python script, import the colored function from the termcolor library:

from termcolor import colored

Step 3: Color-Coding the Output

Now that we have termcolor imported, we can use the colored function to add colors to our JSON diff checker output. Here's an example of how to apply color coding to the differences:

# Print the differences with color-coded output
for category, changes in diff.items():
    if category == 'dictionary_item_added':
        print(colored("Added Keys:", 'green'))
    elif category == 'dictionary_item_removed':
        print(colored("Removed Keys:", 'red'))
    elif category == 'values_changed':
        print(colored("Modified Values:", 'yellow'))

    for change in changes:
        print(f"{change}: {changes[change]}")

In this code:

• We iterate through the categories of differences (dictionary_item_added, dictionary_item_removed, and values_changed).

• We use the colored function to apply colors to the category labels for added keys (green), removed keys (red), and modified values (yellow).

By color-coding the output, we make it much easier to spot differences and understand the nature of the changes between JSON objects.

Section 6: Full Code Example

In this section, we'll provide a complete Python code example that demonstrates our advanced JSON diff checker in action. We'll incorporate everything we've discussed so far, including comparing JSON objects using deepdiff and enhancing the output with color-coded formatting using the termcolor library.

import json
from deepdiff import DeepDiff
from termcolor import colored

# Sample JSON objects
json_obj1 = {
    "name": "John",
    "age": 30,
    "city": "New York",
    "hobbies": ["reading", "hiking"]
}

json_obj2 = {
    "name": "Jane",
    "city": "San Francisco",
    "country": "USA",
    "hobbies": ["hiking", "painting"]
}

# Compare JSON objects
diff = DeepDiff(json_obj1, json_obj2, ignore_order=True)

# Function to colorize the output
def colorize(text, color):
    return colored(text, color, attrs=["bold"])

# Print the differences with color-coded output
for category, changes in diff.items():
    if category == 'dictionary_item_added':
        print(colorize("Added Keys:", 'green'))
    elif category == 'dictionary_item_removed':
        print(colorize("Removed Keys:", 'red'))
    elif category == 'values_changed':
        print(colorize("Modified Values:", 'yellow'))

    for change in changes:
        print(f"{change}: {changes[change]}")

# Print the full JSON objects for reference
print(colorize("\nJSON Object 1:", 'cyan'))
print(json.dumps(json_obj1, indent=4))

print(colorize("\nJSON Object 2:", 'cyan'))
print(json.dumps(json_obj2, indent=4))

In this complete code example:

• We define our sample JSON objects (json_obj1 and json_obj2) that represent the data before and after changes.

• We use DeepDiff to compare these JSON objects and store the differences in the diff variable.

• We create a colorize function to apply color coding to our output using termcolor.

• We iterate through the categories of differences (added keys, removed keys, and modified values) and print them with color-coded formatting.

• Finally, we print the full JSON objects for reference.

With this code, you have a fully functional JSON diff checker that not only identifies differences but also presents them in a visually appealing and user-friendly manner.

Section 7: Testing the JSON Diff Checker

Now that we have our advanced JSON diff checker code in place, it's time to put it to the test. We'll use our sample JSON objects, json_obj1 and json_obj2, to see how our checker performs and how it helps us identify differences between them.

Step 1: Run the Code

Copy the entire code example provided in the previous section and run it in your Python environment. Make sure you have both the deepdiff and termcolor libraries installed.

Step 2: Review the Output

When you run the code, you'll see the JSON diff checker in action. It will categorize the differences between json_obj1 and json_obj2 into added keys, removed keys, and modified values. The output will be color-coded for easy identification:

• Added Keys (Green)

• Removed Keys (Red)

• Modified Values (Yellow)

For example, if "name" has changed from "John" to "Jane," you'll see a yellow-highlighted entry indicating the modified value.

Step 3: Interpret the Results

Review the output carefully, and you'll notice how our JSON diff checker precisely identifies the differences between the two JSON objects. This can be particularly valuable when working with configuration files, tracking changes in data, or debugging code.

Feel free to modify json_obj1 and json_obj2 to create your own test scenarios. This way, you can see how the JSON diff checker handles various types of changes and updates.

Section 8: Conclusion and Further Enhancements

Congratulations! You've successfully created an advanced JSON diff checker in Python using the deepdiff and termcolor libraries. You've learned how to compare JSON objects with precision and present the results in a user-friendly and visually appealing format.

As you explore the possibilities and applications of your JSON diff checker, consider the following:

Further Enhancements:

1. Interactive Input: Allow users to input JSON data interactively, making your tool more versatile.

2. Customization: Provide options for users to customize the color-coding and output format according to their preferences.

3. Batch Processing: Extend your tool to compare multiple JSON files or objects in batch mode.

4. File Input/Output: Implement features to read JSON data from files and save the results to files for future reference.

Use Cases:

1. Configuration Management: Use your JSON diff checker to track changes in configuration files, ensuring that modifications are properly documented.

2. Data Validation: Validate changes in data structures, such as database schemas or API responses, to ensure data integrity.

3. Version Control: Incorporate your checker into version control workflows to detect and manage JSON changes in software projects.

4. Debugging: Debugging complex JSON data can be easier when you can quickly spot differences between expected and actual data.

By continuously improving your JSON diff checker and exploring various use cases, you'll find that it becomes a valuable tool in your arsenal, simplifying tasks that involve comparing and analyzing JSON data.

Thank you for joining us on this journey to create an advanced JSON diff checker in Python. We hope this guide has been informative and that your new tool serves you well in your coding and data management adventures.

Feel free to leave any feedback or questions in the comments, and happy coding!

This concludes our guide on building an advanced JSON diff checker in Python. You now have a versatile tool that can help you in various programming and data management tasks. If you have any further questions or need assistance with any aspect of this project, please don't hesitate to ask.

Stock technical analysis is a critical component of the investment decision-making process. Traditionally, analysts rely on historical price charts and technical indicators to predict future stock price movements. However, with the advancement of technology, we can enhance this process by incorporating machine learning techniques into our analysis. In this article, we will explore how to use Python and machine learning to improve stock technical analysis.

1. Data Collection

The first step in this journey is to gather historical stock price data. Python offers several libraries and APIs for this purpose. One popular choice is the yfinance library, which allows you to fetch historical stock data from Yahoo Finance. For example:

import yfinance as yf
ticker = "AAPL"
data = yf.download(ticker, start="2020-01-01", end="2021-01-01")

2. Data Preprocessing

Once you have your data, it's crucial to preprocess it. This involves cleaning and transforming the data into a suitable format for analysis. You may also need to handle missing values and outliers.

3. Feature Engineering

In stock analysis, technical indicators play a significant role. These indicators can be calculated from historical price data and provide valuable insights. Common technical indicators include moving averages (e.g., Simple Moving Average or SMA), Relative Strength Index (RSI), Moving Average Convergence Divergence (MACD), and more.

data['SMA'] = data['Close'].rolling(window=20).mean()
data['RSI'] = compute_rsi(data['Close'])
data['MACD'] = compute_macd(data['Close'])

4. Model Selection

For stock price prediction, machine learning models come into play. While there are various models to choose from, let's consider a straightforward example using a Random Forest Regressor from the scikit-learn library.

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=100, random_state=42)

5. Data Splitting

To evaluate the model's performance, you need to split your data into training and testing sets. The training set is used to train the model, while the testing set assesses its predictive capabilities.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

6. Model Training

With your data prepared and split, you can now train your machine learning model. This involves feeding historical features (e.g., technical indicators) to the model to predict future stock prices.

model.fit(X_train, y_train)

7. Model Evaluation

After training, it's crucial to evaluate the model's performance. Common evaluation metrics for regression tasks include Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and R-squared (R2).

from sklearn.metrics import mean_absolute_error
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f"Mean Absolute Error: {mae}")

Conclusion

Incorporating machine learning into stock technical analysis using Python can enhance your ability to predict price movements and make informed investment decisions. However, it's essential to remember that stock markets are influenced by various factors, and no model can guarantee accurate predictions. Therefore, always use these models as tools to aid your analysis rather than relying solely on automated decisions. Additionally, continually update your models and adapt to changing market conditions to maximize their effectiveness.

Artificial General Intelligence (AGI), often referred to as GenAI, represents a technological frontier that promises transformative changes in our society. However, with great power comes great responsibility. In this article, we explore the significant disadvantages associated with GenAI and the strategies to overcome these challenges, ensuring that we harness its potential while mitigating its drawbacks.

Disadvantages of GenAI

1. Ethical Concerns

   AGI's capacity for autonomous decision-making raises ethical dilemmas. For example, in autonomous vehicles, if an AGI system must choose between two potential accident scenarios, who bears responsibility for its decision? To address this, ethical frameworks and regulations must be established, accompanied by oversight bodies to ensure accountability [1].

2. Job Displacement

   The rise of AGI could lead to massive job displacement as it outperforms humans in various industries. For instance, in manufacturing, AGI-powered robots can perform repetitive tasks more efficiently. Reskilling and education programs, such as the one initiated by the European Union [2], can help workers transition to new roles and protect the workforce.

3. Security Risks

   AGI poses significant security risks if misused. Robust cybersecurity measures are essential. For example, AI-powered threat detection systems can help protect AGI systems from hacking attempts [3]. Additionally, authentication and access controls, like those used in blockchain technology, can safeguard AGI from unauthorized access.

4. Lack of Accountability

   As AGI becomes more autonomous, attributing responsibility becomes complex. Systems for tracing and attributing AGI actions, like blockchain-based audit trails, should be put in place, alongside clear lines of responsibility [4].

5. Bias and Fairness

   AGI systems can inherit biases from their training data, perpetuating societal biases. For example, facial recognition software trained on biased data may misidentify certain ethnic groups. Ongoing auditing and diversifying AI development teams are crucial to mitigate bias [5].

6. Dependency and Control

   Striking the right balance between AGI and human control is essential. AGI should complement human capabilities, not replace them. For example, in healthcare, AGI can assist doctors in diagnosis but should not replace their expertise. Human-AI collaboration in decision-making processes can help achieve this balance [6].

7. Privacy Concerns

   AGI's data analysis capabilities can threaten individual privacy. Strengthened data protection regulations, like the European Union's GDPR, and practices such as federated learning can protect personal data [7]. Federated learning allows AI models to be trained locally on users' devices without sharing raw data.

8. Economic Inequality

   The development and deployment of AGI may exacerbate economic inequality. Policies to redistribute AGI benefits, such as a progressive tax system, and considering safety nets like universal basic income can address this issue [8].

9. Unintended Consequences

   The complexity of AGI systems makes predicting their behavior challenging. For example, in autonomous finance, AGI trading algorithms can lead to market volatility. Research and development should focus on understanding and mitigating potential risks and consequences [9].

Overcoming the Disadvantages

1. Ethical Frameworks and Regulations: Establish clear ethical guidelines and regulations for AGI development and deployment, backed by oversight bodies [1].

2. Job Transition and Reskilling: Invest in education and training programs to help individuals transition to new job roles and protect the workforce [2].

3. Cybersecurity Measures: Enhance cybersecurity to protect AGI systems from potential threats and unauthorized access [3].

4. Accountability Mechanisms: Develop systems for tracing and attributing AGI actions, ensuring clear lines of responsibility [4].

5. Bias Mitigation: Continuously audit and improve AGI algorithms to reduce bias and promote diversity in AI development teams [5].

6. Human-AI Collaboration: Encourage collaboration between humans and AI in decision-making processes [6].

7. Data Privacy Protections: Strengthen data protection regulations and implement data anonymization and encryption practices [7].

8. Economic Policies: Implement policies to redistribute AGI benefits and consider safety nets like universal basic income [8].

9. Risk Assessment and Mitigation: Invest in research to understand and mitigate the potential risks and unintended consequences of AGI [9].

Conclusion

GenAI holds the promise of revolutionizing our world, but it comes with its share of challenges. By proactively addressing ethical concerns, job displacement, security risks, and other issues, we can maximize the benefits of AGI while minimizing its drawbacks. The key lies in a collaborative effort involving governments, industries, researchers, and society to ensure a responsible and sustainable future with GenAI.

References:

[1] "Ethics Guidelines for Trustworthy AI." European Commission, April 2019.

[2] "Skills for Industry: The European Commission's Blueprint for Sectoral Cooperation on Skills." European Commission, December 2020.

[3] Doshi, Amisha et al. "Adversarial Attacks on Machine Learning Systems for Remote Sensing: Challenges and Future Directions." arXiv preprint arXiv:2012.11592, 2020.

[4] Tapscott, Don, and Alex Tapscott. "Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World." Penguin, 2016.

[5] Obermeyer, Ziad et al. "Dissecting racial bias in an algorithm used to manage the health of populations." Science, 2019.

[6] Topol, Eric J. "High-Performance Medicine: The Convergence of Human and Artificial Intelligence." Nature Medicine, 2019.

[7] "General Data Protection Regulation (GDPR)." European Union, May 2018.

[8] Bessen, James E. "AI and Jobs: The Role of Demand." NBER Working Paper No. 24235, 2018.

[9] Verma, Hitesh et al. "Algorithmic Trading: A Review and Evaluation of Recent Strategies." AI & Society, 2020.

Reference Links:

1. Ethics Guidelines for Trustworthy AI - European Commission
2. Skills for Industry: The European Commission's Blueprint for Sectoral Cooperation on Skills - European Commission
3. Adversarial Attacks on Machine Learning Systems for Remote Sensing - arXiv
4. Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World - Penguin
5. Dissecting racial bias in an algorithm used to manage the health of populations - Science
6. High-Performance Medicine: The Convergence of Human and Artificial Intelligence - Nature Medicine
7. General Data Protection Regulation (GDPR) - European Union
8. AI and Jobs: The Role of Demand - NBER
9. Algorithmic Trading: A Review and Evaluation of Recent Strategies - AI & Society

Python's extensive library ecosystem is a goldmine of tools that can turbocharge your daily coding tasks. While libraries like NumPy, Pandas, and Matplotlib are celebrated, a host of lesser-known Python libraries can revolutionize your daily work. In this article, we'll delve into the top 25 hidden gems that may have slipped under your radar but hold immense potential to enhance your productivity. Each library will be accompanied by practical examples to illustrate its utility.

1. Arrow - Simplify Date and Time Handling

Dealing with dates and times can be a source of frustration, but Arrow simplifies the process with its user-friendly API. Say goodbye to obscure datetime formats and cumbersome calculations.

Example:

import arrow

now = arrow.now()
tomorrow = now.shift(days=1)
print(tomorrow.format('YYYY-MM-DD'))

2. Fuzzywuzzy - Fuzzy String Matching

Fuzzywuzzy is your secret weapon for approximate string matching. It calculates similarity scores between strings, making it perfect for tasks like deduplication or record linkage.

Example:

from fuzzywuzzy import fuzz

similarity = fuzz.ratio("apple", "apples")
print(similarity)  # Output: 91

3. Pyquery - jQuery-Like Web Scraping

Pyquery simplifies web scraping with its intuitive jQuery-like syntax. No more wrestling with complex selectors; Pyquery makes extracting data from HTML or XML documents a breeze.

Example:

from pyquery import PyQuery as pq

html = "<div><p>Hello, World!</p></div>"
doc = pq(html)
text = doc('p').text()
print(text)  # Output: Hello, World!

4. PrettyTable - Create Readable Tables

Displaying data in tabular form is made elegant with PrettyTable. It allows you to generate attractive ASCII tables from your data, making your reports and presentations shine.

Example:

from prettytable import PrettyTable

table = PrettyTable()
table.field_names = ["Name", "Age"]
table.add_row(["Alice", 28])
table.add_row(["Bob", 35])
print(table)

5. Rich - Beautify Terminal Output

Rich enhances terminal-based applications, enabling you to create colourful text-based user interfaces for command-line applications. Impress your users with visually appealing outputs.

Example:

from rich import print

print("[bold green]Hello, World![/bold green]")

6. Typer - Rapid CLI Development

Typer is your go-to library for rapidly developing command-line interfaces (CLIs). It simplifies argument parsing and helps text generation, reducing the hassle of CLI development.

Example:

import typer

app = typer.Typer()

@app.command()
def greet(name: str):
    typer.echo(f"Hello, {name}!")

if __name__ == "__main__":
    app()

7. dateutil - Powerful Date and Time Handling

Working with dates and times just got a whole lot easier with dateutil. This library offers extensive functionality for parsing and manipulating dates and times, surpassing Python's built-in datetime module.

Example:

from dateutil import parser

date_string = "2023-09-30T15:30:00"
parsed_date = parser.parse(date_string)
print(parsed_date)

8. tqdm - Beautiful Progress Bars for Loops

Visualizing the progress of tasks within loops is a breeze with tqdm. Say goodbye to boring print statements and hello to elegant progress bars that keep you informed.

Example:

from tqdm import tqdm
import time

for i in tqdm(range(100)):
    time.sleep(0.1)  # Simulate some work

9. pandasql - SQL Queries on Pandas DataFrames

If you love working with Pandas DataFrames and SQL, pandasql is your perfect match. It seamlessly integrates SQL queries with Pandas DataFrames, allowing you to leverage your SQL skills for data manipulation.

Example:

import pandas as pd
from pandasql import sqldf

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
pysqldf = sqldf()
result = pysqldf("SELECT * FROM df WHERE A > 1")
print(result)

10. TinyDB - Lightweight NoSQL Database

TinyDB is a small but mighty NoSQL database that's perfect for small to medium-sized projects. It's easy to use and doesn't require a separate server, making it a versatile choice.

Example:

from tinydb import TinyDB, Query

db = TinyDB('my_db.json')
db.insert({'name': 'Alice', 'age': 30})
db.insert({'name': 'Bob', 'age': 25})

User = Query()
result = db.search(User.name == 'Alice')
print(result)

11. Unidecode - Transliterate Unicode to ASCII

Unidecode comes to the rescue when you need to transliterate Unicode text into ASCII characters. It simplifies text normalization and handling non-ASCII characters.

Example:

from unidecode import unidecode

text = "Tōkyō"
ascii_text = unidecode(text)
print(ascii_text)  # Output: "Tokyo"

12. tablib - Handle Tabular Data Formats with Ease

Tablib simplifies the handling of tabular data formats like Excel, CSV, and JSON. It's a versatile library that can save you time when dealing with diverse data sources.

Example:

import tablib

data = tablib.Dataset()
data.headers = ['Name', 'Age']
data.append(['Alice', 28])
data.append(['Bob', 35])

# Export to Excel
with open('data.xlsx', 'wb') as f:
    f.write(data.export('xlsx'))

13. psutil - System Monitoring and Management

Psutil provides a wealth of information about system utilization, allowing you to monitor processes and manage system resources efficiently.

Example:

import psutil

# Get CPU usage
cpu_usage = psutil.cpu_percent()
print(f'CPU Usage: {cpu_usage}%')

# List running processes
processes = psutil.process_iter(attrs=['pid', 'name'])
for process in processes:
    print(process.info())

14. APScheduler - Job Scheduling in Python

APScheduler empowers you to schedule Python functions to run at specific intervals or times. It's a handy tool for automating repetitive tasks.

Example:

from apscheduler.schedulers.blocking import BlockingScheduler

def job_to_run():
    print("Scheduled job ran!")

scheduler = BlockingScheduler()
scheduler.add_job(job_to_run, 'interval', seconds=10)
scheduler.start()

15. docx2txt - Extract Text from DOCX Files

Docx2txt is a valuable library for extracting text content from Microsoft Word (DOCX) files. It's particularly useful for document analysis and data extraction.

Example:

from docx2txt import process

text = process("document.docx")
print(text)

16. pyperclip - Simplify Clipboard Operations

Pyperclip simplifies clipboard operations in Python, making it easy to copy and paste text across different applications and platforms.

Example:

import pyperclip

# Copy text to clipboard
pyperclip.copy("This text is copied to the clipboard")

# Paste text from clipboard
clipboard_text = pyperclip.paste()
print(clipboard_text)

17. mtranslate - Multilingual Text Translation

Mtranslate is a powerful library for multilingual text translation. It offers quick and easy translation capabilities to multiple languages using various translation services, making it an asset for globalized applications.

Example:

from mtranslate import translate

text = "Hello, World!"
translated_text = translate(text, 'es')  # Translate to Spanish
print(translated_text)

18. simpy - Discrete Event Simulation

Simpy is a discrete event simulation library that's invaluable for modeling and simulating complex systems, such as traffic flow, supply chains, or resource allocation scenarios.

Example:

import simpy

def car(env):
    while True:
        print(f"Car at {env.now}")
        yield env.timeout(2)

env = simpy.Environment()
env.process(car(env))
env.run(until=10)

19. phonenumbers - Phone Number Parsing and Formatting

Phonenumbers simplifies phone number parsing, formatting, and validation. It's an essential tool for applications dealing with international phone numbers.

Example:

import phonenumbers

phone_number = phonenumbers.parse("+1 650-253-0000", None)
formatted_number = phonenumbers.format_number(phone_number, phonenumbers.PhoneNumberFormat.E164)
print(formatted_number)

20. isort - Import Sorting and Formatting

Isort takes care of sorting and formatting your import statements in Python code. It ensures consistency and readability in your codebase.

Example:

# Before running isort
from datetime import date
import os

# After running isort
import os
from datetime import date

21. emoji - Working with Emojis in Python

The emoji library simplifies working with emojis in Python, making it easy to insert emojis into strings or analyze emoji usage in text.

Example:

import emoji

text = "I love Python! :snake:"
emoji_text = emoji.emojize(text)
print(emoji_text)

22. Pydub - Audio Processing in Python

Pydub is a versatile library for audio processing in Python. It can be used for tasks like audio format conversion, slicing, and more.

Example:

from pydub import AudioSegment

sound = AudioSegment.from_file("my_audio.mp3")
# Convert to WAV format
sound.export("output.wav", format="wav")

23. textract - Text Extraction from Various File Formats

Textract is a handy library for extracting text content from a wide range of file formats, including PDFs, Word documents, and more.

Example:

import textract

text = textract.process("document.pdf")
print(text.decode("utf-8"))

24. PDFMiner - PDF Parsing and Text Extraction

PDFMiner is an underrated PDF parsing library that allows you to extract text and metadata from PDF files. It's highly customizable and useful for data extraction.

Example:

from pdfminer.high_level import extract_text

text = extract_text("document.pdf")
print(text)

25. Pytube - YouTube Data Manipulation

Pytube simplifies working with YouTube data and videos. It allows you to download, manipulate, or extract information from YouTube videos.

Example (Downloading a YouTube video):

from pytube import YouTube

url = "https://www.youtube.com/watch?v=example_video_id"
yt = YouTube(url)
stream = yt.streams.get_highest_resolution()
stream.download(output_path="downloads/")

Conclusion

By incorporating these lesser-known Python libraries into your toolkit, you'll not only simplify your daily work but also expand your capabilities as a Python programmer. So, embrace these hidden gems, explore their potential, and let them enhance your Python development journey.

Thank you for joining us on this exploration of unsung Python libraries, and we hope you find these tools useful in your coding endeavour.

That concludes our article on unsung Python libraries! We hope you've discovered valuable tools to enhance your daily coding tasks. If you have any more questions or need further guidance, feel free to ask. Happy coding!

Data preparation is a critical step in any Machine Learning Operations (MLOps) workflow. It involves collecting, cleaning, and transforming raw data into a format suitable for model training and deployment. With AWS's powerful suite of data management services, data scientists can streamline this process to ensure high-quality data inputs and efficient machine learning pipelines. In this article, we will explore the best practices and AWS tools to perform data preparation effectively for MLOps.

1. Data Collection and Storage

AWS offers Amazon S3, a reliable and scalable object storage service, to store vast amounts of raw and processed data securely. Use S3 buckets to organize and manage data for different machine learning projects. Ensure data versioning and encryption to maintain data integrity and security.

- Use Amazon S3 (Simple Storage Service) to store raw data securely and durably.

- Upload your datasets to S3 buckets, organizing them by project or dataset types.

1. Data Exploration and Cleaning

Before proceeding with model training, explore the data using AWS Glue or Amazon Athena. Identify missing values, outliers, and inconsistencies that might affect the model's performance. Cleanse the data by removing irrelevant features and handling missing values appropriately.

- You can use AWS Glue or Amazon Athena to query and analyze data directly from S3.

- AWS Glue can also help you discover the schema of your data and catalog it.

1. Data Transformation

AWS Glue ETL jobs can be leveraged to transform data into the required format for machine learning models. Use AWS Glue's DataBrew for visual data preparation tasks, simplifying data transformations even for non-technical users.

- Utilize AWS Glue or AWS DataBrew to clean and transform raw data into a suitable format for training.

1. Data Sampling and Splitting

To avoid overfitting and ensure unbiased model evaluation, split the dataset into training, validation, and testing sets using AWS Glue or SageMaker Processing Jobs.

- Divide the prepared data into training, validation, and testing sets.

- You can use AWS services like Amazon SageMaker or AWS DataBrew to perform data splitting and sampling.

1. Feature Engineering

AWS SageMaker provides a range of built-in algorithms and tools for feature engineering. Utilize these capabilities to create new features that can improve model performance.

1. Data Versioning and Tracking

AWS SageMaker Model Registry enables versioning of data and models. Ensure that data scientists and engineers can access and track data changes throughout the development and deployment lifecycle.

- Employ version control systems like AWS CodeCommit or GitHub to manage changes to your data preparation scripts and configurations.

- This enables you to track the evolution of data preprocessing steps and roll back if necessary.

1. Automated Data Pipelines

Build automated data pipelines using AWS Step Functions or Apache Airflow to orchestrate the data preparation workflow seamlessly. This ensures consistency and reduces manual errors.

- Construct data pipelines using AWS services like AWS Data Pipeline or AWS Step Functions to automate data ingestion and preprocessing.

- These pipelines can schedule and orchestrate the data preparation process, making it efficient and reproducible.

1. Data Security and Compliance

Implement AWS Identity and Access Management (IAM) roles and policies to control access to data. Ensure compliance with data protection regulations to safeguard sensitive information.

1. Monitoring and Logging

Set up monitoring using Amazon CloudWatch to track data processing and identify any issues in real-time. Log data preparation steps to facilitate debugging and performance optimization.

Conclusion

Efficient data preparation is the foundation of successful MLOps. AWS provides a comprehensive set of services that empower data scientists to handle data effectively for machine learning projects. By following best practices and leveraging AWS tools like Amazon S3, AWS Glue, and SageMaker, organizations can streamline data preparation, leading to more accurate and reliable machine learning models. As MLOps continues to evolve, leveraging AWS services will play a pivotal role in achieving seamless data management and driving innovation in the world of machine learning.

• Structured Programming paradigm - adopted by C,Pascal
• Object Oriented Programming (OOP) paradigm - adopted by C++,Java,C#,VB.NET

Before we move towards OOPs concepts Lets talk a bit about Structured Programming paradigm and it's disadvantages.

Structured Programming paradigm

1. Emphasis on breaking the given task into smaller sub-tasks
2. For each sub-task functions are written
3. These functions are called directly or indirectly from main()
4. No importance given to data, it is just passed from one function to another as required.

Structured Programming in Everyday Life

1. Sequence Execute a list of statements in order.

Example: Baking Bread Add flour. Add salt. Add yeast. Mix. Add water. Knead. Let rise. Bake.

2. Repetition Repeat a block of statements while a condition is true.

Example: Washing Dishes Stack dishes by sink. Fill sink with hot soapy water. While moreDishes Get dish from counter, Wash dish, Put dish in drain rack. End While Wipe off counter. Rinse out sink.

3. Selection Choose at most one action from several alternative conditions.

Example: Sorting Mail Get mail from mailbox. Put mail on table. While moreMailToSort Get piece of mail from table. If pieceIsPersonal Then Read it. ElseIf pieceIsMagazine Then Put in magazine rack. ElseIf pieceIsBill Then Pay it, ElseIf pieceIsJunkMail Then Throw in wastebasket. End If End While

Disadvantages of Structured programming

1. The primary components of structured programming - functions and data structures - didn't model the real world problems in a natural way.
2. Mechanisms to reuse existing code were limited.
3. Maintaining, debugging and upgrading large programs were a difficult task.

OOPs (Object Oriented Programing)

1. Emphasis on identifying objects in a given problem and then writing programs to facilitate interaction between objects
2. Objects contain data and functions that can access/manipulate the data
3. Equal importance to data as data and functions go together.
4. Object-oriented programming aims to implement real-world entities like inheritance, hiding, polymorphism etc. in programming

Characteristics of OOPs

1. Objects : In Structured Programming a problem is approached by dividing it into functions. Unlike this, in OOPs the problem is divided into objects. Thinking in terms of objects rather than functions make the designing of program easier. For example:

   1. Employees in a Payroll Processing System
   2. GUI elements like windows, menus ,icons, etc.
   3. Elements in games like guns, characters, vehicle, etc.

2. Classes : The class is one of the Basic concepts of OOPs which is a group of similar entities. t is only a logical component and not the physical entity. A class serves as a blueprint or a plan or a template. It specifies what data and what functions will be included in objects of that type. For example:

   1. if you had a class called “Expensive Cars” it could have objects like Mercedes, BMW, Toyota, etc.
   2. if you had an online shopping system it could have objects such as “shopping cart”, “customer”, and “product”
   3. if you had a house it could have objects like rooms, restrooms, kitchen etc

3. Inheritance : Inheritance is an important pillar of OOP(Object Oriented Programming). It is the mechanism in java by which one class is allow to inherit the features(fields and methods) of another class.

   Let us discuss some of frequent used important terminologies:

   • Super Class: The class whose features are inherited is known as superclass(or a base class or a parent class).
   • Sub Class: The class that inherits the other class is known as subclass(or a derived class, extended class, or child class). The subclass can add its own fields and methods in addition to the superclass fields and methods.
   • Reusability: Inheritance supports the concept of “reusability”, i.e. when we want to create a new class and there is already a class that includes some of the code that we want, we can derive our new class from the existing class. By doing this, we are reusing the fields and methods of the existing class.

4. Abstraction : Data Abstraction is the property by virtue of which only the essential details are displayed to the user. The trivial or the non-essentials units are not displayed to the user. Ex: A car is viewed as a car rather than its individual components. Data Abstraction may also be defined as the process of identifying only the required characteristics of an object ignoring the irrelevant details. The properties and behaviors of an object differentiate it from other objects of similar type and also help in classifying/grouping the objects. Consider a real-life example of a man driving a car. The man only knows that pressing the accelerators will increase the speed of car or applying brakes will stop the car but he does not know about how on pressing the accelerator the speed is actually increasing, he does not know about the inner mechanism of the car or the implementation of accelerator, brakes etc in the car. This is what abstraction is. In java, abstraction is achieved by interfaces and abstract classes. We can achieve 100% abstraction using interfaces.

5. Encapsulation : It is defined as the wrapping up of data under a single unit. It is the mechanism that binds together code and the data it manipulates. Another way to think about encapsulation is, it is a protective shield that prevents the data from being accessed by the code outside this shield.

   • Technically in encapsulation, the variables or data of a class is hidden from any other class and can be accessed only through any member function of own class in which they are declared.
   • As in encapsulation, the data in a class is hidden from other classes, so it is also known as data-hiding.
   • Encapsulation can be achieved by Declaring all the variables in the class as private and writing public methods in the class to set and get the values of variables.

6. Polymorphism

   It refers to the ability of OOPs programming languages to differentiate between entities with the same name efficiently. This is done by Java with the help of the signature and declaration of these entities.

   Note: Polymorphism in Java are mainly of 2 types:

   1. Overloading
   2. Overriding

As we know Python is being used widely across every domain. And I bet, every programmer had thought about building some kind of virtual assistance(let's call it VA) after watching "Iron Man Movie". In VA we add logic to do different tasks like opening some application, searching on the web, solving mathematical calculations, weather updates, Reminders, To-dos and this can go on and on. So, You might be thinking...

​ Come Straight to the Point

Okay, But Hindustani Bhau has something to say to you!!

​ Please be patient

We can add an Email Reader as well in your VA.To do so we need to first understand these few things mentioned below.

• Libraries to communicate with email services providers

• A purpose like downloading Bills, Tracking shopping orders, data analytics, reminders, follow-ups,auto-replies and much more dependence on the need

Let's get started with some technicalities and as an example will serve one purpose through python. Excited Right??

​ Now it's going to be fun

Libraries: imap_tools or imaplib

imap_tools

The easiest tool I've found for reading emails in Python is imap_tools. It has an elegant interface to communicate with your email provider using IMAP (which almost every email provider will have).

First, you access the MailBox; for which you need to get the IMAP server and login credentials (username and password). You should be able to find this in your email provider's help or settings (e.g. here's a guide for Gmail).

from imap_tools import MailBox, AND

# Server is the address of the IMAP server
mb = MailBox(server).login(user, password)

Then you can search for messages based on RFC 3501 Search Criteria. There are lots of examples in the imap_tools README; you can search based on the sender, subject, text, date, and others.

# Fetch all unseen emails containing "xyz.com" in the from field
# Don't mark them as seen
# Set bulk=True to read them all into memory in one fetch
# (as opposed to in streaming which is slower but uses less memory)
messages = mb.fetch(criteria=AND(seen=False, from_="xyz.com"),
                        mark_seen=False,
                        bulk=True)

Then you can access things like the subject, from address, date, and text and HTML content using simple attributes.

files = []
for msg in messages:
    # Print form and subject
    print(msg.from_, ': ', msg.subject)
    # Print the plain text (if there is one)
    print(msg.text)
    # Add attachments
    files += [att.payload for att in msg.attachments if att.filename.endswith('.pdf')]

It also handles actions on emails such as flagging as seen, moving, and deleting messages.

imaplib

Python has the built-in imaplib for IMAP and email for processing emails. Unfortunately, they're quite a low level and require a bit more work to use than imap_tools.

import imaplib
import email

mb = imaplib.IMAP4_SSL(server)
rv, mesasge = mb.login(user, password)
# 'OK', [b'LOGIN completed']
rv, num_emails = M.select('Inbox')
# 'OK', [b'22']

# Get unread messages
rv, messages = M.search(None, 'UNSEEN')
# 'OK', [b'21 22']

# Download a message
typ, data = M.fetch(b'21', '(RFC822)')

# Parse the email
msg = email.message_from_bytes(data[0][1])
print(msg['From'], ":", msg['Subject'])

# Print the Plain Text (is this always the plain text?)
print(msg.get_payload()[0].get_payload())

Once you go through these libraries you will get an idea about how to serve a specific purpose according to your requirement. Then you will be like...

​ I know everything, I'm an expert!!

Purpose: Let's make an Email Alert with a voice in python.

​ Then, Do It

In order to make a solution for this purpose we have to do certain things as listed below:

• Install required Libraries such as

  • GTTS (Google Text to Speech) : for text to speech conversion.

  • playsound : to play a audio files.

• Fetch the latest unread emails from the email provider.

• convert the text from email to Speech using GTTS and finally play it using playsound

Let's import all the required libraries we need in this script.

from imap_tools import MailBox,AND
import getpass
import json
from gtts import gTTS
import playsound

In the above, we have used the getpass module to get the “login name” of the user. json to read the mailconfig.json that holds the user credentials and server configs.

{
    "mail":
    {
        "ORG_EMAIL":"@example.com",
        "FROM_EMAIL":"abc",
        "FROM_PWD":"password",
        "SMTP_SERVER":"imap.example.com",
        "SMTP_PORT":"993"
    }
}

imap_tools for communication to the email service provider such as gmail,outlook etc.

Now Let's create two functions, read_email_from_email(username,configdata) that takes two parameters username(logged in user) and configdata(data from mailconfig.json) for communicating with the email provider and speak(text) that takes single parameter as String to convert it to speech.

def speak(text):
    tts = gTTS(text, lang='en') #gtts API to convert text to speech
    tts.save("output.mp3") #saving as the audio file
    playsound.playsound('output.mp3') #playing from audio file

def read_email_from_email(username,configdata):
    ORG_EMAIL   = configdata['mail']['ORG_EMAIL']
    FROM_EMAIL  = configdata['mail']['FROM_EMAIL'] + ORG_EMAIL
    FROM_PWD    = configdata['mail']['FROM_PWD']
    SMTP_SERVER = configdata['mail']['SMTP_SERVER']
    mail = MailBox(SMTP_SERVER).login(FROM_EMAIL, FROM_PWD)
    messages = mail.fetch(criteria=AND(seen=False),mark_seen=True,bulk=True)

    msg=list(messages)
    count=len(msg)
    if count>0:
        text=username+", You have an Email From "+msg[0].from_+",with a Subject Saying "+msg[0].subject
        print(text)
        speak(text)

    else:
        print("You Don't have any new emails!!")
        speak("You Don't have any new emails")

In read_email_from_email, we are connecting the Mail server using Mailbox with user credentials. With fetch(criteria=AND(seen=False),mark_seen=True,bulk=True) we are fetching all emails that are unseen in bulk and also marking them as seen. Then we are just creating a String using From Email and Subject. When we run this script it will read the latest email and also mark it as Read. I have just used the first fetched message. But you can play around with it and process those data according to your requirement.

I hope this article helped you to get familiar with how to read emails using python. and I wish your reaction would be like this.

​ I enjoyed It!!

Stay tuned for more exciting blogs related to How to-dos and Connect with me on my social handles to share the feedback or you can also comment your thoughts, I would love it.

Let's Create a Demo App in Express js

To create a new npm project, let's create a directory named "node-ssl-server" and open the node-ssl-server directory in the terminal using this command.

cd node-ssl-server

Then run this command to create a new npm project.

 npm init -y

Now let's install the dependency i.e express, to do so run this command:

npm install --save express

Now let's create a start script in package.json, just add this line inside the "script{}" as shown below:

"scripts": {
    "start":"node index.js"
  },

you can also use nodemon if you have nodemon installed in your system like this:

"scripts": {
    "start":"nodemon index.js"
  },

Now let's add a index.js file in our app and add few lines in it as shown below:

const express = require('express') 
const https = require("https") // https module to create a ssl enabled server
const path = require("path") // path module 
const fs = require("fs") //file system module

const app =express()

app.use("/",(req,res,next)=>{
    res.send("hello from ssl secured server!!")
})

const options ={
  key:'',
  cert:'' 
}
const sslserver =https.createServer(options,app)

sslserver.listen(port,()=>{console.log(`Secure Server is listening on port ${port}`)});

Let's Generate SSL Certificates

before we proceed further let's create a directory to store the certificates inside our app folder.

mkdir cert

now move to the cert directory using cd command

cd cert

To generate the SSL Certificate we need to follow these steps as shown below:

• Generate a Private Key
• Create a CSR ( certificate signing request) using the private key.
• Generate the SSL certification from CSR

Generate a Private Key

To generate a private key we will run this command as shown below:

 openssl genrsa -out key.pem

Once we ran the above command it will generate the private key and save it in key.pem file inside cert directory and gives this type of message in the terminal.

Generating RSA private key, 2048 bit long modulus
...+++
.................+++
e is 65537 (0x10001)

Create a CSR ( Certificate Signing Request)

Since we are our own certificate authority, we need to use CSR to generate our certificate. To do so we need to run the below command.

openssl req -new -key key.pem -out csr.pem

Once we ran this command it will ask a few questions as shown below:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields, there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:IN
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

you can skip any question by simply press enter else if you want to provide the details you can provide it, it's totally upto you.

Once you done with these question it will generate the CSR in csr.pem file inside cert folder.

Generate the SSL Certificate

Now for the final steps, we need to use the key.pem and crs.pem files to generate our SSL certificate.

let's run the below command to generate it.

openssl x509 -req -days 365 -in csr.pem -signkey key.pem -out cert.pem

Note:

• we are using x509 because it is the standard defining the format of the public-key certificate.
• we set the validity of the certificate as 365 days.

After running the above command it will save the certificate in the cert.pem file inside cert folder. Now you can remove the csr.pem file or you can keep it.

Integration of the SSL Certificate in Express

Now let's use these certificates inside our app using file system (fs) and path module. To do so, we need to edit a few lines in our app as mentioned below:

Earlier we had created a constant variable options. now we will update that part of the code by adding the path of the generated certificates inside it as shown below.

Before:

const options ={
  key:'',
  cert:'' 
}

After:

const options ={
  key:fs.readFileSync(path.join(__dirname,'./certs/key.pem')),
  cert:fs.readFileSync(path.join(__dirname,'./certs/cert.pem')) 
}

Once it's done save it and run the server by

npm start

You can check if HTTPS is working or not by just accessing it from this URL:

https://localhost:3002

Conclusion:

• You might see Not Secure in your browser though we have a valid certificate, it is just because we have generated the certificate and it is not generated by some known certificate authorities, so, your browser doesn't Trust you as a valid certificate authority. But we should typically use this process for development purposes and for Production we should be using a certificate that is generated by a certificate authority like Let's Encrypt.

let & const

Read more about let : https://developer.mozilla.org/en-US/ docs/Web/JavaScript/Reference/Statements/let Read more about const : https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/const

let and const basically replace var You use let instead of var and const instead of var if you plan on never re-assigning this "variable" (effectively turning it into a constant therefore).

ES6 Arrow Functions

Read more: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions

Arrow functions are a different way of creating functions in JavaScriptBesides a shorter syntax, they offer advantages when it comes to keeping the scope of the this keyword (see here).

Arrow function syntax may look strange but it's actually simple.

 function callMe(name) { 
    console.log(name);
 }

which you could also write as:

 const callMe = function(name) { 
    console.log(name);
 }

becomes:

 const callMe = (name) => { 
    console.log(name);
 }

Important:

When having no arguments, you have to use empty parentheses in the function declaration:

 const callMe = () => { 
    console.log('Max!');
 }

When having exactly one argument, you may omit the parentheses:

 const callMe = name => { 
    console.log(name);
 }

When just returning a value, you can use the following shortcut:

 const returnMe = name => name

That's equal to:

const returnMe = name => { 
    return name;
 }

Exports & Imports

In React projects (and actually in all modern JavaScript projects), you split your code across multiple JavaScript files - so-called modulesYou do this, to keep each file/ module focused and manageable.

To still access functionality in another file, you need export (to make it available) and import (to get access) statements.

You got two different types of exports: default (unnamed) and named exports:

default => export default ...;

named => export const someData = ...;

You can import default exports like this:

import someNameOfYourChoice from './path/to/file.js';

Surprisingly, someNameOfYourChoice is totally up to you. Named exports have to be imported by their name:

import { someData } from './path/to/file.js';

A file can only contain one default and an unlimited amount of named exportsYou can also mix the one default with any amount of named exports in one and the same file. When importing named exports, you can also import all named exports at once with the following syntax:

import * as upToYou from './path/to/file.js';

upToYou is - well - up to you and simply bundles all exported variables/functions in one JavaScript objectFor example, if you

export const someData = ..(/path/to/file.js )

you can access it on upToYou like this:

upToYou.someData

Classes

Classes are a feature which basically replace constructor functions and prototypesYou can define blueprints for JavaScript objects with them

Like this:

class Person {
    constructor () {
        this.name = 'Max';
    }
}

const person = new Person();
console.log(person.name); // prints 'Max'

In the above example, not only the class but also a property of that class (=> name ) is definedThey syntax you see there, is the "old" syntax for defining propertiesIn modern JavaScript projects (as the one used in this course), you can use the following, more convenient way of defining class properties:

class Person {
    name = 'Max';
}

const person = new Person();
console.log(person.name); // prints 'Max'

You can also define methodsEither like this:

class Person {
    name = 'Max';
    printMyName () {
    console.log(this.name); // this is required to refer to the class!
    }
}

const person = new Person();
person.printMyName();

Or like this:

class Person {
    name = 'Max';
    printMyName = () => {
    console.log(this.name);
    }
}

const person = new Person();
person.printMyName();

The second approach has the same advantage as all arrow functions have: The this keyword doesn't change its reference. You can also use inheritance when using classes:

class Human {
    species = 'human';
}
.
class Person extends Human {
    name = 'Max';
    printMyName = () => {
    console.log(this.name);
}
}
.
const person = new Person();
person.printMyName();
console.log(person.species); // prints 'human'

Spread & Rest Operator

The spread and rest operators actually use the same syntax:

..

Yes, that is the operator - just three dotsIt's usage determines whether you're using it as the spread or rest operator.

Using the Spread Operator:

The spread operator allows you to pull elements out of an array (=> split the array into a list of its elements) or pull the properties out of an objectHere are two examples:

const oldArray = [1, 2, 3];
const newArray = [...oldArray, 4, 5]; // This now is [1, 2, 3, 4, 5];

Here's the spread operator used on an object:

const oldObject = {
    name: 'Max'
};
const newObject = {
    ...oldObject,
    age: 28
};

newObject would then be

{
name: 'Max',
age: 28
}

The spread operator is extremely useful for cloning arrays and objectsSince both are reference types (and not primitives), copying them safely (i.epreventing future mutation of the copied original) can be trickyWith the spread operator you have an easy way of creating a (shallow!) clone of the object or array

Destructuring

Destructuring allows you to easily access the values of arrays or objects and assign them to variables. Here's an example for an array:

const array = [1, 2, 3];
const [a, b] = array;
console.log(a); // prints 1
console.log(b); // prints 2
console.log(array); // prints [1, 2, 3]

And here for an object:

const myObj = {
    name: 'Max',
    age: 28
}
const {name} = myObj;
console.log(name); // prints 'Max'
console.log(age); // prints undefined
console.log(myObj); // prints {name: 'Max', age: 28}

Destructuring is very useful when working with function argumentsConsider this example:

const printName = (personObj) => {
console.log(personObj.name);
}
printName({name: 'Max', age: 28}); // prints 'Max'

Here, we only want to print the name in the function but we pass a complete person object to the functionOf course this is no issue but it forces us to call personObj.name inside of our functionWe can condense this code with destructuring:

const printName = ({name}) => {
console.log(name);
}
printName({name: 'Max', age: 28}); // prints 'Max')

We get the same result as above but we save some code By destructuring, we simply pull out the name property and store it in a variable/ argument named name which we then can use in the function body.

JS Array Functions

Not really next-gen JavaScript, but also important: JavaScript array functions like map() , filter() , reduce() etc.

You'll see me use them quite a bit since a lot of React concepts rely on working with arrays (in immutable ways).

The following page gives a good overview over the various methods you can use on the array prototype - feel free to click through them and refresh your knowledge as required: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array

Particularly important in this course are:

map() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/map

find() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/find

findIndex() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/findIndex

filter() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter

reduce() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce?v=b

concat() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat?v=b

slice() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice

splice() => https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/splice

import re

s = 'GeeksforGeeks: A computer science portal for geeks'

match = re.search(r'portal', s)

print('Start Index:', match.start())
print('End Index:', match.end())

Start Index: 34
End Index: 40

MetaCharacters

To understand the RE analogy, MetaCharacters are useful, important, and will be used in functions of module re. Below is the list of metacharacters.

#examples

import re

s = 'geeks.forgeeks'

# without using \
match = re.search(r'.', s)
print(match)

# using \
match = re.search(r'\.', s)
print(match)

<re.Match object; span=(0, 1), match='g'>
<re.Match object; span=(5, 6), match='.'>

Special Sequences

Special sequences do not match for the actual character in the string instead it tells the specific location in the search string where the match must occur. It makes it easier to write commonly used patterns.

re.findall()

Return all non-overlapping matches of pattern in string, as a list of strings. The string is scanned left-to-right, and matches are returned in the order found.

# A Python program to demonstrate working of
# findall()
import re

# A sample text string where regular expression
# is searched.
string = """Hello my Number is 123456789 and
            my friend's number is 987654321"""

# A sample regular expression to find digits.
regex = '\d+'

match = re.findall(regex, string)
print(match)

# This example is contributed by Ayush Saluja.

['123456789', '987654321']

re.compile()

Regular expressions are compiled into pattern objects, which have methods for various operations such as searching for pattern matches or performing string substitutions.

# Module Regular Expression is imported
# using __import__().
import re

# compile() creates regular expression
# character class [a-e],
# which is equivalent to [abcde].
# class [abcde] will match with string with
# 'a', 'b', 'c', 'd', 'e'.
p = re.compile('[a-e]')

# findall() searches for the Regular Expression
# and return a list upon finding
print(p.findall("Aye, said Mr. Gibenson Stark"))

['e', 'a', 'd', 'b', 'e', 'a']

import re

# \w is equivalent to [a-zA-Z0-9_].
p = re.compile('\w')
print(p.findall("He said * in some_lang."))

# \w+ matches to group of alphanumeric character.
p = re.compile('\w+')
print(p.findall("I went to him at 11 A.M., he \
said *** in some_language."))

# \W matches to non alphanumeric characters.
p = re.compile('\W')
print(p.findall("he said *** in some_language."))

['H', 'e', 's', 'a', 'i', 'd', 'i', 'n', 's', 'o', 'm', 'e', '_', 'l', 'a', 'n', 'g']
['I', 'went', 'to', 'him', 'at', '11', 'A', 'M', 'he', 'said', 'in', 'some_language']
[' ', ' ', '*', '*', '*', ' ', ' ', '.']

re.split()

Split string by the occurrences of a character or a pattern, upon finding that pattern, the remaining characters from the string are returned as part of the resulting list.

from re import split

# '\W+' denotes Non-Alphanumeric Characters
# or group of characters Upon finding ','
# or whitespace ' ', the split(), splits the
# string from that point
print(split('\W+', 'Words, words , Words'))
print(split('\W+', "Word's words Words"))

# Here ':', ' ' ,',' are not AlphaNumeric thus,
# the point where splitting occurs
print(split('\W+', 'On 12th Jan 2016, at 11:02 AM'))

# '\d+' denotes Numeric Characters or group of
# characters Splitting occurs at '12', '2016',
# '11', '02' only
print(split('\d+', 'On 12th Jan 2016, at 11:02 AM'))

['Words', 'words', 'Words']
['Word', 's', 'words', 'Words']
['On', '12th', 'Jan', '2016', 'at', '11', '02', 'AM']
['On ', 'th Jan ', ', at ', ':', ' AM']

re.sub()

The ‘sub’ in the function stands for SubString, a certain regular expression pattern is searched in the given string(3rd parameter), and upon finding the substring pattern is replaced by repl(2nd parameter), count checks and maintains the number of times this occurs.

import re

# Regular Expression pattern 'ub' matches the
# string at "Subject" and "Uber". As the CASE
# has been ignored, using Flag, 'ub' should
# match twice with the string Upon matching,
# 'ub' is replaced by '~*' in "Subject", and
# in "Uber", 'Ub' is replaced.
print(re.sub('ub', '~*', 'Subject has Uber booked already',
            flags=re.IGNORECASE))

# Consider the Case Sensitivity, 'Ub' in
# "Uber", will not be reaplced.
print(re.sub('ub', '~*', 'Subject has Uber booked already'))

# As count has been given value 1, the maximum
# times replacement occurs is 1
print(re.sub('ub', '~*', 'Subject has Uber booked already',
            count=1, flags=re.IGNORECASE))

# 'r' before the pattern denotes RE, \s is for
# start and end of a String.
print(re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam',
            flags=re.IGNORECASE))

S~*ject has ~*er booked already
S~*ject has Uber booked already
S~*ject has Uber booked already
Baked Beans & Spam

re.search()

his method either returns None (if the pattern doesn’t match), or a re.MatchObject that contains information about the matching part of the string. This method stops after the first match, so this is best suited for testing a regular expression more than extracting data.

# A Python program to demonstrate working of re.match().
import re

# Lets use a regular expression to match a date string
# in the form of Month name followed by day number
regex = r"([a-zA-Z]+) (\d+)"

match = re.search(regex, "I was born on June 24")

if match != None:

    # We reach here when the expression "([a-zA-Z]+) (\d+)"
    # matches the date string.

    # This will print [14, 21), since it matches at index 14
    # and ends at 21.
    print ("Match at index %s, %s" % (match.start(), match.end()))

    # We us group() method to get all the matches and
    # captured groups. The groups contain the matched values.
    # In particular:
    # match.group(0) always returns the fully matched string
    # match.group(1) match.group(2), ... return the capture
    # groups in order from left to right in the input string
    # match.group() is equivalent to match.group(0)

    # So this will print "June 24"
    print ("Full match: %s" % (match.group(0)))

    # So this will print "June"
    print ("Month: %s" % (match.group(1)))

    # So this will print "24"
    print ("Day: %s" % (match.group(2)))

else:
    print ("The regex pattern does not match.")

Match at index 14, 21
Full match: June 24
Month: June
Day: 24

For more on Regular expressions

# welcome.py
def welcome_message(course):
    print("Thank you for subscribing to our " + course + " course. You will get all the details in an email shortly.")

import welcome
welcome.welcome_message ("Python Essentials")

Thank you for subscribing to our Python Essentials course. You will get all the details in an email shortly.

from welcome import welcome_message
welcome_message ("Python Essentials")

Thank you for subscribing to our Python Essentials course. You will get all the details in an email shortly.

If you have some experience with Python, you’ve likely used modules. For example, you may have used the:

• random module to generate pseudo-random number generators for various distributions.
• html module to parse HTML pages.
• datetime module to manipulate date and time data.
• re module to detect and parse regular expressions in Python.

Modules introduce numerous benefits into your Python code:

• Improved development process. Python modules help you focus on one small portion of a task rather than an entire problem. This simplifies the development process and makes it less prone to errors. Furthermore, modules are usually written in a way that minimizes interdependency. Thus, it’s more viable for a team of several programmers to work on the same application.
• The functionality you define in one module can be used in different parts of an application, minimizing duplicate code.Separate namespaces. With Python modules, you can define separate namespaces to avoid collisions between identifiers in different parts of your application.

Examples

import sys
sys.path

['C:\\Users\\rahuldubey\\Documents\\My Learnings\\Training\\Python Advance\\6. Modules & Packages',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\python37.zip',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\DLLs',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3',
 '',
 'C:\\Users\\rahuldubey\\AppData\\Roaming\\Python\\Python37\\site-packages',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\win32',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\win32\\lib',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\Pythonwin',
 'C:\\Users\\rahuldubey\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\IPython\\extensions',
 'C:\\Users\\rahuldubey\\.ipython']

import platform

x = platform.system()
print(x)

Windows

import datetime

x = datetime.datetime.now()
print(x)

2021-09-22 21:54:34.483628

import math

x = math.sqrt(64)

print(x)

8.0

Python Packages

When developing a large application, you may end up with many different modules that are difficult to manage. In such a case, you’ll benefit from grouping and organizing your modules. That’s when packages come into play.

Python packages are basically a directory of a collection of modules. Packages allow the hierarchical structure of the module namespace. Just like we organize our files on a hard drive into folders and sub-folders, we can organize our modules into packages and subpackages.

To be considered a package (or subpackage), a directory must contain a file named init.py. This file usually includes the initialization code for the corresponding package.

Requests : The requests package is an HTTP library for Python. It is built on top of urllib3 (another HTTP client for Python), but has a much simpler and more elegant syntax.

import requests
r = requests.get('https://api.spotify.com/')
r.status_code

200

NumPy : NumPy is the essential package for scientific and mathematical computing in Python. It introduces n-dimensional arrays and matrices, which are necessary when performing sophisticated mathematical operations. It contains functions that perform basic operations on arrays, such as sorting, shaping, and other mathematical matrix operations.

For example, to create two 2×2 complex matrices and print the sum:

import numpy as np

a = np.array([[1+2j, 2+1j], [3, 4]])
b = np.array([[5, 6+6j], [7, 8+4j]])
print(a+b)

[[ 6.+2.j  8.+7.j]
 [10.+0.j 12.+4.j]]

Pandas : The pandas package introduces a novel data structure, the DataFrame, optimized for tabular, multidimensional, and heterogeneous data. Once your data has been converted to this format, the package provides intuitive and practical means to clean and manipulate it.

Manipulations such as groupby, join, merge, concatenate data or filling, replacing and imputing null values can be executed in a single line. The developers of the package have the primary goal of producing the world’s most powerful data analysis and manipulation tool that exists in any language — a daunting task that they may actually achieve.

import pandas as pd

df_1 = pd.DataFrame({'col1': [1,2], 'col2': [3,4]})

df_2 = pd.DataFrame({'col3': [5,6], 'col4': [7,8]})
df = pd.concat([df_1,df_2], axis = 1)
df

Modules Vs Packages

NumPy - Introduction

NumPy is a Python package. It stands for 'Numerical Python'. It is a library consisting of multidimensional array objects and a collection of routines for processing of array.

Numeric, the ancestor of NumPy, was developed by Jim Hugunin. Another package Numarray was also developed, having some additional functionalities. In 2005, Travis Oliphant created NumPy package by incorporating the features of Numarray into Numeric package. There are many contributors to this open source project.

Operations using NumPy

Using NumPy, a developer can perform the following operations −

• Mathematical and logical operations on arrays.

• Fourier transforms and routines for shape manipulation.

• Operations related to linear algebra. NumPy has in-built functions for linear algebra and random number generation.

NumPy - Environment

Standard Python distribution doesn't come bundled with NumPy module. A lightweight alternative is to install NumPy using popular Python package installer, pip.

pip install numpy

NumPy - Ndarray Object

The most important object defined in NumPy is an N-dimensional array type called ndarray. It describes the collection of items of the same type. Items in the collection can be accessed using a zero-based index.

import numpy as np 
a = np.array([1,2,3]) 
print (a)

[1 2 3]

import numpy as np 
a = np.array([[1, 2], [3, 4]]) 
print( a)

[[1 2]
 [3 4]]

# dtype parameter 
import numpy as np 
a = np.array([1, 2, 3], dtype = complex) 
print( a)

[1.+0.j 2.+0.j 3.+0.j]

NumPy - Array Attributes

ndarray.shape This array attribute returns a tuple consisting of array dimensions. It can also be used to resize the array.

import numpy as np 
a = np.array([[1,2,3],[4,5,6]]) 
print (a.shape)

(2, 3)

ndarray.ndim This array attribute returns the number of array dimensions.

# this is one dimensional array 
import numpy as np 
a = np.arange(24) 
print(a.ndim) 

# now reshape it 
b = a.reshape(2,4,3) 
print (b )
# b is having three dimensions

1
[[[ 0  1  2]
  [ 3  4  5]
  [ 6  7  8]
  [ 9 10 11]]

 [[12 13 14]
  [15 16 17]
  [18 19 20]
  [21 22 23]]]

numpy.itemsize This array attribute returns the length of each element of array in bytes.

# dtype of array is int8 (1 byte) 
import numpy as np 
x = np.array([1,2,3,4,5], dtype = np.int8) 
print (x.itemsize)

1

# dtype of array is now float32 (4 bytes) 
import numpy as np 
x = np.array([1,2,3,4,5], dtype = np.float32) 
print (x.itemsize)

4

numpy.zeros Returns a new array of specified size, filled with zeros.

# array of five zeros. Default dtype is float 
import numpy as np 
x = np.zeros(5) 
print( x)

[0. 0. 0. 0. 0.]

numpy.ones Returns a new array of specified size and type, filled with ones.

# array of five ones. Default dtype is float 
import numpy as np 
x = np.ones(5) 
print (x)

[1. 1. 1. 1. 1.]

numpy.asarray This function is similar to numpy.array except for the fact that it has fewer parameters. This routine is useful for converting Python sequence into ndarray.

# convert list to ndarray 
import numpy as np 

x = [1,2,3] 
a = np.asarray(x) 
print( a)

[1 2 3]

numpy.linspace This function is similar to arange() function. In this function, instead of step size, the number of evenly spaced values between the interval is specified.

import numpy as np 
x = np.linspace(10,20,5) 
print (x)

[10.  12.5 15.  17.5 20. ]

For more on numpy

Pandas - Introduction

Pandas is an open-source Python Library providing high-performance data manipulation and analysis tool using its powerful data structures. The name Pandas is derived from the word Panel Data – an Econometrics from Multidimensional data.

Standard Python distribution doesn't come bundled with Pandas module. A lightweight alternative is to install NumPy using popular Python package installer, pip.

pip install pandas

Pandas deals with the following three data structures −

• Series
• DataFrame
• Panel

These data structures are built on top of Numpy array, which means they are fast.

Series

Series is a one-dimensional array-like structure with homogeneous data. For example, the following series is a collection of integers 10, 23, 56, …

Key Points

• Homogeneous data
• Size Immutable
• Values of Data Mutable

DataFrame

DataFrame is a two-dimensional array with heterogeneous data. For example,

Key Points

• Heterogeneous data
• Size Mutable
• Data Mutable

Panel

Panel is a three-dimensional data structure with heterogeneous data. It is hard to represent the panel in graphical representation. But a panel can be illustrated as a container of DataFrame.

Key Points

• Heterogeneous data
• Size Mutable
• Data Mutable

#series example

#import the pandas library and aliasing as pd
import pandas as pd
import numpy as np
data = np.array(['a','b','c','d'])
s = pd.Series(data)
print (s)

0    a
1    b
2    c
3    d
dtype: object

#import the pandas library and aliasing as pd
import pandas as pd
import numpy as np
data = np.array(['a','b','c','d'])
s = pd.Series(data,index=[100,101,102,103])
print( s)

100    a
101    b
102    c
103    d
dtype: object

#import the pandas library and aliasing as pd
import pandas as pd
import numpy as np
data = {'a' : 0., 'b' : 1., 'c' : 2.}
s = pd.Series(data,index=['b','c','d','a'])
print (s)

b    1.0
c    2.0
d    NaN
a    0.0
dtype: float64

#dataframe examples
import pandas as pd
data = [1,2,3,4,5]
df = pd.DataFrame(data)
print (df)

   0
0  1
1  2
2  3
3  4
4  5

import pandas as pd
data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pd.DataFrame(data,columns=['Name','Age'])
print (df)

     Name  Age
0    Alex   10
1     Bob   12
2  Clarke   13

import pandas as pd
data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pd.DataFrame(data,columns=['Name','Age'],dtype=float)
print (df)

     Name   Age
0    Alex  10.0
1     Bob  12.0
2  Clarke  13.0

import pandas as pd
data = [{'a': 1, 'b': 2},{'a': 5, 'b': 10, 'c': 20}]
df = pd.DataFrame(data)
print (df)

   a   b     c
0  1   2   NaN
1  5  10  20.0

For more on pandas

Python abs()

The abs() function returns the absolute value of the given number. If the number is a complex number, abs() returns its magnitude.

number = -20

absolute_number = abs(number)
print(absolute_number)

# Output: 20

20

Python all()

The all() function returns True if all elements in the given iterable are true. If not, it returns False.

boolean_list = ['True', 'True', 'True']

# check if all elements are true
result = all(boolean_list)
print(result)

# Output: True

True

Python any()

The any() function returns True if any element of an iterable is True. If not, it returns False.

boolean_list = ['True', 'False', 'True']

# check if any element is true
result = any(boolean_list)
print(result)

# Output: True

True

Python ascii()

The ascii() method returns a string containing a printable representation of an object. It escapes the non-ASCII characters in the string using \x, \u or \U escapes.

otherText = 'Pythön is interesting'
print(ascii(otherText))

'Pyth\xf6n is interesting'

Python bin()

The bin() method converts and returns the binary equivalent string of a given integer. If the parameter isn't an integer, it has to implement index() method to return an integer.

number = 5
print('The binary equivalent of 5 is:', bin(number))

The binary equivalent of 5 is: 0b101

Python enumerate()

The enumerate() method adds a counter to an iterable and returns it (the enumerate object).

languages = ['Python', 'Java', 'JavaScript']

enumerate_prime = enumerate(languages)

# convert enumerate object to list
print(list(enumerate_prime))

# Output: [(0, 'Python'), (1, 'Java'), (2, 'JavaScript')]

[(0, 'Python'), (1, 'Java'), (2, 'JavaScript')]

Python filter()

The filter() function extracts elements from an iterable (list, tuple etc.) for which a function returns True.

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# returns True if number is even
def check_even(number):
    if number % 2 == 0:
          return True  

    return False

# Extract elements from the numbers list for which check_even() returns True
even_numbers_iterator = filter(check_even, numbers)

# converting to list
even_numbers = list(even_numbers_iterator)

print(even_numbers)

# Output: [2, 4, 6, 8, 10]

[2, 4, 6, 8, 10]

Python map()

The map() function applies a given function to each item of an iterable (list, tuple etc.) and returns an iterator.

numbers = [2, 4, 6, 8, 10]

# returns square of a number
def square(number):
    return number * number

# apply square() function to each item of the numbers list
squared_numbers_iterator = map(square, numbers)

# converting to list
squared_numbers = list(squared_numbers_iterator)
print(squared_numbers)

# Output: [4, 16, 36, 64, 100]

[4, 16, 36, 64, 100]

Python pow()

The pow() function returns the power of a number.

# positive x, positive y (x**y)
print(pow(2, 2))    # 4

# negative x, positive y
print(pow(-2, 2))    # 4  

# positive x, negative y
print(pow(2, -2))    # 0.25

# negative x, negative y
print(pow(-2, -2))    # 0.25

4
4
0.25
0.25

Python reversed()

The reversed() function returns the reversed iterator of the given sequence.

# for string
seq_string = 'Python'
print(list(reversed(seq_string)))

# for tuple
seq_tuple = ('P', 'y', 't', 'h', 'o', 'n')
print(list(reversed(seq_tuple)))

# for range
seq_range = range(5, 9)
print(list(reversed(seq_range)))

# for list
seq_list = [1, 2, 4, 3, 5]
print(list(reversed(seq_list)))

['n', 'o', 'h', 't', 'y', 'P']
['n', 'o', 'h', 't', 'y', 'P']
[8, 7, 6, 5]
[5, 3, 4, 2, 1]

Python round()

The round() function returns a floating-point number rounded to the specified number of decimals.

number = 13.46

# round the number
rounded_number = round(number)
print(rounded_number)

# Output: 13

13

Python sorted()

The sorted() function sorts the elements of a given iterable in a specific order (ascending or descending) and returns it as a list.

numbers = [4, 2, 12, 8]

sorted_numbers = sorted(numbers)
print(sorted_numbers)

# Output: [2, 4, 8, 12]

[2, 4, 8, 12]

Python zip()

The zip() function takes iterables (can be zero or more), aggregates them in a tuple, and returns it.

languages = ['Java', 'Python', 'JavaScript']
versions = [14, 3, 6]

result = zip(languages, versions)
print(list(result))

[('Java', 14), ('Python', 3), ('JavaScript', 6)]

Note: Find more on Built-in Function

Working of open() function

We use open () function in Python to open a file in read or write mode. As explained above, open ( ) will return a file object. To return a file object we use open() function along with two arguments, that accepts file name and the mode, whether to read or write. So, the syntax being: open(filename, mode). There are three kinds of mode, that Python provides and how files can be opened:

• “ r “, for reading.
• “ w “, for writing.
• “ a “, for appending.
• “ r+ “, for both reading and writing

One must keep in mind that the mode argument is not mandatory. If not passed, then Python will assume it to be “ r ” by default. Let’s look at this program and try to analyze how the read mode works:

# a file named "geek", will be opened with the reading mode.
file = open('test.txt', 'r')
# This will print every line one by one in the file
for each in file:
    print (each)

 this is test file for understanding 

the concept of file handling

Working of read() mode

There is more than one way to read a file in Python. If you need to extract a string that contains all characters in the file then we can use file.read(). The full code would work like this:

# Python code to illustrate read() mode
file = open("test.txt", "r")
print (file.read())

 this is test file for understanding 
the concept of file handling

# Python code to illustrate read() mode character wise
file = open("test.txt", "r")
print (file.read(5))

 this

Creating a file using write() mode

Let’s see how to create a file and how write mode works: To manipulate the file, write the following in your Python environment:

# Python code to create a file
file = open('geek.txt','w')
file.write("This is the write command")
file.write("It allows us to write in a particular file")
file.close()

Working of append() mode

Let’s see how the append mode works:

# Python code to illustrate append() mode
file = open('geek.txt','a')
file.write("This will add this line")
file.close()

Using write along with with() function

We can also use write function along with with() function:

# Python code to illustrate with() alongwith write()
with open("file.txt", "w") as f: 
    f.write("Hello World!!!")

split() using file handling

We can also split lines using file handling in Python. This splits the variable when space is encountered. You can also split using any characters as we wish. Here is the code:

# Python code to illustrate split() function
with open("file.txt", "r") as file:
    data = file.readlines()
    for line in data:
        word = line.split()
        print (word)

['Hello', 'World!!!']

Working With Binary

f1=open("python.jpg", "rb")
f2=open("newpic.jpg", "wb")
bytes=f1.read()
f2.write(bytes)
print("New Image is available with the name: newpic.jpg")

New Image is available with the name: newpic.jpg

Python Read and Write File (JSON, XML, CSV, xlsx) – Sample Code

JSON Files – Read and Write from Python :

• Write JSON Objects into Python :

# Converting the JSON objects to print in Python
import json
# some JSON:
x =  '{ "city":"Washington", "population":300000, "country":"US"}'
# parse x:
y = json.loads(x)
# the result is a Python dictionary:
print(y["city"])

Washington

• Converting Python Objects to JSON :

import json
# Python Dictionary object:
x = {
  "city":"Washington", 
  "population":300000, 
  "country":"US"
}

# convert into JSON:
y = json.dumps(x)

# the result is a JSON string:
print(y)

{"city": "Washington", "population": 300000, "country": "US"}

• Read JSON File :

import json
import pandas as pd

with open('file.json') as f:
    data= json.loads(f.read())
print(data)

# We can do the same thing with pandas
data_df = pd.read_json('file.json', orient='records')
print(data_df)

# To Pretty Print
print(json.dumps(data, indent = 4, sort_keys=True))

{'employee': {'name': 'sonoo', 'salary': 56000, 'married': True}}
        employee
married     True
name       sonoo
salary     56000
{
    "employee": {
        "married": true,
        "name": "sonoo",
        "salary": 56000
    }
}

• Write JSON Data to a File :

import json
import pandas as pd

data_dict = {
  "city":"Washington", 
  "population":300000, 
  "country":"US"
}

with open('file.txt', 'w') as json_file:
    json.dump(data_dict, json_file)

# We can do the same thing with pandas
df= pd.DataFrame.from_dict(data_dict, orient='index')
df.to_json('file.json', orient='records')

XML Files – Read and Write from Python :

<nodedetails>
  <name>SERVER-1</name>
  <description/>
  <numExecutors>SOME_NO_XX</numExecutors>
  <mode>SOME_MODE_XX</mode>
  <launcher class="xxxxx" plugin="xxxxx">
    <hostname>128.0.0.1</hostname>
    <portno>9900</portno>
    <credentialsId>TESTID</credentialsId>
    <maxNumRetries>3</maxNumRetries>
  </launcher>
  <label>somelabel</label>
</nodedetails>

• Read from the XML File using ElementTree :

import xml.etree.ElementTree as ET
tree = ET.parse("TEST.xml")
root = tree.getroot()

for item in root.iter('nodedetails'):
    for name in item.iter("name"):
        print (name.text)
    for portno in item.iter("portno"):
        print (portno.text)

SERVER-1
9900

• Read from any XML File using Beautifulsoup :

from bs4 import BeautifulSoup

with open('TEST.xml', 'r') as f: 
    data = f.read()

# Beautifulsoup will parse the data 
All_data = BeautifulSoup(data, "xml")

# Finding all instances of any tag  
tag_data = All_data.find_all('hostname')
print(tag_data)

[<hostname>128.0.0.1</hostname>]

• Writing an XML File using ElementTree :

import xml.etree.ElementTree as ET

# Parent (root) tag 
data1 = ET.Element('parents')

# Adding a subtag named `children`  inside our root tag "parents"
data2 = ET.SubElement(data1, 'children')

# Adding subtags under the `children` subtag 
grandChild1 = ET.SubElement(data2, 'E4') 
grandChild2 = ET.SubElement(data2, 'D4')

# Adding attributes to the tags under 
# `items` 
grandChild1.set('card', 'Credit_Card') 
grandChild2.set('card', 'Debit_Card')

# Adding text to grandchildren subtags
grandChild1.text = "Only Credit Cards Accepted"
grandChild2.text = "Only Debit Cards Accepted"

# Convert xml data to byte object to write into filestream 
data_xml = ET.tostring(data1)

# Write to a file with operation mode `wb` (write + binary) 
with open("OUTPUT.xml", "wb") as f: 
    f.write(data_xml)

from bs4 import BeautifulSoup

with open('OUTPUT.xml', 'r') as f: 
    data = f.read()

# Beautifulsoup will parse the data 
All_data = BeautifulSoup(data, "xml")
print(All_data)

<?xml version="1.0" encoding="utf-8"?>
<parents><children><E4 card="Credit_Card">Only Credit Cards Accepted</E4><D4 card="Debit_Card">Only Debit Cards Accepted</D4></children></parents>

• Writing to an XML using Beautifulsoup :

#Open file
soup = BeautifulSoup(open('TEST.xml'),'xml')

modified_xml_content = '''<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>'''

#Write to a file
f = open('OUTPUT.xml', "w")
f.write(str(modified_xml_content))
f.close()

CSV Files – Read and Write from Python :

• Read from CSV File using Native Python Lib :

import csv
rows=[]
with open('sample4.csv') as inputfile:
    csvreader = csv.reader(inputfile, delimiter=',')
    # Extracting each data row one by one 
    for row in csvreader: 
        rows.append(row)

# Printing out the first 5 rows 
for row in rows[:5]: 
    print(row)

['Game Number', ' "Game Length"']
['1', ' 30']
['2', ' 29']
['3', ' 31']
['4', ' 16']

• Read from CSV File using Pandas Lib :

import pandas as pd
data = pd.read_csv('sample4.csv')
# Display first 10 Lines of data
data.head(10)

• Write to a CSV File using Native Python lib :

import csv

data = [ ['Emily', '12', '18', '112'], 
         ['Katie', '8', '24', '96'], 
         ['John', '16', '9', '101'], 
         ['Mike', '3', '14', '82']]

writer = csv.writer(open("OUTPUTFILE.csv", 'w'))
for row in data:
    print(row)
    writer.writerow(row)

['Emily', '12', '18', '112']
['Katie', '8', '24', '96']
['John', '16', '9', '101']
['Mike', '3', '14', '82']

• Write to a CSV File using pandas :

import pandas as pd

data_dict = {
  "city":"Washington", 
  "population":300000, 
  "country":"US"
}

df= pd.DataFrame.from_dict(data_dict, orient='index')
df.to_csv('OUTPUTFILE.csv')

xlsx Files – Read and Write from Python :

Write to a xlsx File using pandas :

import pandas as pd

data_dict = {
  "city":"Washington", 
  "population":300000, 
  "country":"US"
}

df= pd.DataFrame.from_dict(data_dict, orient='index')
df.to_excel('OUTPUTFILE.xlsx')

• Read from xlsx File using Pandas Lib :

import pandas as pd

df=pd.read_excel('OUTPUTFILE.xlsx')
print(df)

   Unnamed: 0           0
0        city  Washington
1  population      300000
2     country          US