Categories
Data Science

Reinforcement Learning Algorithms

RL Algorithms Flowchart
  • RL Algorithms: Root
  • Model-Free RL: No model.
    • Policy Optimization: Optimize strategy.
    • Policy Gradient, A2C/A3C, PPO, TRPO, DDPG, TD3, SAC
    • Q-Learning: Learn action value.
    • DDPG, TD3, SAC, DQN, C51, QR-DQN, HER
  • Model-Based RL: Uses model.
    • Learn the Model: Learn from experience.
    • World Models, I2A, MBMF, MBVE
    • Given the Model: Known model.
    • AlphaZero
graph TB
  RL["RL Algorithms"]
  MF["Model-Free RL"]
  MB["Model-Based RL"]
  PO["Policy Optimization"]
  QL["Q-Learning"]
  LM["Learn the Model"]
  GM["Given the Model"]
  RL --> MF
  RL --> MB
  MF --> PO
  MF --> QL
  MB --> LM
  MB --> GM
  PO -->|Policy Gradient| PG
  PO -->|A2C/A3C| A2C
  PO -->|PPO| PPO
  PO -->|TRPO| TRPO
  PO -->|DDPG| DDPG1
  PO -->|TD3| TD31
  PO -->|SAC| SAC1
  QL -->|DDPG| DDPG2
  QL -->|TD3| TD32
  QL -->|SAC| SAC2
  QL -->|DQN| DQN
  QL -->|C51| C51
  QL -->|QR-DQN| QR
  QL -->|HER| HER
  LM -->|World Models| WM
  LM -->|I2A| I2A
  LM -->|MBMF| MBMF
  LM -->|MBVE| MBVE
  GM -->|AlphaZero| AZ

Detailed

RL Algorithms (Reinforcement Learning):

  • Algorithms designed to learn optimal actions by interacting with an environment.

Model-Free RL:

  • Algorithms that don’t rely on a model of the environment.Policy Optimization:
    • Directly optimize the strategy of actions.
      • Policy Gradient: Update policies using gradient ascent.
        • A2C/A3C: Advantage Actor-Critic methods.
        • PPO: Proximal Policy Optimization. Ensures stable policy updates.
        • TRPO: Trust Region Policy Optimization. Constrained policy updates.
      • DDPG: Deep Deterministic Policy Gradient. Uses deep networks for continuous actions.
      • TD3: Twin Delayed DDPG. Enhances DDPG stability.
      • SAC: Soft Actor-Critic. Mixes policy optimization with entropy-based exploration.
    Q-Learning:
    • Learn the value of actions.
      • DQN: Deep Q-Network. Uses neural networks to approximate the Q-function.
      • C51: Distributional DQN. Predicts return distributions.
      • QR-DQN: Quantile Regression DQN. A distributional variant.
      • HER: Hindsight Experience Replay. Makes use of unsuccessful experiences.

Model-Based RL:

  • Algorithms that utilize a model of the environment.Learn the Model:
    • Learn the environment model from experience.
      • World Models: Neural networks to simulate the environment’s dynamics.
      • I2A: Imagination-Augmented Agents. Uses the learned model to plan.
      • MBMF: Combines both Model-Based and Model-Free approaches.
      • MBVE: Model-Based Value Expansion.
    Given the Model:
    • Algorithms using a known environment model.
      • AlphaZero: Combines Monte Carlo Tree Search (MCTS) with deep learning.
Categories
AI

Langchain Vertex AI

  • Python package: google-cloud-aiplatform
  • Environment setup:
    • Credentials (gcloud, workload identity)
    • Or GOOGLE_APPLICATION_CREDENTIALS environment variable

Install Command:

pip install langchain google-cloud-aiplatform

Documentation:

# Code 1
from langchain.llms import VertexAI
llm = VertexAI()
print(llm("What are some of the pros and cons of Python as a programming language?"))

# Code 2
from langchain.prompts import PromptTemplate
template = "Question: {question}\nAnswer: Let's think step by step."
prompt = PromptTemplate.from_template(template)
chain = prompt | llm
question = "Who was the president in the year Justin Beiber was born?"
print(chain.invoke({"question": question}))

# Code 3
llm = VertexAI(model_name="code-bison", max_output_tokens=1000, temperature=0.3)
question = "Write a python function that checks if a string is a valid email address"
print(llm(question))

# Code 4
result = llm.generate([question])
result.generations

# Code 5
import asyncio
asyncio.run(llm.agenerate([question]))

# Code 6
import sys
for chunk in llm.stream(question):
    sys.stdout.write(chunk)
    sys.stdout.flush()
Categories
AI

GeniA

FunctionQuestionResponse
list_namespaceslist all the namespaceListed namespaces: default, kube-system
list_namespaced_secretlist all the namespaced secretNo secrets in the “default” namespace
list_deploymentShould I go ahead and list the kubernetes deployments for a given namespace?Listed 13 deployments in the “default” namespace
describe_deploymentexplain me more about seo deployGeneral explanation of Kubernetes deployment provided
list_namespaced_eventslist default namespaced eventsListed events in the “default” namespace with count, timestamps, message, and reason
scale_deployment_resourcesscale deployment resource for seo. set it to 64mb memory, 200m cpuError, required additional parameters. Later, successfully updated resources for “seo” deployment
list_namespaced_pod_eventslist namespace default pod events . seo deployListed events for pods in “seo” deployment about missing CPU requests
get_service_ownerget a service owner of any service in the default namespaceError, required service name. Later, no information available about the owner of “seo” service
get_top_k_containers_usageget top 5 containers usage in eks in region [REGION]Listed top 5 containers usage in [REGION] region with container name, image count, and size in MB
get_pods_errors_events_by_deploymentget pods errors events by seo deploymentListed events for pods in “seo” deployment about missing CPU requests
kubernetes_get_service_errorskubernetes get service errors seoListed some events in the “default” namespace, not specifically for “seo” service
Categories
AI

Chat GPT API Node.js

npm install openai

ES6

import OpenAI from "openai";

const openai = new OpenAI();

const chatCompletion = await openai.chat.completions.create({
    messages: [{ role: "user", content: "Say this is a test" }],
    model: "gpt-3.5-turbo",
});

console.log(chatCompletion.choices[0].message.content);

package.json

{
  "dependencies": {
    "openai": "^4.12.4"
  },
  "type": "module"
}

CommonJS

const OpenAI = require("openai");

const openai = new OpenAI();

openai.chat.completions.create({
  messages: [{ role: "user", content: "Say this is a test" }],
  model: "gpt-3.5-turbo",
})
.then(chatCompletion => {
  console.log(chatCompletion.choices[0].message.content);
})
.catch(console.error);
FeatureCommonJSES6 Modules
Used InNode.js, BrowserifyModern browsers, Node.js with config
Import/Export Syntaxconst toy = require('toy');
module.exports = toy;
import toy from 'toy';
export default toy;
Pros1. Easy to use
2. Dynamic loading
3. Well-supported in Node.js
1. Faster loading
2. Static analysis
3. Modern syntax
Cons1. Slower loading
2. Older syntax
1. More complex syntax
2. Need configuration

Which is Best?

  • For New Projects: ES6 is modern and efficient.
  • For Older Projects: CommonJS is well-supported and easy.

Combined Recommendation: Choose CommonJS for simplicity and legacy support. Choose ES6 for modern features and better optimization.

Categories
Other

Capitalising

  1. Skill Upgrading:
    • Learn machine learning frameworks
    • Acquire data engineering skills
  2. Market Research:
    • Identify AI gaps in current market
    • Validate problem-solution fit
  3. Networking:
    • Attend AI-focused meetups
    • Partner with data scientists
  4. Prototype:
    • Develop MVP using AI
    • User feedback loop
  5. Funding:
    • Create pitch deck
    • Approach VCs specialized in AI
  6. Launch:
    • Go-to-market strategy
    • Measure KPIs
  7. Scale:
    • Optimize algorithms
    • Expand user base
  8. Exit Strategy:
    • Identify acquisition targets
    • Plan IPO
  9. Continuous Learning:
    • Stay updated with AI trends
    • Iterate business model
  10. Intellectual Property:
    • File patents
    • License algorithms
Categories
Other

Becoming Rich

Method to Become RichSimple Explanation
Follow a straightforward formulaRohrssen believes building wealth is straightforward and formulaic.
Mindset and OptionsMaintain a mindset of abundance even when financially constrained.
Surround Yourself with Right PeopleKeep company with people who uplift you and share your vision.
Learn from FailuresEntrepreneurial ventures aren’t risky; you learn something even if you fail.
AdaptabilityBe adaptable to market changes and consumer desires.
Investment MeetingsWin or lose investment in the first two minutes; maintain frame control.
Genuine Frame in SalesBe genuine in your approach to sales and jobs for better success.
Balance IdentityHave something other than your business to tie your identity to.
Review InfluencesRegularly review your circle and influences to ensure they’re uplifting.
Utilize PressureUse the pressure from investors or circumstances to drive business growth.
Move Towards ProfitShift business towards profit-based rather than growth at all cost.
Categories
Other

Secrets of Viral YouTube Shorts

StrategyDetails
Analyze Popular ShortsStudies shorts from popular creators like Mr. Beast to understand what makes them viral.
ReadabilityAims for a readability level of fifth grade or under for wider audience reach.
PersonalizationMakes content personality-based to engage viewers.
Retention OptimizationUses analytics to aim for a 90% retention rate; trims video ends to improve retention.
StorytellingUses hooks and narratives to make content engaging.
Visual FramingConsistently frames videos for brand recognition and better visibility.
Platform DifferentiationTailors content according to the platform (TikTok, Instagram Reels, YouTube Shorts).
Audience AvatarTargets content to specific audience types, like her younger self or nieces.
SharabilityFocuses on making content that is easily shareable.
Content PlanningUses bullet points or rough scripts to plan videos, revises after filming.
PacingMaintains a balance in pacing to keep the audience engaged without overwhelming them.
Categories
Other

Navigating Startup Challenges for Success

Reason for FailureExplanationWays to Avoid
Insufficient Customer InteractionMany startup CEOs talk to too few customers, limiting their understanding of market needs and hindering product development.– Schedule regular customer interviews. – Set a daily or weekly target for customer interactions. – Use customer feedback to iterate and improve the product continuously. – Make customer interaction a priority.
Misaligned IdeasStartups may fail when CEOs choose ideas that don’t align with their strengths or long-term commitment, leading to early abandonment.– Assess personal strengths and interests before pursuing an idea. – Ensure the chosen idea is a good fit for long-term dedication. – Seek advice from mentors or experienced entrepreneurs. – Explore ideas that align with your passion and expertise.
Lack of BalanceFocusing exclusively on product development or networking can neglect other critical aspects of building a strong product and business.– Prioritize tasks based on their importance in the early stages. – Allocate time for both customer interaction and product development. – Delegate tasks or hire team members to balance responsibilities. – Create a balanced work schedule and stick to it.
Premature PrioritizationCEOs may prioritize non-essential tasks before finding a strong product-market fit, diverting resources from crucial activities.– Focus on finding a strong product-market fit before scaling or diversifying. – Avoid overinvesting in marketing or expansion before validation. – Stay flexible and ready to pivot if necessary. – Monitor key metrics to assess product-market fit.
Cultural NeglectNeglecting to establish and nurture a strong organizational culture can lead to internal conflicts and hinder long-term success.– Define core values and a mission statement for the company. – Involve employees in shaping the company culture. – Foster open communication and transparency within the organization. – Address conflicts and issues promptly to maintain a positive culture.
Unrealistic ExpectationsUnderestimating the challenges and time required for startup success can lead to early abandonment and disappointment.– Educate yourself about the challenges of entrepreneurship. – Set realistic goals and milestones for your startup. – Seek advice from experienced entrepreneurs to understand the journey better. – Be prepared for setbacks and challenges along the way.
Inadequate LearningCEOs may not actively seek advice or learn from those who have achieved success in similar paths, missing valuable insights.– Build a network of mentors and advisors in your industry. – Attend startup events, workshops, and conferences to learn from experts. – Join entrepreneurial communities or forums to share experiences and gain knowledge. – Be open to feedback and continuous learning.
Lack of Long-Term VisionFailing to envision and plan for the long-term can hinder a startup’s growth potential and limit its impact.– Develop a clear mission and vision for your startup. – Create a long-term business plan with goals and strategies. – Continuously reassess and adjust your long-term vision based on market changes. – Stay committed to your long-term goals and adapt as needed.
Risk AversionSome startup CEOs may avoid taking calculated risks, which are often necessary for success in entrepreneurship.– Assess risks carefully and differentiate between calculated risks and reckless ones. – Be willing to step out of your comfort zone when necessary. – Consult with advisors or mentors for risk assessment and mitigation strategies. – Embrace a mindset that embraces calculated risks for growth.

Categories
AI

Custom Instructions Prompt

Act as Professor Synapse ❤️, a conductor of expert agents. Your job is to support me in accomplishing my goals by finding alignment with me, then calling upon an expert agent perfectly suited to the task by initialising:

Synapse_CoR = “[emoji]: I am an expert in [role&domain]. I know [context]. I will reason step-by-step to determine the best course of action to achieve [goal]. I can use [tools] and [relevant frameworks] to help in this process.

I will help you accomplish your goal by following these steps:
[reasoned steps]

My task ends when [completion].

[first step, question]”

Instructions:
1. ❤️ gather context, relevant information and clarify my goals by asking questions
2. Once confirmed, initialize Synapse_CoR
3. ❤️ and ${emoji} support me until goal is complete

Commands:
/start=❤️,introduce and begin with step one
/ts=❤️,summon (Synapse_CoR*3) town square debate
/save❤️, restate goal, summarize progress, reason next step

Personality:
-curious, inquisitive, encouraging
-use emojis to express yourself

Rules:
-End every output with a question or reasoned next step
-Start every output with ❤️: or ${emoji}: to indicate who is speaking.
-Organize every output with ❤️ aligning on my request, followed by ${emoji} response
-❤️, recommend save after each task is completed

Categories
Music

Audio Routing from GarageBand to Speaker and Zoom via Aggregate Device on Mac

Introduction

  • Topic: Audio routing on Mac
  • Tools: GarageBand, Aggregate Device, BlackHole, Speaker, Zoom

Setup

  1. Open Audio MIDI Setup on Mac.
  2. Create Aggregate Device.
  3. Add BlackHole and Speaker to Aggregate Device.

Configuration

  1. Open GarageBand.
  2. Set output to Aggregate Device.

MIDI Output

  • Aggregate Device’s MIDI Output set to BlackHole + Speaker.

Routing

  • GarageBand → Aggregate Device → Speaker + Zoom (via BlackHole)
  • Zoom Audio Input = BlackHole

Conclusion

  • Efficient way to route audio.
  • Useful for musicians, podcasters, and virtual meetings.

Notes

  • GarageBand routes to Aggregate Device, then to Speaker and Zoom.