# About Short technical notes, experiments, and memorable experiences. Hi, I'm **Aldrin Jenson** β€” an engineer working on multi-agent AI systems at [Athena Intelligence](https://athenaintelligence.ai/) in New York. This is a small wiki where I keep: * **Quick technical notes** β€” things I had to look up once and wanted to keep handy (LLM inference options, CLI tricks, media tooling, git workflows). * **Experiments & side projects** β€” small builds from voicenotes and free afternoons. * **Talks & recognitions** β€” a place to remember workshops I've spoken at and moments I'm proud of. ## Elsewhere * πŸ“ Longer-form writing, projects & talks: [aldrinjenson.com](https://aldrinjenson.com/) * πŸ’» Code and side projects: [github.com/aldrinjenson](https://github.com/aldrinjenson) * πŸ’Ό Work & professional: [linkedin.com/in/aldrinjenson](https://www.linkedin.com/in/aldrinjenson/) * 🐦 Short builds and demos: [x.com/aldrinjenson](https://x.com/aldrinjenson) *** This site is inspired by a [similar one](https://til.kurianbenoy.com/) from my good friend [Kurian Benoy](https://kurianbenoy.com/). Hope you find something useful here. # Different options for doing LLM inference Note formatted by Claude after from my voicenote transcription. About different options I've explored for hosting gen-ai models ### Intro While many developers default to using OpenAI's APIs for large language model (LLM) inference, there are numerous alternatives available that offer different advantages in terms of cost, performance, and flexibility. This guide explores various options for LLM inference, including both self-hosted and cloud-based solutions, as well as serverless and traditional hosting approaches. ### Cloud-Based Solutions #### 1. Replicate * **Pros**: * Offers thousands of models, including popular and fine-tuned versions * Supports custom model deployment (both public and private) * Easy to get started quickly * **Cons**: * Initial pricing may be higher compared to other options * If you want to build a prototype quickly for a startup, this is recommended #### 2. Modal.com * **Pros**: * Affordable pricing (e.g., 10 applications for $9 on GPUs) * Serverless architecture * **Cons**: * Cold start issues, which can lead to longer initial processing times * Based on experience with Indic Subtitler, this is cheap but the free tier will have sufficient lag if you have a large model which needs to be loaded to memory once the serverless system boots up #### 3. Hugging Face Spaces * **Pros**: * Always running, no cold start issues * Exposes API using Gradio * **Cons**: * No pay-as-you-go pricing #### 4. Fal.ai * Offers serverless inference options * Compare with Replicate and make a decision #### 5. Groq * **Pros**: * Significantly cheaper than OpenAI (e.g., 10x less cost for Whisper model) * Faster processing due to specialized LPU chips * **Cons**: * Limited model availability (currently offering Whisper v3, Llama 2, Llama 3, Gemma, Mixtral) * Requires contacting for premium trial, with potential delays as of now #### 6. AWS Options * **Bedrock**: Umbrella service for various AI tools * **SageMaker**: For model training and deployment #### 7. NVIDIA NIM (New) * **Pros**: * Containerizes LLM models (similar to Kubernetes approach) * Allows local development and easy cloud deployment * Simple inference, similar to OpenAI APIs * Supports local testing and cloud deployment * **Cons**: * Currently not open for everyone (requires filling out a form to try) * How it works: * Containerize your LLM model (e.g., GPT-3.5, GPT-4) * Develop locally * Push to cloud, which spins up a separate container for the model to run ### Self-Hosted Solutions #### 1. eXLama V2 * **Pros**: * One of the fastest open-source LLM inference options * Supports quantized models * Based on LLaMA.cpp * **Cons**: * Cannot offload to CPU; requires good VRAM #### 2. VLLM * **Pros**: * High-performance inference * **Cons**: * Does not support quantized models #### 3. LLaMA.cpp * **Pros**: * Supports quantized models * Offers HTTP server functionality for API exposure #### 4. BentoML * Supports serverless inference * Kubernetes based * Open Source #### 5. Seldon.io * Kubernetes-based open-source solution for scaling models * Alternative to BentoML #### 6. Ollama * **Pros**: * One of the easiest ways to run models locally * Supports multiple GGUF format versions easily pullable from Ollama hub * Offers both CLI and API interfaces * Recently added support for concurrency and parallelization * **How it works**: * Pull models from Ollama hub * Run locally with simple commands * Use CLI or API for inference ### Considerations When Choosing an LLM Inference Solution 1. **Cost**: Compare pricing structures, especially for long-term usage. 2. **Performance**: Consider factors like cold start times and inference speed. 3. **Scalability**: Evaluate how well the solution can handle increasing loads. 4. **Flexibility**: Check support for custom models and fine-tuning capabilities. 5. **Ease of Use**: Consider the learning curve and integration complexity. 6. **Hardware Requirements**: For self-hosted solutions, ensure you have the necessary hardware (e.g., GPUs with sufficient VRAM). ### Conclusion While OpenAI's APIs are popular, exploring alternative LLM inference options can lead to cost savings, improved performance, or better alignment with specific project requirements. For those just starting, cloud-based solutions like Replicate or Modal.com offer an easy entry point. As your needs grow or become more specific, you might consider transitioning to more comprehensive services like AWS or self-hosted solutions for greater control and customization. For local development and testing, tools like Ollama provide an excellent balance of ease of use and flexibility. NVIDIA NIM presents an interesting containerized approach that could bridge the gap between local development and cloud deployment. Remember to regularly reassess your chosen solution as the field of AI and LLMs is rapidly evolving, with new options and improvements emerging frequently. # Running parallel Ollama inference Easy way to parallely infer with multiple ollama instances all running in the same machine ### Problem Ollama runs based on a single backend concept. Even if you have good specs in your system, every new request is processed in a queue manner to make maximum resource usage for a single query. This is useful for personal laptop usage. But if you have a powerful maching with 10s of GPU RAM, and multiple users inferring at same time, it would be a problem ### Solution Run multiple ollama instances as isolated container using docker. ### References: * [Ollama blog ](https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image)announcing docker containers ### Approach 1 Use docker swarm ```bash docker swarm init ``` ```bash docker service create --replicas 10 \ --name ollama \ --constraint 'node.labels.gpu'==true \ --mount type=volume,source=ollama,target=/root/.ollama \ --publish published=11434,target=11434 \ ollama/ollama ``` This creates 10 replicas managed by docker swarm. The base url is same as that of a single instance which you can connect to just like connecting to a regular standalone instance of ollama **Problem:** This one doesn't have GPU support ### Approach 2 Manage everything ourselves with a tiny bit of help from litellm ### Steps: 1. Write a script to create the specified number(n) of ollama container from port 11434 to 11434+n 2. Ensure that all of them uses the same volume to store the models so as to save up space 3. Create litellm proxy server using a custom config file(generated automatically from create script) 4. Connect to this litellm proxy just as connecting to a single ollama instance. ```bash #!/bin/bash # create.sh - to create a given set of docker containers running llama2 CONFIG_FILE="config.yml" VOLUME_NAME="ollama_volume" MODEL="llama2" echo "model_list:" > $CONFIG_FILE # Define function to create and manage ollama containers create_ollama_container() { local container_name="ollama$1" local port=$2 # Check if the container with the same name exists, if yes, remove it if docker ps -a --format '{{.Names}}' | grep -q "^$container_name$"; then echo "Removing existing container: $container_name" docker rm -f $container_name fi # Check if the container on the specified port exists, if yes, remove it if docker ps -a --format '{{.Ports}}' | grep -q "0.0.0.0:$port->"; then echo "Removing existing container on port $port" docker rm -f $(docker ps -a --format '{{if (index (split .Ports "/") 0) | (index (split . "->") 0) | (eq "'"0.0.0.0:$port"'")}}{{.Names}}{{end}}') fi # Create and run the new container docker run -d --gpus=all \ -v $VOLUME_NAME:/root/.ollama \ -p $port:11434 \ --name $container_name \ ollama/ollama echo "Created container: $container_name on port $port" # Add entry to the config file echo " - model_name: $MODEL" >> $CONFIG_FILE echo " litellm_params:" >> $CONFIG_FILE echo " model: ollama/$MODEL" >> $CONFIG_FILE echo " api_base: http://localhost:$port" >> $CONFIG_FILE } # Create n different ollama containers with unique names and ports for i in {1..6}; do port=$((11434 + i)) create_ollama_container $i $port done ``` destroy.sh ```bash #!/bin/bash # Script to destroy running docker containers docker stop $(docker ps -a -q) && docker rm $(docker ps -a -q) ``` test.sh ```bash #!/bin/bash # test.sh - to make request to a given number of llama2 instances rm file.txt && echo "file.txt cleare" # Define the command to be executed curl_command="curl --location 'http://0.0.0.0:8000/chat/completions' --header 'Content-Type: application/json' --data ' { \"model\": \"llama2\", \"messages\": [ { \"role\": \"user\", \"content\": \"tell a long story about unicorns\" } ] }' | jq .choices[0].message.content >> file.txt" # Run the command n times in parallel for i in {1..3}; do eval "$curl_command" && echo "Command $i succeeded." & # Run the command in the background done # Wait for all background processes to finish wait echo "All commands completed." ``` Sample config.yaml ```yaml model_list: - model_name: llama2 litellm_params: model: ollama/llama2 api_base: http://localhost:11435 - model_name: llama2 litellm_params: model: ollama/llama2 api_base: http://localhost:11436 - model_name: llama2 litellm_params: model: ollama/llama2 api_base: http://localhost:11437 - model_name: llama2 litellm_params: model: ollama/llama2 api_base: http://localhost:11438 ``` # Generate subtitles locally using Whisper Easy script to take in an audio or video file and generate srt files using whisper cpp ```bash #!/bin/bash # Single script to convert a file to wav and have it be transcribed by whisper # Check if a filename is provided as an argument if [ -z "$1" ]; then echo "Usage: $0 " exit 1 fi fileName="$1" # If the file is not already a WAV file, convert it if [[ $(file -b --mime-type "$fileName") != audio/wav ]]; then echo "Converting to WAV file" # Modify the conversion command as needed ffmpeg -i "$fileName" -acodec pcm_s16le -ar 16000 "$(basename "$fileName" .${fileName##*.}).wav" convertedFileName="$(basename "$fileName" .${fileName##*.}).wav" else convertedFileName="$fileName" fi if [ ! -f "$convertedFileName" ]; then echo "Error: Conversion failed." exit 1 fi # Transcribe the WAV file echo -e "\n\n\nTranscribing..\n\n\n" ~/code/whisper.cpp/main -osrt -m /Users/aldrinjenson/code/whisper.cpp/models/ggml-medium.en.bin "$convertedFileName" && echo "Transcribed successfully" || echo "Error in transcribing!" ``` ### Next steps: Add the above script to path P.S. Do remember to change the path of whsiper.cpp to the correct location in your computer. # Changing git commit email address Easy bash script to change the emails used in a git repo To change the email address for all commits in your Git repository to a new correct one, you can use the `git filter-branch` command. Here's a step-by-step process: 1. First, ensure you have the correct email set for future commits: ``` git config user.email "your-new-correct-email@example.com" ``` 2. Then, run the following command to rewrite the history: ``` git filter-branch --env-filter ' OLD_EMAIL="your-old-incorrect-email@example.com" CORRECT_EMAIL="your-new-correct-email@example.com" if [ "$GIT_COMMITTER_EMAIL" = "$OLD_EMAIL" ] then export GIT_COMMITTER_EMAIL="$CORRECT_EMAIL" fi if [ "$GIT_AUTHOR_EMAIL" = "$OLD_EMAIL" ] then export GIT_AUTHOR_EMAIL="$CORRECT_EMAIL" fi ' --tag-name-filter cat -- --branches --tags ``` \ Replace `your-old-incorrect-email@example.com` with the wrong email you used, and `your-new-correct-email@example.com` with your correct email.\\ 3. After running this command, Git will rewrite your repository's history, changing the email address for all commits.\\ 4. If you've already pushed your commits to a remote repository, you'll need to force push the changes: ``` git push --force --tags origin 'refs/heads/*' ``` Be cautious with this operation, especially if you're working on a shared repository, as it rewrites history. This can cause issues for other contributors if they've based work on the old history. source: Claude sonnet 3.5 :) # Adding drag-drop from terminal in macOS Easy way to drag and drop from terminal to anywhere else in Mac OS Use this cool tool: {% embed url="" %} ### Steps to install: 1. git clone the above repo 2. cd dragterm 3. `brew install cocoapods` 4. `g++ DTDraggingSourceView.m main.m -framework Cocoa -o drag` 5. Add the folder to PATH `export PATH=":$PATH"` ### Usage: `drag ` ### Bonus If you are using `ranger` or some other CLI based file manager, you can easily add support to dragging from terminal using `dragterm` with a key-map as follows: ```bash map shell drag %p ``` Paste the above to `.config/ranger/rc.conf` to map ctrl+d to drag a file from ranger # Compressing images from the CLI Easy script using imagemagick to compress images from the terminal ### Steps * Move all your image files to a folder * calculate it's folder size using `du` * create a copy of the folder. * install imagemagick `brew install imagemagick` * in this folder, perform the following actions: ```bash mkdir compressed for i in *; do convert $i -resize 40% compressed/$i; done; ``` The above will compress all images by making the size 40% of the initial size and will move it to a folder called `compressed` * Check folder structure again using `du` * if it's still too much, repeat again but this time reduce quality to even lower # Clone any website to local using HTTrack Cool utility to clone any website ```bash brew install httrack httrack ``` Source: # Fetching train availability in IRCTC Easy way to fetch availability of all trains at once in IRCTC page ### Problem When you search for trains in [irctc](https://www.irctc.co.in) page, it never shows the seat availability directly. You'll have to manually go and click the refresh button of every listed train

List of trains, but availability is not present

### Steps to fix 1. Search for trains in 2. Apply filters to narrow down the list of trains

Narrow down list of trains using filters

3. Paste the following script in console to get the availability status of all trains at once ```javascript document.querySelectorAll('.fa.fa-repeat').forEach(el=>el.click()) ``` 4. The end. Tada! ### Note: Ensure that you always try this out after narrowing down the list using filters. Else it'll be too many requests going on at once. # YouTube transcript formatter Bash script to download youtube transcript and format them to a prompt you can pas to an LLM **Demo here:**\ ```bash #!/bin/bash # YouTube Transcript Extractor # This script downloads and extracts the transcript from a YouTube video, # calculates some statistics, and copies the result to the clipboard. # Instructions: # 1. Ensure you have the following dependencies installed: # - yt-dlp: https://github.com/yt-dlp/yt-dlp#installation # - jq: https://stedolan.github.io/jq/download/ # - xclip (for Linux) or pbcopy (for macOS, comes pre-installed) # 2. Make this script executable: chmod +x youtube_transcript_extractor.sh # 3. Run the script with a YouTube URL: ./youtube_transcript_extractor.sh "https://www.youtube.com/watch?v=VIDEO_ID" # Requirements: # - Linux or macOS # - Bash shell # - yt-dlp # - jq # - xclip (Linux) or pbcopy (macOS) # Function to copy to clipboard copy_to_clipboard() { if command -v pbcopy > /dev/null; then pbcopy elif command -v xclip > /dev/null; then xclip -selection clipboard else echo "Unable to copy to clipboard: neither pbcopy nor xclip is available." return 1 fi } # Function to format duration format_duration() { local total_seconds=$1 local hours=$((total_seconds / 3600)) local minutes=$(( (total_seconds % 3600) / 60 )) local seconds=$((total_seconds % 60)) printf "%02d:%02d:%02d" $hours $minutes $seconds } # Check if a URL is provided if [ $# -eq 0 ]; then echo "Please provide a YouTube video URL as an argument." echo "Usage: $0 " exit 1 fi # YouTube video URL URL="$1" # Download subtitles and metadata yt-dlp --skip-download --write-subs --write-auto-subs --sub-lang en --sub-format ttml --convert-subs srt --output "transcript.%(ext)s" --write-info-json "$URL" # Clean up the subtitle file sed -i.bak -e '/^[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9] --> [0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9]$/d' \ -e '/^[[:digit:]]\{1,3\}$/d' \ -e 's/<[^>]*>//g' \ ./transcript.en.srt && rm ./transcript.en.srt.bak # Create the final output file sed -e 's/<[^>]*>//g' -e '/^[[:space:]]*$/d' transcript.en.srt > output.txt # Remove the intermediate .srt file rm transcript.en.srt echo "Transcript extraction complete. The result is saved in output.txt" # Calculate word count and line count WORD_COUNT=$(wc -w < output.txt) LINE_COUNT=$(wc -l < output.txt) # Extract metadata TITLE=$(jq -r '.title' transcript.info.json) DURATION=$(jq -r '.duration' transcript.info.json) UPLOAD_DATE=$(jq -r '.upload_date' transcript.info.json) # Format duration DURATION_FORMATTED=$(format_duration $DURATION) # Prepare metadata string METADATA="Video Title: $TITLE Duration: $DURATION_FORMATTED Upload Date: ${UPLOAD_DATE:0:4}-${UPLOAD_DATE:4:2}-${UPLOAD_DATE:6:2} Word Count: $WORD_COUNT Line Count: $LINE_COUNT" echo "Transcript Statistics and Metadata:" echo "$METADATA" # Copy the transcript to clipboard with the specified format and metadata { echo "The following youtube video transcript:" echo "" cat output.txt echo "" } | copy_to_clipboard if [ $? -eq 0 ]; then echo "Transcript and metadata have been copied to clipboard." else echo "Transcript and metadata were not copied to clipboard due to an error." fi # Clean up metadata file rm transcript.info.json ``` # Super fast voice-based translation system > > Super simple voice based translation system with prolly the fastest speeds available right now > > β€” Aldrin Jenson (@aldrinjenson) [Original tweet date](https://twitter.com/aldrinjenson/status/1814683816679903551) # Talking AI companion A short afternoon build β€” a talking AI companion for tough times. A quick afternoon build β€” a voice-based AI companion meant to help people talk things out when they're having a rough time. [Demo on X β†’](https://twitter.com/i/status/1773643579140760053) # Conducting an LLM Workshop at NIT Calicut I got invited as a speaker to conduct a 6 hour Workshop on Large Language Models(LLMs) for FOSS Meet'24 at NIT Calicut in the end of March. It was nice. There was Kurian Benoy and Nithin Sabu as co-speakers to help as well. Met some cool people over there as well.

Me, Kurian and Nitin Sabu. 3 xMEC-ians

With my trusty ol' Thinkpad

Cool friends I've come to know over the years.

# Speaking at MEC.Conf Reminiscence of one of the proudest moments of 2023 This was one of the great memories! I was invited to give a talk on Free and Open Source Software (FOSS) to an audience of around 150 people. At first I was very hesitant. I gave the volunteers the contact number of a lot of other people. Still, in the end it came back to me due to multiple reasons. I spent days making a good presentation. And it paid off! it paid off big time!! This was one of the happiest moments. At the end of the talk, I got such great feedback that I was very proud and happy! I talked about FOSS, the origins, the 4 freedoms, why one should care, how to get started etc. This was a topic that was very close to my heart. ​​​![](https://1259382341-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHL8rOAUAM6ldGpuVUmH7%2Fuploads%2FUPUqWiIQF2bar2AdMLc1%2Fimage.png?alt=media\&token=baf16b9f-3c2c-43ee-a60e-a427e0b0d05f)![](https://1259382341-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHL8rOAUAM6ldGpuVUmH7%2Fuploads%2Fx0gQHntDZ03uPm1pIVIf%2Fimage.png?alt=media\&token=9e71d5ce-a71c-4223-9898-ca288e6ae676)

Me showing how Open Source helped me build one of the most used project of mine - MEC Diffusion​

With Iqbal and the organizing team

I was heavily inspired by a good friend Abraham Raji and a lot of very enthusiastic people from the FOSS community, because of which all the effort put into the talk felt very much worth it. I used several real-life examples and analogies (the 4 Freedoms through a Masala Dosa example) to make the concepts much clearer and more relatable. # GIDS Conference participant representing Thoughtworks Was fortunate enough to be able to represent Thoughtworks and go to the Great International Developer Conference (GIDS) 2024 at Bangalore. We demoed multiple of our creations along with a Text to Song demo which impressed a lot of people including the CTO of Thoughtworks!

With the CTO Emerita of Thoughtworks

Met some great people IRL!

{% file src="/files/B2qyMZyLbQIPDYuye1VJ" %} Experience sharing by a manager at Google about our booth and the Text to Song Demo! {% endfile %}

Goodies and stuff!

# Being a judge at Hack4Tomorrow hackathon I was fortunate to be invited as a Judge for the Hack4Tomorrow hackathon conducted by Excel 2023 This was a nice new experience. From participating in hackathons to Organizing them to now finally Judging one!

With Rindish Krishna, Afnan, Vishruth, and Abid Omar - my fellow judges. Cool people

It was a good hackathon. Nice projects After discussing among the judges, we finalized a few projects for the prize money. But I felt a bit sad thinking of the other teams who didn't win. I don't think I'd want to go into judging again if possible. The judges may have valid reasons to select certain projects, but it's hard for other participants to directly understand them I feel. Especially since they spent so much effort in the past 24-48 hours on their work in the hopes to win the prize. Since I've been on both sides, I personally would prefer to stay against judging I think. # Honours degree from the Kerala Education Minister I was awarded the Bachelor's in Technology with an Honours in Machine Learning degree in Machine Learning by R.Bindu - the honorable Minister of Higher Education of Kerala state. I was awarded the Bachelor's in Technology with an Honours in Machine Learning degree in Machine Learning by R.Bindu - the honorable Minister of Higher Education of Kerala state. This was given to around 50 people from the whole state who qualified.

Model Engineering College Students at University Convocation with our Former Principal