Ollama Experiments ✨

Caveats

Ollama

https://ollama.com/

Getting started

Limitations

(demo)


For, most people this is okay.

But

We can do better for the BEAST

Docker to the rescue

Workaround

config.yml

test.sh

Observations

For 6 llama2 instances

  • Sequential execution :: 41+ seconds.
  • Parallel execution :: 18+ seconds.

Limiting factor

alt text

alt text

Still room to improve

Single GPU instance

5 seconds for 6 requests
only using 6 GB of GPU


splitting up requests


final_final_final.sh

num requests num_containers time taken(s)
20 1 18
20 2 10
20 3 9
20 4 10
20 5 10
40 1 39
40 2 20
40 5 20

Next steps

insert catchy Call to Action here

Thank You✨


May the FOSS be with you ⚔️


This presentation was built using Free Software 🤗

Long live the Open Source revolution!

1. Not much extensive testing done

### Resources

<ul clas="left-align"> <li class="small-text"> www.gnu.org/gnu/manifesto.html </li> <li class="small-text"> www.debian.org/ </li> <li class="small-text"> www.fsf.org/ </li> <li class="small-text"> www.youtube.com/watch?v=vwG5N8SCmZ8 </li> <li class="small-text"> indiafoss.net/2023 </li> <li class="small-text"> en.wikipedia.org/wiki/Free_software_movement </li> <li class="small-text"> slides.aldrinjenson.tech/introduction-to-foss/ </li> </ul>