Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(p2p): Federation and AI swarms #2723

Merged
merged 18 commits into from
Jul 8, 2024
Merged

feat(p2p): Federation and AI swarms #2723

merged 18 commits into from
Jul 8, 2024

Conversation

mudler
Copy link
Owner

@mudler mudler commented Jul 5, 2024 •
edited
Loading

Screenshot 2024-07-08 at 19-31-12 LocalAI - P2P dashboard

How does it work ?

Start LocalAI with --p2p, and for sharing an instance with federation, start with --federated. A token have to be configured for workers and node joining a network (with the TOKEN environment variable).

At first start, if a token isn't supplied it is generated automatically, and can be used when navigating over the Swarm dashboard page.

Video 1 Federation

https://youtu.be/pH8Bv__9cnA

Video 1 Llama.cpp workers:

https://youtu.be/ePH8PGqMSpo

Additional notes

This is a WIP branch, my goal here is to have a very minimal dashboard + general enhancements direction is:

  • dashboard should allow to have a look at which workers are active and let monitor the status
  • give indication on how to add new workers by using the script or docker images
  • detection of dead workers to turn down tunnels
  • federated support for sharing requests and balance them across multiple LocalAI instances

This is a continuation of #2343

Also adds a fix for #2733
@mudler mudler force-pushed the p2p_enhancements branch 2 times, most recently from a2df4ad to 9704512 Compare July 5, 2024 17:25
@mudler mudler added the area/p2p label Jul 5, 2024
@github-actions github-actions bot added the ci label Jul 5, 2024
@mudler mudler added enhancement New feature or request and removed ci labels Jul 5, 2024
@netlify Netlify
Copy link

netlify bot commented Jul 5, 2024 •
edited
Loading

✅ Deploy Preview for localai ready!

Name Link
🔨 Latest commit 0a98a8e
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/668c2501f656a5000868213f
😎 Deploy Preview https://deploy-preview-2723--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.
@github-actions github-actions bot added the ci label Jul 5, 2024
@mudler mudler force-pushed the p2p_enhancements branch 4 times, most recently from 8599e8e to f04c4da Compare July 6, 2024 14:18
dave-gray101
dave-gray101 reviewed Jul 6, 2024
View reviewed changes
core/p2p/p2p_disabled.go Outdated
@@ -19,3 +19,11 @@ func LLamaCPPRPCServerDiscoverer(ctx context.Context, token string) error {
func BindLLamaCPPWorker(ctx context.Context, host, port, token string) error {
return fmt.Errorf("not implemented")
}

func GetAvailableNodes() []NodeData {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mudler if you have a sec, do you mind explaining why p2p_disabled.go is... useful? Mostly for my own understanding here - I'd have either just checked if p2p was enabled when GetAvailableNodes was called, or if performance was to be optimized, dump both versions in the same file and select the mock vs the full impl based on that setting. I'm probably missing something interesting here 😄
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's actually because it is behind the GO_TAGS user flag. If GO_TAGS contains p2p then the p2p.go file is built, otherwise the other one.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks for pointing out this is a compile time option vs a config setting
@mudler mudler force-pushed the p2p_enhancements branch 2 times, most recently from 3af5fe7 to c61ccbe Compare July 6, 2024 17:02
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 7, 2024
View reviewed changes
core/cli/run.go
}
tunnelEnvVar := strings.Join(tunnelAddresses, ",")

os.Setenv("LLAMACPP_GRPC_SERVERS", tunnelEnvVar)

Check warning

Code scanning / gosec

Errors unhandled. Warning

Errors unhandled.
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 7, 2024
View reviewed changes
core/cli/federated.go Dismissed Show dismissed Hide dismissed
core/cli/federated.go

func copyStream(closer chan struct{}, dst io.Writer, src io.Reader) {
defer func() { closer <- struct{}{} }() // connection is closed, send signal to stop proxy
io.Copy(dst, src)

Check warning

Code scanning / gosec

Errors unhandled. Warning

Errors unhandled.
core/cli/federated.go
<-closer

tunnelConn.Close()
conn.Close()

Check warning

Code scanning / gosec

Errors unhandled. Warning

Errors unhandled.
core/cli/federated.go
go copyStream(closer, conn, tunnelConn)
<-closer

tunnelConn.Close()

Check warning

Code scanning / gosec

Errors unhandled. Warning

Errors unhandled.
core/cli/federated.go Fixed Show fixed Hide fixed
@mudler
Copy link
Owner Author

mudler commented Jul 7, 2024 •
edited
Loading

ok I pushed a bit more =) this is actually adding LocalAI federation too. Let me wrap this up and update the description.

UI is kinda messy at the moment, needs improvement:

Screenshot 2024-07-07 at 15-50-03 LocalAI - P2P dashboard

  • now llama.cpp workers can be tracked easily, and from the p2p page instructions are given to add new workers
  • beside llama.cpp workers, federation works completely differently from the distributed worker mechanism (which distributes weights between all nodes) - here we will go by distributing per-request to each node, so it applies to all backends.
  • There is no logic to syncronize models in the federation, you have to start localai with the same models if you want to have a consistent behavior across your AI swarm
  • distribution of load is NOT optimized at the moment in the swarm, nodes are selected randomly (TODO: optimizations)
@mudler mudler changed the title Wip p2p enhancements Jul 7, 2024
@mudler mudler removed the ci label Jul 7, 2024
@mudler mudler marked this pull request as ready for review July 8, 2024 06:26
@github-actions github-actions bot added the ci label Jul 8, 2024
@mudler
Copy link
Owner Author

mudler commented Jul 8, 2024

I'm kinda satisfied for now:

Screenshot 2024-07-08 at 19-31-12 LocalAI - P2P dashboard
@mudler mudler removed the ci label Jul 8, 2024
@mudler mudler merged commit cca881e into master Jul 8, 2024
41 checks passed
@mudler mudler deleted the p2p_enhancements branch July 8, 2024 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/p2p enhancement New feature or request
2 participants
@mudler @dave-gray101