microsoft/mscclpp
1
0
Fork 0
mirror of https://github.com/microsoft/mscclpp.git synced 2026-07-15 11:44:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
658 Commits 125 Branches 17 Tags
b72decbfeb3dd6ee0ab980b0ae4d5752e63ac809
Commit Graph

262 Commits

Author SHA1 Message Date
Caio Rocha
c6e06cfad7 Executor AllGather In-Place Support (#365) 2024-10-21 05:45:56 -07:00
Changho Hwang
0c150e5166 Fix copyright messages (#367) 2024-10-17 21:25:46 -07:00
Ziyue Yang
5c4e105814 Fix NPKit exit event offset (#356) 2024-09-19 13:35:44 +08:00
Binyang Li
b30bb260e3 Tune threads per block for mscclpp executor (#345) 2024-09-18 17:21:47 -07:00
Binyang Li
7bedb25054 Add proxy channel related operations (#351) 
Add Flush, PutWithSignal, PutWithFlushAndSignal operation
 2024-09-15 13:24:57 -07:00
Binyang Li
26a87535f9 Fix bug for construct sempaphore (#341) 
Current semaphore construction requires two-way communication, e.g., to
construct a semaphore signaling from rank 0 to rank 1, both rank 0 and
rank 1 need to send a message to each other. This PR fixes an executor
bug that fails to conduct two-way communication for constructing such
one-way semaphores, and instead hangs during the semaphore construction.
In the future, we may need to change the implementation to construct
semaphore via one-way communication.

---------

Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
 2024-09-04 19:42:03 +08:00
Changho Hwang
72b99a4229 Fix for ROCm 6.0 (#347) 2024-09-01 20:22:33 -07:00
Caio Rocha
4eca6f1e95 Support executors to send packets over ProxyChannel (#344) 
Co-authored-by: Binyang Li <binyli@microsoft.com>
 2024-08-30 22:10:33 +00:00
Caio Rocha
1af62ea43d ProxyChannel Support in Executor (#342) 
Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
 2024-08-27 10:09:44 -07:00
Changho Hwang
1e82dd444f Make ibverbs optional at compile time (#340) 
Co-authored-by: Caio Rocha <caiorocha@microsoft.com>
 2024-08-21 12:47:05 -07:00
Caio Rocha
ead4efc315 Dynamically load libibverbs (#337) 2024-08-13 23:48:39 -07:00
Changho Hwang
8c6fb429e9 bfloat16 support (#336) 
* Add bfloat16 support for executor and NCCL interface
* Changed `gpu_data_types.hpp` into an internal header file
 2024-08-12 15:41:58 -07:00
caiomcbr
67eb9b04cc NCCL API Executor Integration (#331) 
Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
 2024-07-25 15:05:02 -07:00
Ziyue Yang
b5a48f836c Separate NPKit CPU timestamp access from different blocks for AMD platform (#321) 
Reference: https://github.com/ROCm/rccl/pull/1229
 2024-07-02 19:36:48 +08:00
Ziyue Yang
f29095b3b1 Fix NPKit support for AMD (#312) 2024-06-14 16:22:14 +08:00
Ziyue Yang
76328fe623 Add NPKit GPU event support (#310) 2024-06-13 13:59:50 +08:00
Binyang Li
80aefe55bc Cumulative Updates (#309) 
Bug fix: Unable to execute communication primitives with the same
execution plan but varying message sizes.
Add reduce_packets OP
 2024-06-12 19:17:57 +08:00
Changho Hwang
1f62dfd7cd Add C++ executor test (#304) 
- Add C++ executor test
- Fix executor bugs for packet operation
- Enhance executor_test.py

---------

Co-authored-by: Binyang Li <binyli@microsoft.com>
 2024-05-29 10:54:36 +00:00
Binyang Li
6226556ce2 Optimized the execution kernel (#294) 2024-05-03 11:54:50 -07:00
Changho Hwang
d4ede480f4 Ethernet support (#284) 
Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Caio Rocha <caiorocha@microsoft.com>
 2024-04-25 11:06:43 -07:00
Changho Hwang
89896ff94f Include GPU data types only for kernel code (#292) 2024-04-24 20:55:02 -07:00
Changho Hwang
9934c982a8 Seperate headers for GPU data types (#291) 
Prevent unnecessarily including data type headers in everywhere.
 2024-04-19 05:52:43 +00:00
Binyang Li
64d837f9ab Add executor to execute schedule-plan file (#283) 
Add executor to execute the JSON schedule file generated by msccl-tools

---------

Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
 2024-04-18 19:10:41 +00:00
Changho Hwang
5ba6ce00c7 Fix bootstrapping mechanism (#278) 
Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Pashupati Kumar <74680231+pash-msft@users.noreply.github.com>
 2024-03-27 10:24:24 +08:00
Saeed Maleki
91d592dcc0 NVLS support. (#250) 
Co-authored-by: Saeed Maleki <saemal@microsoft.com>
Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
 2024-02-04 20:46:10 -08:00
Changho Hwang
544ff0c21d ROCm support (#213) 
Co-authored-by: Binyang Li <binyli@microsoft.com>
 2023-11-24 16:41:56 +08:00
Changho Hwang
e710701728 Warning ahead of CQ being full (#202) 2023-11-15 08:03:29 +00:00
Changho Hwang
8c0f9e84d0 v0.3.0 (#171) 2023-10-11 22:35:54 +08:00
Saeed Maleki
015e29c138 adding signal for atomic op (#178) 
This address [this](https://github.com/microsoft/mscclpp/issues/177).
 2023-09-11 10:46:25 -07:00
Olli Saarikivi
828be48b21 Add Context and Endpoint classes to enable non-Communicator use-cases (#166) 
This PR implements and closes #137. The new `Endpoint` and `Context`
classes expose the connection establishing functionality from
`Communicator`, which now is only responsible for tying together the
bootstrapper with a context.

The largest breaking change here is that
`Communicator.connectOnSetup(...)` now returns the `Connection` wrapped
inside a `NonblockingFuture`. This is because with the way `Context` is
implemented a `Connection` is now fully initialized on construction.

Some smaller breaking API changes from this change are that
`RegisteredMemory` no longer has a `rank()` function (as there maybe no
concept of rank), and similarly `Connection` has no `remoteRank()` and
`tag()` functions. The latter are replaced by `remoteRankOf` and `tagOf`
functions in `Communicator`.

A new `EndpointConfig` class is introduced to avoid duplication of the
IB configuration parameters in the APIs of `Context` and `Communicator`.
The usual usage pattern of just passing in a `Transport` still works due
to an implicit conversion into `EndpointConfig`.

Miscellaneous changes:
-Cleans up how the PIMPL pattern is applied by making both the `Impl`
struct and the `pimpl_` pointers private for all relevant classes in the
core API.
-Enables ctest to be run from the build root directory.
 2023-09-06 13:10:04 +08:00
Binyang2014
858e381829 Pytest (#162) 
Port python tests to mscclpp.
Please run
`mpirun -tag-output -np 8 pytest ./python/test/test_mscclpp.py -x` to start pytest

---------

Co-authored-by: Saeed Maleki <saemal@microsoft.com>
Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
Co-authored-by: Saeed Maleki <30272783+saeedmaleki@users.noreply.github.com>
 2023-09-01 21:22:11 +08:00
Saeed Maleki
8d1b984bed Change device handle interfaces & others (#142) 
* Changed device handle interfaces
* Changed proxy service interfaces
* Move device code into separate files
* Fixed FIFO polling issues
* Add configuration arguments in several interface functions

---------

Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: root <root@a100-saemal0.qxveptpukjsuthqvv514inp03c.gx.internal.cloudapp.net>
 2023-08-16 20:00:56 +08:00
Saeed Maleki
e7d5e652df Python bindings (#125) 
Co-authored-by: Olli Saarikivi <olsaarik@microsoft.com>
Co-authored-by: Changho Hwang <changhohwang@microsoft.com>
Co-authored-by: Binyang Li <binyli@microsoft.com>
 2023-07-19 15:35:54 +08:00
Saeed Maleki
df2f0c14ab bootstrap now takes interface (#113) 
This PR fixes the issue regarding taking the interface as an input.
 2023-06-29 00:16:06 +08:00
Changho Hwang
21eed722af Add license comments (#106) 2023-06-25 12:40:12 +08:00
Changho Hwang
c4a5958dfc Fix hanging bootstrap issues (#100) 
* Renew socket interfaces and error handling into C++ style
* Fix bootstrap hanging bugs
* Misc code cleanup

---------

Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Saeed Maleki <saemal@microsoft.com>
 2023-06-15 11:29:49 +08:00
Changho Hwang
76718e4015 Saemal/atomic signal (#96) 
* code complelete

* fix correctness issue

* Fix correctness issuee

* fix lint

* ass compile

* Fix build issue

* Fix runtime error

* Fix correctness issue

* Fix crash issue

* minor change

* Fix memory leak

* Fix review comments

* Finish allgather

* address comments

* load element to register first then store to remote address

* Finish allGather

* init

* Build connections

* allreduce_test works

* Bug fix

* Add CUDA flags

* Add packet copy (LL)

* Lint

* Set tmpPtr from constructors

* Lint

* Multiple blocks per peer

* Beautify

* Temporal ring reduce

* Ring reduce works correctly

* Overlapping

* Fix overlapping

* Improve vector sum

* figuring out how to use atomics

* working now

* wip

* Enhance LL AllReduce

* Support multiple blocks per peer

* Fix a ring reduce bug

* Fix a AllReduce kernel 2 bug

* Bug fix

* wip

* Make it compilable

* Lint

* Lint

* Minor changes

* Unit test to reproduce memory consistency bugs

* Unit test bug fixes

* Fixes

* Typo

* wip

* done with core

* wip

* wip

* compiles

* only the atomic is failing

* almost working

* all tests pass now

* clang-12

* More jailbreaks

* bug fix for common.cu

* adding stdint to concurrency.hpp

* Out-of-place for AllReduce kernel 2

* Optimize `sync()`

* Fix mp_unit_tests

* Init TestEngine with TestArgs

* Change common.cu into common.cc

* Cleanup common.hpp

* Lint

* fixes to the mscclpp-tests

* fixed common.cc

---------

Co-authored-by: Binyang Li <binyli@microsoft.com>
Co-authored-by: Saeed Maleki <saemal@microsoft.com>
 2023-06-12 21:38:06 -07:00
Olli Saarikivi
5d5e9a1805 Make bootstrap use persistent sockets (#98) 2023-06-12 15:13:30 +08:00
Changho Hwang
5a4885ccbb Misc updates (#95) 2023-06-12 13:53:43 +08:00
Changho Hwang
798631bd52 Update unit tests (#81) 2023-06-08 09:58:05 +00:00
Changho Hwang
9cee6c4a74 Cleanup old files and functions (#86) 2023-06-01 17:34:57 +08:00
Olli Saarikivi
457c422791 Remove alloc.h and beef up cuda_utils.hpp (#82) 2023-05-24 08:34:18 +00:00
Changho Hwang
8d54bf3301 Update CI (#79) 2023-05-21 11:45:41 -07:00
Binyang2014
a3cf48cc5d Rewrite mscclpp-test with cpp style API (#77) 
- Rewrite mscclpp-test with cpp style API
- Add SM copy
- add new sendRecv test
 2023-05-19 14:14:19 +08:00
Olli Saarikivi
4e4d1972e3 Cuda smart pointers 2023-05-16 16:16:00 -07:00
Olli Saarikivi
d83343ef4e Make getWc not return a void pointer 2023-05-16 22:52:38 +00:00
Olli Saarikivi
dee55997e9 Remove free and most reinterpret_casts in IB code 2023-05-16 22:48:16 +00:00
Saeed Maleki
6f7ca05305 Merge remote-tracking branch 'origin/api-extension' into saemal/offloading 2023-05-12 22:43:22 +00:00
Saeed Maleki
2a7b745972 fully working with double buffering 2023-05-12 22:42:22 +00:00
Saeed Maleki
2691784b88 working -- at least for single node 2023-05-12 20:21:58 +00:00