# MStools

**Repository Path**: adrian0711/mstools

## Basic Information

- **Project Name**: MStools
- **Description**: MindSpore tools
- **Primary Language**: Python
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 3
- **Forks**: 1
- **Created**: 2025-07-25
- **Last Updated**: 2025-12-01

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Distributed Debugging Tools

This repository provides **three lightweight utilities** for debugging and tracing distributed training in PyTorch and MindSpore environments.

---

## 📂 Contents

- [Overview](#overview)
- [1. Distributed Debugger (`pdb_utils`)](#1-distributed-debugger-pdb_utils)
- [2. Execution Tracer (`execution_tracker`)](#2-execution-tracer-execution_tracker)
- [3. Function Call Stack Printer (`trace_print`)](#3-function-call-stack-printer-trace_print)
- [4. Version Snapshot Tool](#-version-snapshot-tool)
- [Documentation](#documentation)
- [License & Contact](#license--contact)

---

## Overview

| Tool                | Purpose                                               |
|---------------------|-------------------------------------------------------|
| `pdb_utils`         | Add rank-specific breakpoints in distributed training |
| `execution_tracker` | Trace Python method entry/exit with timing            |
| `trace_print`       | Print stack when specific functions are called        |

---

## 1. Distributed Debugger (`pdb_utils`)

Enable interactive `pdb` debugging for selected process rank, while other processes are paused (or skipped):

### 🔧 Usage

In your training shell script:

- For **MindSpore**:
```bash
  source tools/debug_utils/dist_debug.sh
  exit
````

* For **PyTorch**:

```bash
  source tools/debug_utils/dist_debug_pt.sh
  exit
 ```

In Python code:

```python
from pdb_utils import breakpoint_, clear_

breakpoint_()          # Pause other ranks
breakpoint_(False)     # Skip pause for others
clear_()               # Resume others after debug
```

➡️ See full guide: [`distributed_debug.md`](./dist_debug/distributed_debug.md)

---

## 2. Execution Tracer (`execution_tracker`)

Trace method entry and exit within specific directories (e.g. `megatron`, `mindspeed_llm`) with indentation and execution time.

### Example

```python
from execution_tracker import ExecutionTracer

with ExecutionTracer():
    your_fn()
```

➡️ Code: [`execution_tracker.py`](./trace_print/execution_tracker.py)

---

## 3. Function Call Stack Printer (`trace_print`)

Print the full call stack when specific functions (by name or regex) are invoked.

### Example

```python
from trace_print import TraceContext

with TraceContext('train_step', regex=['forward.*']):
    main()
```

➡️ Code: [`trace_print.py`](./trace_print/trace_print.py)

---

## 4. Version Snapshot Tool

Copy files Under [`version_control`](./version_control) to MindSpeed-Core-MS, then you can:

- `export_version.sh`: Export current commit info of all repos to a file.
- `import_version.sh`: Restore each repo to the specific commit.

Useful for synchronizing debug experiments or team collaboration.

📖 [See details → version_control/README.md](./version_control/README.md)

---

## Documentation

* [🔍 Distributed Debugging Guide](./dist_debug/distributed_debug.md)
* [🧭 Execution Tracing](./trace_print/execution_tracker.py)
* [📌 Function Stack Printing](./trace_print/trace_print.py)

---

## License & Contact

This project is released under the Apache License.
For issues or suggestions, feel free to submit an issue or pull request.