Abstract: Multi-object tracking of unmanned aerial vehicle (UAV) swarms remains a challenge because UAV targets often have small scales, highly similar appearances, dense spatial distributions, frequent mutual occlusions, and complex relative motions. In these scenarios, detection boxes are easily affected by scale variations, motion blur, and background interference, while confidence scores may fluctuate across frames. As a result, conventional association strategies that rely primarily on bounding-box overlap, motion prediction, or appearance cues may suffer from identity switches and trajectory fragmentation. To address these problems, this paper proposes a topology-aware dual-association multi-object tracking method that integrates geometric and local topology constraints into a two-stage tracking-by-detection framework. In the first stage, high-confidence detections are matched with predicted trajectories. Instead of using only the intersection-over-union (IoU) metric, the proposed method adopts Distance-IoU (DIoU) as the geometric similarity measure. DIoU jointly considers the overlap between bounding boxes and the distance between their center points, which provides stronger spatial discrimination when adjacent UAV targets have similar overlap ratios or when detection boxes fluctuate slightly. This design enhances the spatial consistency between the predicted trajectory and detection boxes while preserving the efficiency of online tracking. In the second stage, unmatched trajectories were associated with low-confidence detections for target recovery. To reduce the association ambiguity under weak detection, dense distribution, and short-term occlusion, a local neighborhood topology constraint is introduced. For each detection target, the local topology is constructed from its nearest neighboring targets and represented by normalized relative distances and directional relationships. For existing trajectories, the topology information is inherited from the most recent reliable matched detection and updated after each successful association. The topology correspondence between a trajectory and candidate detection is formulated as a bipartite matching problem. The Hungarian algorithm is used to obtain the minimum-cost correspondence between two local neighborhoods. The topology cost is then combined with the IoU cost in the low-confidence recovery stage. When valid neighbors are insufficient, the method automatically returns to a pure IoU association to avoid unreliable topology constraints in sparse scenes. To evaluate the proposed method, an enhanced UAV swarm dataset called UAVSwarmV2, was constructed based on the original UAVSwarm dataset. UAVSwarmV2 contains 96 sequences and 21,439 frames, with an expanded data scale, higher image resolution, longer sequences, and more diverse swarm motion patterns. It also provides unified annotations and data splits for both detection and multi-object tracking tasks to make it suitable for evaluating the association robustness and identity preservation in real visible-light UAV swarm scenarios. Experiments were conducted using YOLOv8-n as the detector under the same detection results and evaluation protocol. The proposed method was compared with representative multi-object tracking methods, including ByteTrack, BoT–SORT, OC–SORT, and Deep OC–SORT. Experimental results on the UAVSwarmV2 test set show that the proposed method achieves better overall tracking performance and stronger identity preservation. Compared with the ByteTrack baseline, the proposed method improved the higher-order tracking accuracy from 53.741% to 58.113%, association accuracy from 58.964% to 66.620%, and identification F1 score from 71.356% to 79.157%. The number of identity switches was reduced from 230 to 38, indicating a significant improvement in identity consistency. The proposed method maintains a processing speed close to that of the baseline tracker, indicating that it improves the association robustness while preserving practical tracking efficiency.