Source: examples/training/halo

Halo: post-train a Hugging Face model on any cloud

Halo trains models in their native Hugging Face format with faster kernels and lower peak memory. This example uses SkyPilot to provision one H100, pull Halo’s public Hopper image, and fine-tune Qwen3-4B with LoRA.

Launch

Install SkyPilot and configure at least one cloud:

pip install "skypilot[aws,gcp,kubernetes]"
sky check

From this directory, launch the training job:

sky launch -c halo halo.yaml

The example runs for 20 steps and writes checkpoints to ~/sky_workdir/checkpoints. To run complete training, remove max_steps from train.yaml and set num_train_epochs.

Useful commands:

sky logs halo
rsync -Pavz halo:~/sky_workdir/checkpoints ./checkpoints
sky down halo

Change the hardware

The public image used here targets Hopper GPUs. H100:1 lets SkyPilot select any available H100 provider. Set an infrastructure or region without changing the training command:

sky launch -c halo halo.yaml --infra aws/us-east-2

For B200 or B300, change the accelerator and use public.ecr.aws/whitecircle/halo:blackwell.

See the Halo overview and repository for other training methods and multi-GPU configs.

Included files

halo.yaml

resources:
  accelerators: H100:1
  disk_size: 256
  image_id: docker:public.ecr.aws/whitecircle/halo:hopper

workdir: .

run: |
  halo launch sft train.yaml

train.yaml

model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
attn_implementation: flash_attention_2

dataset:
- HuggingFaceH4/ultrachat_200k@train_sft
conversation_field: messages
assistant_message_template: "<|im_start|>assistant\n"

use_peft: true
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
lora_task_type: CAUSAL_LM

per_device_train_batch_size: 4
per_device_eval_batch_size: 4
gradient_accumulation_steps: 4
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
optim: adamw_torch_fused
learning_rate: 1.0e-04
max_steps: 20
max_length: 4096
packing: true

output_dir: checkpoints/qwen3-4b-ultrachat-lora
save_strategy: steps
save_steps: 10
save_total_limit: 2
eval_strategy: 'no'
logging_steps: 1
logging_first_step: true
report_to: none
enable_efficiency_metrics: true

dataloader_num_workers: 4
remove_unused_columns: true