SecureBERT 2.0

Advanced Domain-Specific Language Model for Cybersecurity Intelligence

SecureBERT 2.0 is Cisco AI's officially released, domain-adapted encoder-based language model for cybersecurity and threat intelligence. Built on the ModernBERT architecture, it incorporates hierarchical encoding and long-context modeling, enabling effective processing of complex cybersecurity documents, source code, and threat intelligence reports.

Pretrained on a massive, multi-modal corpus -- including over 13 billion text tokens and 53 million code tokens -- SecureBERT 2.0 achieves state-of-the-art performance in semantic search, named entity recognition, code vulnerability detection, and threat analysis.

View on GitHub | Paper (arXiv) | Join Discord

Key Features

• Domain-Specific Pretraining -- Extensive cybersecurity corpus, including threat reports, vulnerability advisories, technical blogs, and source code
• Multi-Modal Understanding -- Integrates natural language and code for advanced vulnerability detection and threat intelligence
• Hierarchical & Long-Context Modeling -- Captures both fine-grained and high-level structures across extended documents
• Optimized for Cybersecurity Tasks -- Semantic search, NER, code vulnerability detection, and threat intelligence analysis

Hugging Face Models

Pretraining Dataset

Downstream Task Performance

Document Embedding -- Cross-Encoder

Document Embedding -- Bi-Encoder

Named Entity Recognition (NER)

Code Vulnerability Detection

Installation

Prerequisites: Python 3.10+, PyTorch 2.1+ with CUDA, Hugging Face Transformers

git clone https://github.com/cisco-ai-defense/securebert2.git

python -m venv venv
source venv/bin/activate

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers lightning tqdm pandas pyarrow

Repository Structure

.
├── mlm/                        # Model pretraining (Masked Language Modeling)
│   ├── train.py
│   └── SecureBERT_mlm_eval.py
├── vuln_classification/        # Code vulnerability detection
│   ├── CodeVuln_train.py
│   └── CodeVuln_eval.py
├── rt2/ner/                    # Named Entity Recognition (NER)
│   ├── NER_train.py
│   └── NER_eval.py
├── doc_embedding/              # Document embedding tasks
│   ├── BiEncoder_train.py
│   ├── CrossEncoder_train.py
│   ├── BiEncoder_eval.py
│   └── CrossEncoder_eval.py
├── opensource_data/            # Preprocessed datasets
├── dataset.py                  # Dataset loading utilities
└── requirements.txt

Training and Evaluation

Train on a single GPU:

cd mlm
python train.py

For multi-GPU:

torchrun --nproc_per_node=8 train.py

Evaluation example (MLM):

sentences = [
    "The attacker gained access through a [MASK] vulnerability.",
    "Users should always enable [MASK] authentication for better security.",
    "The malicious [MASK] was detected by the intrusion detection system.",
]

ground_truths = ["software", "multi-factor", "payload"]

model_ids = [
    "cisco-ai/SecureBERT2.0-base",
    "answerdotai/ModernBERT-base",
    "ehsanaghaei/SecureBERT",
]

python SecureBERT2_mlm_eval.py

Contributing

We welcome contributions including new datasets, additional downstream cybersecurity tasks, model architecture enhancements, and optimized evaluation pipelines.

Please review CONTRIBUTING.md for guidelines.

License

Apache 2.0 -- See LICENSE for details.