Skip to content

Latest commit

 

History

History
27 lines (15 loc) · 1.08 KB

README.md

File metadata and controls

27 lines (15 loc) · 1.08 KB
Raw

ALM 1.0

简体中文

Overview

The Arabic Language Model (ALM) 1.0 is a pretrained language model based on autoregressive blank infilling . Below shows the count of model parameters in detail.

Name Params Layers Hidden Size FFN Hidden size Heads Head Size
ALM 1.0 335M 24 1024 4096 16 64

Training data

ALM-1.0 uses the largest open-source Arabic text dataset ArabicText 2022. You can check ArabicText 2022 for more information.

How to use

Finetune

With FlagAI, one can use ALM model directly for Seq2Seq finetuning.

Quick start

  • examples/ALM/train.py provides examples to use ALM for Seq2seq finetuning task, such as text summarization and short/long text generation.

  • examples/ALM/generate.py provides examples to use ALM for masked text prediction in an autoregressive way.