Skip to content

Commit

Permalink
Improve time-series transformer cfg to increase number of attention h…
Browse files Browse the repository at this point in the history
…ead for XTab

Since XTab uses tokens of much bigger size (192 vs 8), increase the number of heads in multi-head attention for the time-series transformer when used w/ XTab
  • Loading branch information
nathanpainchaud committed Nov 1, 2023
1 parent a2d1c90 commit 7fce578
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ _args_:
encoder_layer:
_target_: torch.nn.TransformerEncoderLayer
d_model: ${task.embed_dim}
nhead: 2
nhead: ${oc.select:task.model.encoder.attention_n_heads,${oc.select:task.model.encoder.encoder_layer.nhead,???}}
dim_feedforward: ${op.mul:1.5,${task.embed_dim},int}
dropout: 0.1
activation: relu
Expand Down

0 comments on commit 7fce578

Please sign in to comment.