ark does not support multicore (forked) processing

[DmdCalvinChen]

This s a normal ANN model training code, but the cv_model code cannot execute properly—it gets stuck with no response, and this issue can be reproduced every time. However, it runs normally on RStudio Server. I'm not sure where the scheduling problem lies.

library(neuralnet)
library(NeuralNetTools)
library(caret)
library(pROC)
library(foreach)
library(doParallel)

load("ann_arg.rda")
exp_ann <- exp_vein

set.seed(20240118)

n_samples <- ncol(exp_ann)

train_index <- sample(1:n_samples, size = round(0.7 * n_samples))

train_group_list <- group_list[train_index]
test_group_list <- group_list[-train_index]

traindata <- exp_ann[, train_index]
testdata <- exp_ann[, -train_index]

k <- traindata

k <- as.data.frame(t(k))
fen <- as.data.frame(train_group_list)
fen$lasso <- ifelse(fen$train_group_list == "healthy", 0, 1)

k <- cbind(fen$lasso, k)
colnames(k)[1] <- "group"
data <- k

head(data)

groups <- data$group
gene_expression <- data[, -which(names(data) == "group")]

gene_expression <- data[, -which(names(data) == "group")]

model_data <- data.frame(group = as.factor(groups), gene_expression)

str(model_data)

formula <- as.formula(paste("group ~", paste(colnames(model_data)[colnames(model_data) != "group"], collapse = " + ")))

ann_model <- neuralnet(
formula,
data = model_data,
hidden = c(10, 5),
linear.output = FALSE,
err.fct = "ce",
lifesign = "full",
rep = 1,
algorithm = "rprop+",
stepmax = 1e+08,
)

model_data$group <- factor(model_data$group, levels = c("0", "1"), labels = c("Healthy", "Disease"))

train_control <- trainControl(
method = "cv",
number = 5,
classProbs = TRUE,
summaryFunction = twoClassSummary
)

cv_model <- train(
formula,
data = model_data,
method = "nnet",
trControl = train_control,
metric = "ROC",
tuneLength = 10
)

It gets stuck permanently at this step.

[timtmok]

Thank you for opening an issue. Your feedback will be helpful!

How large is this dataset? Is it expected to complete quickly and how long does it take when running on RStudio Server?

[DmdCalvinChen]

@timtmok Thank you for your reply, the dataset is not large, the memory usage is almost negligible relative to the amount of memory on the server, and it only takes 20s for rstudio server to complete. The point is that when this step is executed, the execution information will pop up in the console, but Positron will not pop up the execution information, and it seems completely stuck

[DmdCalvinChen]

@timtmok
[ann_arg.zip] (https://github.com/user-attachments/files/18631081/ann_arg.zip)
This is the data of the ann_arg.rda. Furthermore, I should mention that this issue was discovered while using remote SSH

[seeM]

@DmdCalvinChen thank you for the extra info! Could you please share the R kernel debug logs? That would help us investigate further.

1. Set the positron.r.kernel.logLevel setting to "debug":

   

2. Restart Positron.

3. Rerun the script.

4. Run the "Interpreter: Show interpreter output" command.

5. Copy and paste the results here (you can wrap it in <details>...</details> to make the text collapsible), or upload it as a file if it's too large.

[DmdCalvinChen]

@seeM Sorry for my ignorance.Is that the date you need?I have totally no idea.

console.txt

[seeM]

Thanks @DmdCalvinChen! That's helpful but not what I was looking for. If you click the dropdown to choose a different output:

Then select the output for the same R version but ending in "Kernel", that should get the logs we need.

Here's what a snippet of my logs looks like, for example:

[R]   2025-02-03T15:47:21.350790Z  INFO  Received shell notification: JupyterMessage { zmq_identities: [[114, 45, 55, 97, 57, 101, 52, 57, 53, 53], [114, 45, 55, 97, 57, 101, 52, 57, 53, 53]], header: JupyterHeader { msg_id: "1a9b280b-c3e5-47b2-b67e-a625ed7be0db", session: "r-7a9e4955", username: "seem", date: "2025-02-03T15:47:21.350Z", msg_type: "comm_msg", version: "5.3" }, parent_header: None, content: CommWireMsg { comm_id: "positron-variables-r-10-d8cd8a66", data: Object {"jsonrpc": String("2.0"), "method": String("list")} } }
[R]     at crates/amalthea/src/socket/shell.rs:232
[R] 
[R]   2025-02-03T15:47:21.351061Z  INFO  Environment: Received message from frontend: Rpc("1a9b280b-c3e5-47b2-b67e-a625ed7be0db", Object {"jsonrpc": String("2.0"), "method": String("list")})
[R]     at crates/ark/src/variables/r_variables.rs:158
[R] 

[juliasilge]

@DmdCalvinChen can you try out some simpler code to see if the problem is using doParallel with a remote SSH session. For example, can you execute this with your setup?

library(doParallel)
registerDoParallel(cores=2)
foreach(i=1:3) %dopar% sqrt(i)

If that works fine, can you run a simpler caret example, to see if the problem is related to that? For example, can you execute this with your setup?

library(caret)
data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

knnFit1 <- train(TrainData, TrainClasses,
                 method = "knn",
                 preProcess = c("center", "scale"),
                 tuneLength = 10,
                 trControl = trainControl(method = "cv"))

If that runs fine, then you could try this with parallel processing.

If you can find the smallest example that exhibits your example, that would be really helpful. You may find the ideas from reprex, especially Reprex do's and don'ts, helpful. Thanks! 🙌

[DmdCalvinChen]

@juliasilge Thank you for your reply and efforts. Following your suggestion, I copied the information from the R4.4.0 kernel during the lagging state and named it "R4.4.0Kernel.txt." The other file contains the kernel information of the code you provided, which is running well.

R4.4.0_YOU_example_code.txt

R4.4.0Kernal.txt

[juliasilge]

Thanks! Can you help us find more specifics about when you are observing this problem?

• Do you see the problem when you run a simple caret example with parallel processing?
• Do you see the problem when you run the neural net example but without parallel processing?
• Do you see the problem when you run a simpler neural net example (for example, with iris or mtcars)?

[DmdCalvinChen]

@juliasilge Please don't close this issue, I need to squeeze out some time to study this issue

[DmdCalvinChen]

@juliasilge Hi！After several days of troubleshooting, I discovered the following method which can stablely reproduce problems and
I used the iris dataset to reproduce.The key to the problem should be parallel operations or function calls

library(neuralnet)
library(NeuralNetTools)
library(caret)
library(pROC)
library(foreach)
library(doParallel)

data(iris)

find_seed <- function(num) {
  set.seed(num)

  n_samples <- nrow(iris)
  train_index <- sample(1:n_samples, size = round(0.7 * n_samples))

  traindata <- iris[train_index, ]
  testdata <- iris[-train_index, ]

  traindata$binary_class <- ifelse(traindata$Species == "setosa", 0, 1)
  testdata$binary_class <- ifelse(testdata$Species == "setosa", 0, 1)

  model_data <- data.frame(
    group = as.factor(traindata$binary_class),
    traindata[, 1:4]
  )

  formula <- as.formula(
    paste("group ~", paste(colnames(model_data)[2:5], collapse = " + "))
  )

  ann_model <- neuralnet(
    formula,
    data = model_data,
    hidden = c(10, 5),
    linear.output = FALSE,
    err.fct = "ce",
    lifesign = "full",
    rep = 1,
    algorithm = "rprop+",
    stepmax = 1e+08
  )

  model_data$group <- factor(
    model_data$group,
    levels = c("0", "1"),
    labels = c("Setosa", "Other")
  )

  train_control <- trainControl(
    method = "cv",
    number = 5,
    classProbs = TRUE,
    summaryFunction = twoClassSummary
  )

  cv_model <- train(
    formula,
    data = model_data,
    method = "nnet",
    trControl = train_control,
    metric = "ROC",
    tuneLength = 10
  )

  print(cv_model)
}

dates <- seq(as.Date("2024-01-01"), as.Date("2024-12-30"), by = "day")
date_nums <- as.numeric(format(dates, "%Y%m%d"))

cl <- 16
registerDoParallel(cl)

foreach(
  i = date_nums,
  .packages = c("neuralnet", "NeuralNetTools", "caret", "pROC")
) %dopar%
  {
    find_seed(i)
  }

stopCluster(cl)

The above code, if the code inside the function is not encapsulated into a function, it can run normally in positron, but once it is encapsulated into a function and called multi-threaded, it will be stuck in the cv_model step. If positron is not restarted at this time and I just simply restart R Kernal in positron or not , we will find that the code inside the function will not work properly at this time when running the code separately