Server won't run on Windows host, Linux container if using volumes

[rogerdawson]

We were trying to run the script below on a Windows host (Windows 10 Pro 1809+) configured for Linux docker containers as per these MS docs:

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=" -p 1433:1433 -v :/var/opt/mssql --name mssql -d mcr.microsoft.com/mssql/server:2017-latest

and the SQL server crashes on startup and the container stops running with a 'kernel bug error'. An image from 3+ weeks ago appears to be fine.

The container runs fine if I remove the host volume reference.

The container runs fine if running on a Linux host.

This problem occurs on multiple Windows PCs (4 different PC's).

It would appear 2017-CU14-ubuntu, 2017-latest, 2017 latest-ubuntu all have this problem (same image id's so expected), and we also tried 2019-CTP2.4 which has the same issue. An older image such as 2017-CU13 or earlier worked fine suggesting it's a problem with the latest releases.

[forkazmoden]

Hi,

Could you share the docker version and file system information for analysis? Here is a similar thread.
Docker mssql-server-linux crash when mounting a volume

[rogerdawson]

Hi,

we saw that thread and the one it refers to but they refer to messages about evaluation versions which we didn't see in our logs. They also seem to predate this problem, with it stretching back to 2017 and seemingly having no effective solution (people are getting it in January of this year though). As far as we can tell the issue we are seeing is only with the newest versions of the SQL server. Here is the other information you requested:

File System: NTFS, on 4 of the PC's. Have also tried mounting to a FAT formatted drive and this didn't work.

Docker Version: 18.09.2, Community edition

We don't believe this is a docker drive sharing issue, as the container does successfully write some of the files including the SQL error logs. We've also tried other containers that just write data to a file and they are fine. My PC was working fine but once I deleted the latest SQL image I had (most likely CU13), and got the very latest it stopped working, in an identical manner to the other 3 PC's we'd been trying to get working.

Thanks,

Roger

[tiptopdevelopment]

Any solutions? My mssql worked for like a month and then after restart it started giving me that error...

[jonny64bit]

Yeah i was getting the same problem. I had to go back to 2017-CU11-ubuntu to find a working verison.

[winterIslander]

2017-CU11-ubuntu did not work for me. @jonny64bit , could you share your docker command please?

[jonny64bit]

@winterIslander no problem.

docker network create netty
docker run --name "SQL" --network="netty" -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=SqlServerSuperPassword2017" -p 1433:1433 -v C:/DockerData/SQL:/var/opt/mssql --restart=always -d mcr.microsoft.com/mssql/server:2017-CU11-ubuntu

[fourpastmidnight]

This is still happening as of SQL Server 2017 CU14 and 2019 CTP 2.5 using Docker Desktop and LCOW.

Docker Desktop: 2.0.4.0 (33772), Edge
Engine: 19.03.0-beta3
Compose: 1.24.0
Windows 10 Enterprise 1809 (17763.475)

My organization does run Sophos Anti-Virus. I added dockerd.exe, com.docker.proxy.exe, Docker.Watchguard.exe, docker-compose.exe, and com.docker.service to A/V exclusions, as well as the D:\ContainerVolumes, C:\ProgramData\Docker and C:\ProgramData\DockerDeskotp folders since it was reported elsewhere that Symantec Endpoint Protection caused issues and Sophos has caused many issues for me with many things in the past as well. (I'm unable to turn off the A/V, nor uninstall it.)

I read in the very long issue #136 that CU11 and/or 12 worked (and possibly CU7, but I haven't tried that one yet), but alas, these versions didn't work either. I went backward from CU14 to CU9. Same error every time:

PS C:\src\git\MyApp> docker-compose up
Creating network "myapp_default" with the default driver
Creating myapp_db_1 ... done
Attaching to myapp_db_1
db_1  | This program has encountered a fatal error and cannot continue running at Sun May  5 04:55:08 2019
db_1  | The following diagnostic information is available:
db_1  |
db_1  |        Reason: 0x00000006
db_1  |        Status: 0x40000015
db_1  |       Message: Kernel bug check
db_1  |       Address: 0x6b844180
db_1  |    Parameters: 0x10861f680
db_1  |    Stacktrace: 000000006b92fc73 000000006b8441db 000000006b831422
db_1  |                000000006b83fe62 000000006b92e08a 000000006b92cb8d
db_1  |                000000006b96c0d2 000000006bcac000 000000006bc38000
db_1  |                000000006bc40000 0000000000000001
db_1  |       Process: 8 - sqlservr
db_1  |        Thread: 109 (application thread 0x4)
db_1  |   Instance Id: e3912638-f443-4bb1-93db-f3c6f0fce006
db_1  |      Crash Id: 825655cf-7d1a-4858-9c77-54fc9a9fbc9a
db_1  |   Build stamp: b2ce95e3a6684060d1d3bb1264841a1a8fc1d501ed6d1cfdb89cfcdde8048253
db_1  |  Distribution: Ubuntu 16.04.6 LTS
db_1  |    Processors: 2
db_1  |  Total Memory: 4107853824 bytes
db_1  |     Timestamp: Sun May  5 04:55:08 2019
db_1  |
db_1  | Ubuntu 16.04.6 LTS
db_1  | Capturing core dump and information to /var/opt/mssql/log...
db_1  | dmesg: read kernel buffer failed: Operation not permitted
db_1  | No journal files were found.
db_1  | No journal files were found.
db_1  | Attempting to capture a dump with paldumper
db_1  | Captured a dump with paldumper
db_1  | Core dump and information are being compressed in the background. When
db_1  | complete, they can be found in the following location:
db_1  |   /var/opt/mssql/log/core.sqlservr.05_05_2019_04_55_08.8.tbz2
myapp_db_1 exited with code 1

Here's the relevant portion of my compose file:

version: '3.7'

services:
  db:
    image: mcr.microsoft.com/mssql/server:2017-CU14-ubuntu
    ports:
      - "11433:1433"
    environment:
      ACCEPT_EULA: "Y"
      SA_PASSWORD: "p@55w0rd"
    volumes:
      - D:\ContainerVolumes\MyApp\MSSQL\SysDbs:/var/opt/mssql
      - D:\ContainerVolumes\MyApp\MSSQL\AppDbs:/var/opt/sqlserver
      #- myapp-sysdbs:/var/opt/mssql
      #- myapp-appdbs:/var/opt/sqlserver

    # I needed this because even though supposedly SQL Server 2017 is supposed to now support
    # VMs with only 2GB memory, I got an error saying I needed at least 2Gb memory.
    # This may have been a leftover since the Docker page for MSSql for Linux says the latest tag
    # is CU3 (and that's what I started with) even though it's not when you view _all_ tags (which
    # lead me to the CU14 image). 
    mem_limit: "4gb"
    healthcheck:
      test: [ "CMD", "sqlcmd", "-U", "sa", "-P", "p@55w0rd", "-Q", "select 1" ]
      interval: 1s
      retries: 20

# Tried this per another issue thread on GH
#volumes:
#  myapp-sysdbs:
#    external: true
#  myapp-appdbs:
#    external: true

I was able to successfully start the container by NOT using volume maps (either volumes or bind mounts). I was able to docker cp the files out of var/opt/mssql to my host bind mounts and then start the container using the bind mounts just fine. But alas, even this stopped working. I've looked at all log files I could find. After looking through the log files, it appears there's an issue with accessing the host filesystem (using either bind mounts or the external docker volumes), which is causing the above kernel bug check, specifically when SQL Server attempts to create the system databases.

For now, I'm going to have to resort to "rolling my own image" for Windows because a) these Linux images just don't work and b) you guys aren't keeping the Windows images updated. 😞 [That last statement, while technically, true, really means that I need to build my own image using the provided Dockerfile in this repository, as the Dockerfile (I believe) will result in an up-to-date image of SQL Server 2017 with all the CUs applied.]

[yang-zhang-syd]

I am having the same problem with mcr.microsoft.com/mssql/server:2017-latest which I cannot mount to local persistent volume. Switching to mcr.microsoft.com/mssql/server:2017-CU11-ubuntu works for me.

[Calvin-Yuhui-Liu]

Thanks @yang-zhang-syd ! Just confirmed the server:2017-latest doesn't, server:2017-CU11-ubuntu works.

[watfordgnf]

On Windows 10 1809, I can confirm that server:2017-latest, server:2017-CU14-ubuntu, server:2017-CU13-ubuntu, server:2017-CU12-unbuntu, server:2017-CU11-ubuntu all fail with the following (version numbers may differ):

PS C:\scratch\docker\compass> docker logs 7642a00dde90
This program has encountered a fatal error and cannot continue running at Mon Jul  1 14:12:46 2019
The following diagnostic information is available:

       Reason: 0x00000006
       Status: 0x40000015
      Message: Kernel bug check
      Address: 0x6b444180
   Parameters: 0x10861f680
   Stacktrace: 000000006b52fc73 000000006b4441db 000000006b431422
               000000006b43fe62 000000006b52e08a 000000006b52cb8d
               000000006b56c0d2 000000006b8ac000 000000006b838000
               000000006b840000 0000000000000001
      Process: 14 - sqlservr
       Thread: 25 (application thread 0x4)
  Instance Id: ec8d52df-1b70-4881-92d0-8a242c5fbe4b
     Crash Id: fd46606b-f34b-4501-858f-712c66824aa9
  Build stamp: b2ce95e3a6684060d1d3bb1264841a1a8fc1d501ed6d1cfdb89cfcdde8048253
 Distribution: Ubuntu 16.04.6 LTS
   Processors: 2
 Total Memory: 2061176832 bytes
    Timestamp: Mon Jul  1 14:12:46 2019

Ubuntu 16.04.6 LTS
Capturing core dump and information to /var/opt/mssql/log...
dmesg: read kernel buffer failed: Operation not permitted
No journal files were found.
No journal files were found.
Attempting to capture a dump with paldumper
WARNING: Capture attempt failure detected
Attempting to capture a filtered dump with paldumper
WARNING: Attempt to capture dump failed.  Reference /var/opt/mssql/log/core.sqlservr.14.temp/log/paldumper-debug.log for details
Attempting to capture a dump with gdb
Captured a dump with gdb
Core dump and information are being compressed in the background. When
complete, they can be found in the following location:
  /var/opt/mssql/log/core.sqlservr.07_01_2019_14_12_46.14.tbz2

Launched with:

docker run -m 2GB -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=so-secret" -p 1401:1433 -v datavolume:/var/opt/mssql -d mcr.microsoft.com/mssql/server:2017-blah-blah

I'm still searching for something, anything, that works with SQL Server for Docker. (this is after going through a few versions of Docker for Windows itself to even run the container)

Same can be said for CU's 9 and 10, and 2019's CTP.

[apozgaj]

I am trying to run the same thing inside of the Kubernetes that comes with Docker for Windows and I'm getting the same error as mentioned above.

I was able to get the docker container run with the mounted volume (using docker comands) when I was targeting :2017-CU11-ubuntu tag but when I pushed the same thing into the k8s cluster, it crashed and the behavior was the same (immediatelly after couple of seconds the container would crash) and the error was the one mentioned above.

[sirio3mil]

Same for me, reported here

[tristanvandam]

Also getting the same issue. running the following compose:

db:
    image: mcr.microsoft.com/mssql/server:2017-latest-ubuntu
    container_name: localDb
    ports:  
      - "1433:1433"
    environment:
      SA_PASSWORD: "MySuperRandomStrong!Passw0rd!"
      ACCEPT_EULA: "Y"
    volumes:
      - C:\Docker\MountedVolumes\mySqlServerData:/var/opt/mssql