Open Google Drive on PC or in App. Go to Shared with Me section and select the Files/Folders you wanna remove. Now the most important part, After selecting the stuffs you wanna remove, use your mouse/touchpad to drag the Files/Folders into Trash folder section and after that just clear your Trash.